Focus Testing your work

A student who’s been working on his short film for a long 7 months recently asked how one manages to keep a fresh perspective. Animators frequently reach a stage of creative fatigue and begin to question their judgment. It can happen over months on a short, and it can happen over days on a difficult shot. I suggested he do a little focus testing — put the film in front of other people, and get their reaction. Which reminded me of formal audience testing, long a hot topic among animators. Many seem to think that audience testing is poisonous, and leads to the lowest common denominator in entertainment.

The late great Chuck Jones said:

What I did, I always tried to do for my own sake—I never thought much about the audience, I never made pictures for audiences—but for my own sake…*

As a counterpoint, Hans Perk recently posted a fascinating item about Walt Disney’s ‘Laugh-O-meter.’ Here’s the news that Disney not only used audience testing (which I’ve seen alluded to many times), but that he tried to make it a relatively objective process. I particularly love that Disney came up with a system that measured an audience’s actual reaction, and not what they claimed their reaction was. They either laughed, or they didn’t. (Such a system wouldn’t work for dramatic work, but I think there are systems that could be useful in those cases.)

Perhaps what the Laugh-O-meter looked like?

I understand the reluctance about audience testing. Living in Los Angeles, I’ve been recruited to be in test audiences numerous times. If you look like you fit the right demographic and go to the right places, you’ll often be approached by people with clipboards inviting you to a free screening of an unreleased movie. Two things have always bothered me about these experiences. First, as I sit there in the test audience waiting for the film to start, I scan the rest of the crowd (and you always end up sitting for a long while, then they give this little talk about how the film is unfinished, then some producers, actors, and director slip in as the lights go down, then the film starts). The audiencs always look the same — an odd bunch of what seem like semi-professional audience members, retirees, out of work industry folk, and cheapskates. There’s apparently a whole network of people who know how to get into these free screenings, and they don’t look much like a typical movie audience to me. It’s a weird mix of people who seem to exist at the periphery of the industry, and who don’t seem to have much to do with a real audience.

Second, and probably more importantly, is the system for rating the films. After you watch the movie, they give you a stiff, colored sheet of paper with questions on front and back. There are lots of multiple choice questions that, at least for me, always miss the point. At the end of the film I know exactly what worked and what didn’t for me, but there’s nowhere to put that information. I wish I could remember some of the inane and useless questions that are asked, but they’re so silly I’ve pushed them out of my mind. All in all it would be surprising if much useful information came out of these screenings.

All that said, I think audience testing is useful, if it’s done right. And any animator who is averse to showing their work-in-progress to others is setting themselves up for real disappointment in the end. Animation is such a laborious, long-term process that it’s close to impossible to look at your work with a fresh eye after weeks or months of micromanaging.

So is Chuck Jones wrong? No. Should you make the kinds of films you want to see yourself? Yes. Should you trust your own instincts? Absolutely yes. But never be so arrogant to think that you won’t have blind spots, that you won’t get a little lost in the process, or that someone else can’t give you helpful input.

If you’re making work to be seen by an audience, then get an audience involved at various key points, and gather some invaluable information. As an individual animator within a larger production, your focus group may be one or two coworkers you trust. It doesn’t have to be a formal process. Just another set of eyes or two, to tell you where you’re going astray, and what’s working that you should leave alone. For maximal usefulness, do this before you show it to your supervisor and director. You’ll be amazed how useful this process is, if you can handle the extra critique.

When you have that friend or two looking over your shoulder, be as relaxed as you can. If you’re anxious about how they’ll respond, they’ll sense it and won’t be frank. Instead, they’ll try not to hurt your feelings. And if they tell you things you don’t want to hear, don’t argue and don’t be defensive. I’ve been through the process of being asked my opinion, only to get a spirited defense of what I think are some bad choices. You can be sure the next time that animator asked me for feedback I gave the old, “Yeah, looks good, keep going” pat-on-the-back response and moved on. So recognize if you’re looking for actual critique, or for affirmation. If it’s the former, make it clear to your ‘focus group’ you want both barrels. If it’s the latter, well, why bother?

*From Michael Barrier and Bill Spicer’s 1971 interview here.

2 Responses to “Focus Testing your work”

  1. Alonso Says:

    I try and be really open to any critique’s that come my way. I’ve taken to just instantly start taking notes, because I have noticed myself trying to explain or argue things, I’m not trying to be defensive consciously, and there are often constraining circumstances they aren’t taking into account (here in games animation lot’s of constraints) but I realized that excuses are excuses and if it looks bad, it just looks bad-period. So now I just take notes so I don’t discourage the critiques and try and fix issues as well as my constraints will let me. can.

  2. Kevin Says:

    Good tip, Alonso. I’ve found sometimes it’s not enough to be non-defensive. I’ll ask for critique and have someone just say something like, “It looks pretty good to me.” So I’ll ask, “If this was your shot, what one thing would you change.” Sometimes they say “nothing” (and you secretly celebrate), but usually they’ll give a note that’s helpful. Your taking-notes idea is a good way to show the person that you both really want serious feedback, and that you’re actually going to do something with those notes.

Leave a Reply

The animation and animation-related musings of Kevin Koch