Focus Group Testing

Tim Cain's Love-Hate Relationship with Focus Groups

Tim Cain describes having a love-hate relationship with focus group testing that has lasted decades. Focus groups weren't common in the 80s or 90s but became very popular in the 2000s and continue into the 2020s. They're almost always publisher-driven — publishers want them, often because they're not confident in their own opinion of the game and want external validation. The practice originates from marketing (like testing a new Doritos flavor).

What He Loves: Behavioral Data

Video Recordings Reveal Friction

The first major benefit comes from recording players. Cameras capture both the player's face and their screen, letting the development team watch exactly where people get confused, frustrated, or angry. This is invaluable for identifying problems:

• UI issues: "I hate this inventory system," "I don't understand this popup," "I don't know what those sparkly things around the icon mean"
• Narrative problems: NPCs that are annoying or unclear in their role
• Pacing issues: Players feeling too directed ("I wanted to explore but was told I had two minutes to get somewhere")

When many focus group participants give the same feedback, it's a strong signal that something needs attention.

Heat Maps Are Gold

The second thing Tim loves is heat maps — logged data of player positions and events. The game records the player's location whenever key events happen (movement, death, combat start, item pickup), and these logs are later visualized as heat maps showing where activity concentrates.

Tim shares a great story from Carbine Studios (WildStar): They had planned to place a dungeon instance on top of a cliff that was only accessible by flying or NPC transport. During a focus group, a woman managed to reach the top by jumping between two small rocks on the cliff face. The heat map trail showed exactly how she did it — something the team never anticipated. "We need to fix that."

Other heat map uses include death maps (revealing a level 30 creature accidentally placed in a level 5 zone) and item drop maps. The key advantage is objectivity — heat maps show what players actually do, free from the biases of verbal feedback. Even when a high-level enemy is killing players, the solution isn't necessarily to remove it — maybe just add an NPC warning: "Be careful, there's a gorgon in there."

What He Hates: The Unreliability of Player Opinions

People Don't Know Why They Like Things

Most people don't understand why they like or dislike something. They think they do and will explain at length, but their stated reasons often don't match reality. Someone who loves crafting might praise a game's crafting system even though the UI is terrible and crafted items are worse than found loot — "I liked it because I could dye my armor black." That's not useful feedback about your crafting system.

Frustration Bleeds Across Features

Players who hate one feature (like combat) will start complaining about completely unrelated systems. Their frustration poisons their perception of everything. You can't even be sure they genuinely dislike those other features.

Contradictions Everywhere

Focus group members contradict each other — some love turn-based combat, some hate it, for the exact same reason ("it's turn-based"). They also contradict themselves: "I hate when games hand-hold me" followed immediately by "you need to add quest markers to your map."

Cherry-Picking by Stakeholders

Team members, managers, and publishers will selectively quote focus group comments to support their pre-existing agendas. "See, I told you they wouldn't like combat!" — based on one person out of fifty testers.

The Silence Problem

Some players only mention what they dislike, never what they enjoy. Others are too polite to criticize anything. When nobody mentions a feature like UI, it might mean the UI is excellent — or it might mean it's so bad people have moved on to complaining about other things. Good UI is invisible; people only mention it when it frustrates them.

The Deeper Philosophical Objection

Tim connects this to his broader design philosophy: he makes games for himself, hoping others will enjoy them too. He draws an analogy to music — a singer-songwriter who writes from the heart versus a performer backed by a professional songwriting staff. The songwriter's work is more personal and particular; the staff-written songs might chart higher but lack that distinctive character.

Games like Fallout, Arcanum, Temple of Elemental Evil, Vampire: Bloodlines, and The Outer Worlds all share a similar vibe because Tim had significant creative authority over them. The ones that feel different are the ones where someone else had final say.

Once you rely too heavily on focus groups, game design becomes design-by-committee — and Tim doesn't think committee games are good. Whether it's management, publisher, or focus group committee, group-think applied to artistic work produces a lesser, less particular product.

The Verdict

Focus group testing has its uses, particularly the objective behavioral data from recordings and heat maps. But the verbal feedback is fraught with dangers that developers must understand. The tool works best as a diagnostic instrument for identifying friction, not as a compass for creative direction.

References

• Tim Cain. YouTube video. https://www.youtube.com/watch?v=sCGeOsHV4Ag