Abstract
Problem: How do you identify and fix design problems — not bugs, but cases where everything works as specified yet the result is still bad?
Approach: Tim Cain presents a four-step method (Notice, Confirm, Fix, Test) using the energy weapon skill from the original Fallout as a running example.
Findings: Design problems are insidious because they work "correctly" — QA may not report them and playtests may miss them. Telemetry, systematic confirmation, creative system-level fixes (not just content fixes), and dedicated re-testing are all essential.
Key insight: The best fix for a design problem is often a system-level change (like skill specialization in The Outer Worlds) rather than a content-level band-aid (like scattering early energy weapons into loot tables).
The Distinction: Design Problems vs Bugs
Tim draws a sharp line between bugs and design problems. A bug is something not working as specified. A design problem is when everything works exactly as the design document described — programmers implemented it correctly, artists delivered the assets — but the result is still wrong. The design itself is the problem.
His running example: in the original Fallout, players could tag the Energy Weapons skill at character creation, but energy weapons don't appear until the last third of the game. Every system worked as designed. It was still a terrible experience.
Step 1: Notice the Problem
The hardest step. Design problems are invisible to people who aren't affected by them:
- Playtesters using characters without Energy Weapons never notice that none are dropping
- QA might have the skill but assume the late availability is intentional
- Sample size matters — with many possible character builds, any single playthrough only covers a fraction of the design space
Tim advocates for telemetry and logging as the solution. Record game sessions, then run reports: "Show me when energy weapons first appear and where they come from." The data reveals what individual playthroughs can't.
Step 2: Confirm the Problem
A step many people skip. Once you think you've spotted a problem, verify it's real:
- Is the energy weapon truly unavailable, or is there a vendor with a 33% chance to stock one that just hasn't triggered in your playthroughs?
- A 1-in-3 chance failing five times in a row (1/3⁵ ≈ 1/243) isn't that rare — it will happen to someone
- Check the design docs: is this intentional? Does any goal statement justify it?
Tim notes that Fallout's design doc didn't use explicit goals — which made it harder to determine whether the late energy weapons were a deliberate choice or an oversight. This is another argument for putting goals in your design documents.
Step 3: Fix the Problem
The obvious fix isn't always the right one. Tim walks through two approaches:
The Content Fix (Fragile)
Drop energy weapons earlier in the game. But where?
- If placed in an inventory players rarely access (a character they won't kill or pickpocket), it's still effectively missing
- Low drop chances on vendor tables or creature loot tables may not help
- If you add many early energy weapons, they'll need to be weak — making players feel their skill investment was wasted on "sad weapons"
The System Fix (Robust)
Change the skill system itself. This is what Tim's team did in The Outer Worlds: skills had base categories that required 50 points before specialization unlocked. Ranged Combat split into Single-Handed Guns, Long Guns, and Heavy Weapons.
The elegance: players spending points toward Heavy Weapons were actually investing in generic Ranged Combat first — using weapons already available in the game. By the time they hit the specialization threshold, heavy weapons were naturally appearing in the world.
No loot table changes. No re-itemizing the world. The fix lived in the skill system itself.
Step 4: Test the Solution
The step people forget. After implementing a fix:
- Go back and verify it actually works — don't assume
- Create a specific test plan for QA: "Note when you first find energy weapons. Is the skill useful? Are you actually using it?"
- While you're at it, audit adjacent systems: Are there enough locks for lockpicking? Enough traps for trap disarming? Do medical conditions occur often enough to justify First Aid and Doctor skills? Are there items that trivialize those skills?
None of these are bugs. They're all design problems that can ship if nobody looks.
The Core Lesson
You can write a perfect design document. It can be implemented exactly as specified. Every asset can be beautiful and functional. And you can still look at the result and say: "This is what I asked for, but it's not what we should have."
The four steps: Notice it. Confirm it. Fix it. Test it.
References
- Tim Cain. YouTube video. https://www.youtube.com/watch?v=DdxAhEocaE0