Confirmation Bias

Product teams walk into research hoping for a green light. We phrase interview questions in ways that lead people toward the answer we want. We lean on the responses that match our hypothesis and quietly discount the ones that don't. Nobody does this on purpose. We just notice the nodding users more than the hesitant ones.

And by the time the findings are written up, nuance gets compressed. The one genuinely excited user becomes the quote everyone remembers. The three people who said 'I guess so, maybe' get averaged into 'feedback was generally positive'.

The fix is simple to describe and hard to do: before you run a session, write down — in one sentence — what a finding would need to look like for you to kill the idea. Not to iterate on it. To kill it.

If you can't imagine a finding that would stop you, or if your questions make it impossible for that finding to show up, then you're not doing research anymore. You're building a case for a decision you've already made, and using users as witnesses.

Social feed launch. A team is building a social feed inside the trading app. They run 8 user interviews. The debrief reads: 'users are excited about the social feed.' The PM digs into the transcripts. Six users said 'yeah, could be cool' when asked 'would you use a social feed?' Two described an active behaviour ('I already check Twitter for stock tips every morning'). Nobody was asked 'and if we removed this feed after launch, would you notice?' — the falsification question. The team ships. Three months post-launch, engagement with the feed is 3% of DAU. The interviews weren't wrong — they just measured politeness, which the team read as enthusiasm.

Theranos. Theranos is the grown-up version of this failure. Elizabeth Holmes had a compelling story — a single drop of blood, run through a proprietary device, producing 240 diagnostic tests — and for over a decade, every piece of evidence the organisation encountered was filtered through 'how does this confirm the vision?' When internal scientists raised doubts, they were marginalised or fired; when results didn't match reference labs, the tests were run on conventional machines and relabelled. Investors like Rupert Murdoch and the Walton family invested hundreds of millions without demanding peer-reviewed validation, because the story was too good to disprove. An entire organisation, a board full of former Secretaries of State, and the most sophisticated investors in the world all failed to ask 'what would need to be true for this to be fraudulent?' — the falsification question the Wall Street Journal's John Carreyrou eventually asked, alone, and answered by doing actual reporting.