Observer Effect
The moment you measure user behaviour, you change it — beta testers, survey respondents, and A/B subjects all behave a bit differently from everyone else.
Why PMs should care
Users on a moderated Zoom usability test behave differently from users alone at home on a Tuesday night. Users who opted into a beta behave differently from the general population — they selected themselves for tolerance and curiosity. Users in an A/B test sometimes notice they're in one, especially when the difference is visible (a missing feature, a different price).
None of these effects are fatal to good research. But every readout needs to acknowledge them. Otherwise you'll end up shipping for a population that doesn't actually exist.
The habit that helps: pair moderated sessions with unmoderated ones. Pair beta feedback with silent data from users who didn't opt in. Pair visible A/B tests with a server-side measurement you didn't tell anyone about.
Example in product work
A beta user posts in the community Slack: 'The new dashboard is incredibly clear and the onboarding flow is really well thought out.' The team screenshots it for the launch deck.
But the same user has also: completed a 40-minute interview with the PM, answered three NPS surveys, been quoted in a marketing asset, and received a handwritten thank-you postcard. They are not the median user. They are a co-owner of the feature at this point, and their judgement cannot be trusted as an independent signal.
The analytics from silent users in the same cohort show 30-second average time on the dashboard and a 22% drop-off at the second screen. That's the number that belongs in the launch deck. The Slack post belongs in a thank-you card.
What to do when you see it
- Users on a moderated Zoom call behave differently from users alone at home on a Tuesday night.
- Beta users self-selected for tolerance and curiosity. They're not your average user.
- A/B test users sometimes notice they're in a test — especially when the difference is visible (a missing feature, a different price).
- Pair moderated sessions with unmoderated ones. Pair beta feedback with silent cohort data. Pair visible A/B tests with a server-side measurement nobody was told about.
Sources & further reading
- The Hawthorne Effect — Economist, 2008A skeptical re-examination of the original Hawthorne studies that gave the observer effect its modern meaning in social science.
- Observer Effect — The Decision LabAn explainer covering how the act of measurement changes the behaviour being measured.