Filtering out the noise

by Samantika Gokhale

Last year, The Sims creator, Will Wright, launched a new online TV show “Bar Karma” which invited a community of fans to drive the show’s content. Users could pitch one-line story concepts using an online tool, members would vote on the ideas, and the most popular plots would be made into a 30-minute episode produced by a professional cast and crew. A start-up which focused so strongly on user-centric design must have been a hit series, right? Wrong. The show flopped and is now ancient history.

Lean start-up theory encourages entrepreneurs to launch early, put a minimum viable product into the hands of real customers, learn quickly about how they use it, and iterate the business model based on their feedback. However, the problem with listening to your users is that they can often be unreliable. They may not actually know what they want until you show it to them. They may tell you what they think you want to hear. Their suggestions maybe skewed towards their experiences with existing products. And finally, their opinions may not be representative of the majority of your target consumers, particularly if they are early adopters. So how do you stop yourself diluting your product vision with hundreds of erroneous features? Should a true visionary simply ignore consumer testing altogether? How do you filter out the noise and understand what you truly need to change about your product?

I have a few ideas for how for a budding entrepreneur can make user testing more useful:

  1. Ask users to make trade-offs: User feedback can often lead companies astray because the users make product suggestions without understanding what the trade-off would be, either in terms of increased complexity, reduced functionality or higher prices. A good example here is Drew Houston’s refusal to incorporate Dropbox’s most requested feature, the synchronisation of files outside of the Dropbox folder, because of the conflict with the product’s simplicity and the safety of the user’s data. An entrepreneur could create surveys which ask users to make trade-offs between pairs of features, and hence isolate modifications that are truly valued by users.
  2. Craft tests to reveal actual preferences rather than stated preferences: Landing page smoke tests which ask users to ‘click to create an account’ or even ‘click to pay a sign-up fee’ can reveal an actual intent to join a service. These tests which attract users using ads can also produce more genuine data as users do not know they are ‘testers’, and so will not feel compelled to provide feedback for feedback’s sake or fall victim to group think, as can often be the case in focus group settings. A great case here is Facebook’s introduction of the Newsfeed feature which received a negative consumer backlash. However, the company chose to keep it as actual user data revealed that the feature was being heavily used.
  3. Listen for the user’s underlying problem, not their recommended solution: When a user asks for a product modification, the entrepreneur must try to distil what the underlying problem is that they are trying to solve. By listening to suggested solutions, the entrepreneur risks making adjustments that bring the product closer to existing solutions in the market. Instead, simply understanding the problem and brainstorming creative solutions can lead to a better product.
  4. Pick a diverse group of users: Early adopters of beta product versions tend to be power users who have very different needs to mainstream consumers. Whilst they are important advocates for the product, they will request too much functionality and so product refinement based on their feedback must be tempered. It may make sense to pick a diverse set of users with whom to test your product and solicit feedback. Furthermore, involving creative types (musicians, advertisers, artists) may lead to ideas that surprise you.
  5. Rely more on tests further down the road: User feedback can be incredibly insightful when refining product features and improving the usability of a product, once the key elements of the business model have been nailed down.

User feedback definitely has a role to play in product development, but the smart entrepreneur will make sure that he filters out the noise.