Launching Tech Ventures: Customer research methods

by Tiffany Niver

We don’t know what we want. Decades of psychology, economics, and sociology research has shown that people are quite poor at predicting what they want, how it will make them feel, and the long-term implications of gratification. This makes ground-breaking product innovations all the more complicated as understanding users becomes an exercise of mind-reading…and then some. Knowing the limitations of consumer insights and going into the process with a healthy sense of skepticism can empower creativity in coming up with the most impactful tests and analyses. The key to getting great consumer information is focusing on obtaining as much behavioral insight as possible, while following the lean start-up principle of “maximizing learning for unit of time and effort expended.”

Many entrepreneurs and researchers follow a fairly standard path of gaining customer insights.

Baselining Through Consumer Self-Reporting: Before any development or investment begins, they rely on surveys, focus groups, and interviews to learn about markets and consumers. These self-reported methods can be extremely useful in understanding purchasing behaviors, usage patterns, pain points, and customer needs. Oftentimes consumers can accurately articulate their frustrations and desires, and these become cheap and easy ways to get validation for business ideas. However, consumers base their recommendations or experiences on the status quo and thus have difficulty imagining a world which is much different from their current situation. Thus for incremental changes to existing products, these techniques can be extremely useful, but for dramatic deviations from the status quo, the reliability of these methodologies decreases.

Piecemeal / Feature Testing: Once entrepreneurs have their concept finalized and some features or interactions available, they complete smoke tests and feature tests to see how consumers interact with their concept or feature. These tests can further analyze one or two hypotheses and can be helpful in understanding how much appetite there is for a concept or how users react to pieces of a product. A major limitation is that the consumer (and the entrepreneur) still doesn’t know how the set of features or hypotheses will interact together.

Full Product Testing / Behavioral Insights: Much further down the road, a full product is available and a consumer can test every aspect of the experience. At this point, entrepreneurs have access to a tremendous amount of information provided by every click or every piece of feedback that the consumer gives. This helps get into the mind of the consumer and is usually the first time where an entrepreneur can get the actions rather than just the words or intention of consumers.

Each phase of consumer feedback is extremely important, but - with the limitations of self-reported information - oftentimes behavioral feedback is the ideal methodology. A key obstacle, however, is that this necessitates actually having something for users to test which requires time and money. The “holy grail” of consumer testing would be a low-development behavioral test where one could get the benefits of usage data without the costs. While this is clearly difficult to do, entrepreneurs should push themselves to be as creative as possible with testing by doing things like:

Using existing data on consumer behavior more thoroughly (i.e. Wings using Facebook data to launch a product based on consumer behavior)

Creating quantitative data / observing consumers in similar environments (i.e. Cake Financial observing consumers on Quicken, on their trading platforms, or doing research)

Continuing to build the most minimally viable products to test consumer behavior (i.e. Rent the Runway putting together trunk shows)

A key learning I’ve come away from my own experiences and LTV thus far is that actions speak louder than words and until you have something in front of the consumer, you don’t know exactly how they will react. So get out there and do some behavioral testing!!

by Samantika Gokhale

Last year, The Sims creator, Will Wright, launched a new online TV show “Bar Karma” which invited a community of fans to drive the show’s content. Users could pitch one-line story concepts using an online tool, members would vote on the ideas, and the most popular plots would be made into a 30-minute episode produced by a professional cast and crew. A start-up which focused so strongly on user-centric design must have been a hit series, right? Wrong. The show flopped and is now ancient history.

Lean start-up theory encourages entrepreneurs to launch early, put a minimum viable product into the hands of real customers, learn quickly about how they use it, and iterate the business model based on their feedback. However, the problem with listening to your users is that they can often be unreliable. They may not actually know what they want until you show it to them. They may tell you what they think you want to hear. Their suggestions maybe skewed towards their experiences with existing products. And finally, their opinions may not be representative of the majority of your target consumers, particularly if they are early adopters. So how do you stop yourself diluting your product vision with hundreds of erroneous features? Should a true visionary simply ignore consumer testing altogether? How do you filter out the noise and understand what you truly need to change about your product?

I have a few ideas for how for a budding entrepreneur can make user testing more useful:

Ask users to make trade-offs: User feedback can often lead companies astray because the users make product suggestions without understanding what the trade-off would be, either in terms of increased complexity, reduced functionality or higher prices. A good example here is Drew Houston’s refusal to incorporate Dropbox’s most requested feature, the synchronisation of files outside of the Dropbox folder, because of the conflict with the product’s simplicity and the safety of the user’s data. An entrepreneur could create surveys which ask users to make trade-offs between pairs of features, and hence isolate modifications that are truly valued by users.

Craft tests to reveal actual preferences rather than stated preferences: Landing page smoke tests which ask users to ‘click to create an account’ or even ‘click to pay a sign-up fee’ can reveal an actual intent to join a service. These tests which attract users using ads can also produce more genuine data as users do not know they are ‘testers’, and so will not feel compelled to provide feedback for feedback’s sake or fall victim to group think, as can often be the case in focus group settings. A great case here is Facebook’s introduction of the Newsfeed feature which received a negative consumer backlash. However, the company chose to keep it as actual user data revealed that the feature was being heavily used.

Listen for the user’s underlying problem, not their recommended solution: When a user asks for a product modification, the entrepreneur must try to distil what the underlying problem is that they are trying to solve. By listening to suggested solutions, the entrepreneur risks making adjustments that bring the product closer to existing solutions in the market. Instead, simply understanding the problem and brainstorming creative solutions can lead to a better product.

Pick a diverse group of users: Early adopters of beta product versions tend to be power users who have very different needs to mainstream consumers. Whilst they are important advocates for the product, they will request too much functionality and so product refinement based on their feedback must be tempered. It may make sense to pick a diverse set of users with whom to test your product and solicit feedback. Furthermore, involving creative types (musicians, advertisers, artists) may lead to ideas that surprise you.

Rely more on tests further down the road: User feedback can be incredibly insightful when refining product features and improving the usability of a product, once the key elements of the business model have been nailed down.

User feedback definitely has a role to play in product development, but the smart entrepreneur will make sure that he filters out the noise.