Smoke (and other) tests: smoke in your face? Pitfalls of lean start-up testing

by Lauren Miller

The lean start-up approach has the advantage of using genuine customer feedback as a means of improving the entrepreneur’s product specifically for those perceived as the niche consumer, but numerous stories from entrepreneurs and case protagonists about acting on false positives and negatives has caused me to wonder: can smoke and other lean start-up tests just end up being smoke in the entrepreneur’s face?” I question whether an entrepreneur should take the results of his or her tests as “gospel” since pitfalls in the experiment design and interpretation of results can be numerous. I will focus on a few of these potential pitfalls.

Focus groups
Steven Carpenter shared in the Cake Financial case that he has “learned that focus groups… can be unreliable because people can’t always say what they want until they can actually see it and play around with it.” This highlights the fact that customers often can’t or won’t tell you what they want or need and are often limited in what they can even think to ask for. It further underscores the important lessons revealed in Dropbox and Aardvark. Be careful in design of focus groups to utilize the most valuable feedback, as it is not all created equal. Also, make sure to listen to your customers but don’t simply obey them.

Usability tests
Martin Kessner’s study reveals the necessity of running usability tests iteratively. In his study, six usability teams could not find a single common usability problem when independently testing the same product, and no team found 50% or more of the total problems found by all. This should be a sign of caution to the entrepreneur. Though there are a plethora of studies and start-up case examples that document the improvement achieved from usability testing, Kessner’s work could show that in testing products or usability, the iterations make the test more reliable and decrease the likelihood of wide variations in responses. My takeaway: the repetition of usability tests allows an entrepreneur to better qualify the feedback he or she receives and in turn use it as a better determinant of when and how to pivot in hopes of achieving product-market fit.

Feature lists
I am working on an education software start-up, and we currently have a survey on our site that asks teachers which features they would like. Results have been great, as teachers seem to want all of the features we plan to create. Nevertheless, I have recently begun to question the reliability of their responses. I hypothesize that if we added fifteen more features, teachers would likely indicate that they want all of those as well, but the famous Columbia University marketing study implies that products with too many features often overwhelm customers. So how much should an entrepreneur trust customers when they say they want everything? Cindy Alvarez, formerly of KISSmetrics, says that customers want everything because “wanting” is free. Testing how much value features provide by adding a cost to additional features may yield more accurate results. If entrepreneurship is an art and a science, utilizing results of tests like this is the science and the decision of whether to follow them and how is the art.

I advocate that in the lean start-up model, focus should be placed as highly on test design and accuracy in feedback interpretation as it is on running tests. This way, entrepreneurs will avoid smoke-hazed and unreliable results that end up leading to pivots that could destroy their ability to connect with and address the needs of their potential consumer base. However, I’m interested in your thoughts. What other reliability traps have you seen entrepreneurs fall into when using various mechanisms to get customer feedback?

Read More

Lean Startup Tradeoffs

by Sarah Elliot

The primary objective for most entrepreneurs is to create a product that customers want and are willing to pay for. The process by which this objective is reached often requires frequent product iterations and customer interactions. It is through this process that the entrepreneur identifies which aspects of the product vision are incommensurate with the wants and needs of the customer. Closing the communication gap with the customer is critical to the success of a venture. However, there are real and substantial consequences to exposing a customer to a product that isn’t “right”. Therefore, there seems to be an inherent tension between getting the product “right” and getting the product out the door and into the hands of customer. So the question remains,

When does it make sense to launch early with a buggy product, and when is it important to perfect the product?

The answer to this question varies and is contingent upon several contextual factors such as the nature of the product, the degree of exposure, and the core objectives of the entrepreneur. Entrepreneurs that face this question should approach this decision by asking themselves: (1) What is the minimal viable product necessary to reach my core objectives?; (2) What are the trade-offs between launching now and iterating further?; and (3) Who bears the brunt of the risk of failure --the entrepreneur or the customer?

1. What is the minimal viable product necessary to reach core objectives?

One of the core components of Eric Ries’ lean start-up methodology (http://theleanstartup.com/) is the concept of the minimal viable product (MVP). At its most basic level, the MVP is the minimum product and associated feature-set that enables an entrepreneur to validate/test her core hypothesis regarding product/market fit. The MVP process begins with a clearly defined hypothesis about the customer pain-point (problem) and product vision (solution). The next step in the MVP process is to test this hypothesis through the design and execution of a barebones “experiment” whose results demonstrate cause-and-effect relationships. The MVP methodology suggests that a product should be launched upon the creation of the absolute minimum product (or experiment). It is important to highlight that the MVP doesn’t need to be a proprietary product. The objective of the MVP is to quickly validate the core hypothesis; therefore, the use of pre-packaged solutions is ideal if it allows you to quickly and efficiently collect valuable information about the customer and your hypothesis.

As we saw in the case of Cake Financials, Steven Carpenter created a web-based solution that addressed his hypothesis of the customer pain-points: (1) a lack of transparency; (2) distrust of advisors; and (3) complex investment offerings. He believed that his solution would address these concerns by (1) aggregating financial information; (2) comparing portfolios; and (3) automating the buy/sell process. Steve and his team invested a considerable amount of time and resources building out the back-end of his product at the expense of rapid MVP tests. Had he taken an MVP approach, rather than perfecting the product in the absence of reliable customer feedback, he would have learned that his hypothesis was incorrect much sooner – a realization that could have saved his company.

2. What are the trade-offs between launching now and iterating further?

When evaluating the trade-offs between launching an imperfect product now vs. launching a more perfect product later, a two factors should be considered. First, the degree of exposure should be considered. A PR induced broad product launch will magnify the reputational risks associated with product imperfections. One key prescriptive of the MVP method is the use of early adopters as the core test group. The rationale being that early adopters are more likely to be forgiving in the event that the product isn’t quite right. One critical question to ask is, “With whom should I launch this product?” Second, one should consider contextual factors when deciding when to launch. In the case of ScoreBig (Ent Fin), Adam Kanner, made the conscious decision to wait and perfect his product for nearly two years before launch. In his case, he felt that the key stakeholders (sport and media executives) would be reluctant to supply him with ticket inventory (core element of b-model) in the absence of a nearly perfect algorithm and an aesthetically appealing front-end. While he presumably could have tested his hypothesis with an MVP approach much sooner, contextual factors prohibited him from doing so.

3. Who bears the brunt of the risk of failure --the entrepreneur or the customer?

One final consideration when deciding when to launch is the degree of risk shared by the customer. If the launch of an imperfect product exposes the customer to the brunt of the risk, think twice. Integrity must be upheld in all stages of the product development process as trust is not easily restored once broken. If the product could expose a customer to harm (i.e. exposing confidential information or jeopardizing physical safety), keep iterating! The additional information you receive is not worth the risk and the reputational damage is irreversible.

In conclusion, as an entrepreneur, every interaction with the customer is an opportunity to learn important information that ultimately drives product development. Like the time value of money theory, information today is better than information in the future. Entrepreneurs should air on the side of quick iterations to capture information as quickly and efficiently as possible. However, it is important to understand that there are consequences to “getting it wrong”, and entrepreneurs should be mindful of the impact of a failed attempt. While failure is unavoidable, entrepreneurs can adopt key risk mitigation strategies to increase the risk/reward ratio.

Read More

Help Me Help You: Getting Great Consumer Insights

by Tiffany Niver

 

We don’t know what we want. Decades of psychology, economics, and sociology research has shown that people are quite poor at predicting what they want, how it will make them feel, and the long-term implications of gratification. This makes ground-breaking product innovations all the more complicated as understanding users becomes an exercise of mind-reading…and then some. Knowing the limitations of consumer insights and going into the process with a healthy sense of skepticism can empower creativity in coming up with the most impactful tests and analyses. The key to getting great consumer information is focusing on obtaining as much behavioral insight as possible, while following the lean start-up principle of “maximizing learning for unit of time and effort expended.”

Many entrepreneurs and researchers follow a fairly standard path of gaining customer insights.

• Baselining Through Consumer Self-Reporting: Before any development or investment begins, they rely on surveys, focus groups, and interviews to learn about markets and consumers. These self-reported methods can be extremely useful in understanding purchasing behaviors, usage patterns, pain points, and customer needs. Oftentimes consumers can accurately articulate their frustrations and desires, and these become cheap and easy ways to get validation for business ideas. However, consumers base their recommendations or experiences on the status quo and thus have difficulty imagining a world which is much different from their current situation. Thus for incremental changes to existing products, these techniques can be extremely useful, but for dramatic deviations from the status quo, the reliability of these methodologies decreases.
• Piecemeal / Feature Testing: Once entrepreneurs have their concept finalized and some features or interactions available, they complete smoke tests and feature tests to see how consumers interact with their concept or feature. These tests can further analyze one or two hypotheses and can be helpful in understanding how much appetite there is for a concept or how users react to pieces of a product. A major limitation is that the consumer (and the entrepreneur) still doesn’t know how the set of features or hypotheses will interact together.
• Full Product Testing / Behavioral Insights: Much further down the road, a full product is available and a consumer can test every aspect of the experience. At this point, entrepreneurs have access to a tremendous amount of information provided by every click or every piece of feedback that the consumer gives. This helps get into the mind of the consumer and is usually the first time where an entrepreneur can get the actions rather than just the words or intention of consumers.

Each phase of consumer feedback is extremely important, but - with the limitations of self-reported information - oftentimes behavioral feedback is the ideal methodology. A key obstacle, however, is that this necessitates actually having something for users to test which requires time and money. The “holy grail” of consumer testing would be a low-development behavioral test where one could get the benefits of usage data without the costs. While this is clearly difficult to do, entrepreneurs should push themselves to be as creative as possible with testing by doing things like:

1. Using existing data on consumer behavior more thoroughly (i.e. Wings using Facebook data to launch a product based on consumer behavior)
2. Creating quantitative data / observing consumers in similar environments (i.e. Cake Financial observing consumers on Quicken, on their trading platforms, or doing research)
3. Continuing to build the most minimally viable products to test consumer behavior (i.e. Rent the Runway putting together trunk shows)

A key learning I’ve come away from my own experiences and LTV thus far is that actions speak louder than words and until you have something in front of the consumer, you don’t know exactly how they will react. So get out there and do some behavioral testing!!

Read More

Prototyping: Just Get It Out There

by Stirling Cox

The idea of a product that would enable commuters in a given city to access information on the location of their buses, has been occupying my thoughts since the summer of last year. With a background in Banking and Consulting, what had been bashed into my head through various training programs and countless deals and projects is that you need to be 100% sure of your facts before opening your mouth. As such I was convinced that it was only through significant primary research and presentation that I would ever be able to prove the viability of my idea and start a business.

I spent much time during the end of the summer and fall term of EC year researching number of commuters, their habits, Smartphone proliferation, looking at different markets that product could be applied to, producing possible revenue models in Excel, preparing financial statements and fancy PowerPoint slides explaining the business. Years of “bad” entrepreneurial practice had me convinced that I needed to have all research completely covered before even approaching possible users or potential funders.

By Christmas time, whilst I had actually spent hours on this idea, I still actually had nothing to show for it. Everytime I discussed the idea with someone and showed them my swanky PowerPoint deck, they loved it, but this was no different from 5 months prior when I voiced over the idea in 30 seconds.

Getting to grasps with the concepts of “just get it out there” or “just get on and do it” was new to me. It seemed irrational that I should start off with a half baked product, a product that wasn’t perfect, a product that may not work in its first version. But countless tales and case examples of Millionaire entrepreneurs taking this approach made me think there was a faint chance that I may be round. After much internal kicking and screaming to stick to what I knew how to do, I decided to give it a go, and developed what I viewed as a first iteration of the product.

Not having any development skills, I built with some assistance (and a tool called SWiSH) a flash live video demonstration of the product. Whilst this was not a formal website or application, nor was it plugged in to any data, this demo now allowed me to showcase to whoever I was discussing the concept with, really what the product would look like, what it would feel like, and demonstrate various capabilities through playing with the video.

Whilst this first step was definitely a step in the right direction, I now realize this was barely a prototype. People saw this and said: “Awesome, can I use it?” This was not a product, merely an enhanced graphical representation of what I was imagining.

My next step is to actually build the first real live version of the prototype. Having learned that I actually really needed to have something usable, I scraped the initial inclination to get a developer to build an application that was not plugged in to any real time data. Whilst this would be a real product, it would not be something that users could use and give feedback on. I also scraped my next idea which was endless discussions with London to open up their data so that I could build the product on their systems.

Instead, after what is now seeming like a no brainer, and a 1 year after the initial idea, I have finally found what may be a useful direction to take the business forward: leverage what is already available. As it turns out, the MBTA in Boston opened up at the start of 2011 all their data on bus positioning to the development community. This provides the perfect testing ground for the product. I am presently in talks with a developer, who has built similar geopositioning applications, about finally getting the first real prototype up and running for Boston. Whilst there are other players out there in the market, Boston provides a perfect testing ground for the first prototype. We are hoping within the next month or so to have an application out there that customers can really play with, try, and critique; an application upon which we can iterate and draw conclusions as to the viability of the idea.

Talking with people who have not taking the course, I realize that most people face this same problem. Lots of people have business ideas, and 90% of them don’t do anything at all with them; they just horde the idea. Of the remaining 10%, the most ambitious 9% spend time researching a market, preparing fancy documents and excel models, but never really get the business launched. The 1% that launch a product are not afraid of it failing, are not afraid of getting it wrong, are not afraid of criticism. They have realized that their business will never get off the ground sat in front of their computer researching the internet or pulling together fancy presentations. Ex-Consultants and Bankers who want to be entrepreneurs… you are going to have to undo a lot of bad practices.

Read More

Lean is for Wimps

by Lorin Pace & Iris Guerra

In the new era of all things lean, fat gets a bad rap. Even the terminology is loaded; in the U.S. we are facing an obesity health crisis like nothing we have faced in our nation’s history. Of course no-one would want to be ‘fat’ when the term has such a negative connotation. But when did ‘fat’ become the only alternative to lean? What about medium or athletic builds? Painting a picture of two polarized options and demonizing the other is a storied psychological tactic for building momentum around your own philosophy. For better or for worse, Eric Ries has done a great job of depicting epic failure as the product of the ‘other’ approach. And he has a point. It IS senseless to build a product no one wants, and no one is a better example of that than Ries himself (he did it!) and he knows how painful it is to pour your heart and soul into something that ends up being discarded. Ries would have you believe that not only can you apply the lean startup method to everything – but you should apply the lean startup method to everything.

One of the cornerstones of Ries’ lean startup method is the notion of the Minimum Viable Product (MVP). In Ries’ own words, “The minimum viable product is that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.” It sounds like a helpful, leveraged approach, and it is, but we’d like to slap a warning label on this product:

1. Don’t prioritize validated learning by gambling with key customer relationships. We found rather quickly that warm leads are absolutely critical for winning the business of large enterprise customers, regardless of how far along your product is. Warm leads with large enterprise customers in ideal segments are rare. Even if you have a brilliant concept, while you’re learning about what these customers truly value, you are exhausting much of their valuable time. Don’t expect them to hold their breath while you quickly iterate on the product that you figured out that they actually do want. They may not have the patience to re-engage with you. Had you been more prepared, you might have just landed a huge customer. 
2. If you are constantly validating your gut, the guy who doesn’t may beat you to market. In the wake of the Bush presidency and the ultimate failure to unearth weapons of mass destruction (WMDs) in Iraq, the notion of the ‘gut’ is almost as unpopular as ‘fat.’ Trusting one’s ‘gut’ is synonymous with making whimsical decisions based on mood and temperament. The reality is quite different. The ‘gut’ refers to the part of our brain known as the ‘limbic system.’ The limbic system assembles powerful elements of memory and feeling and association that can distill complex patterns of information into a singular decision path. It can be very powerful and trusting such hunch-driven decision making has produced many of the greatest successes in the history of entrepreneurship. It might not work every time, but startups rarely do. Overly handicapping your gut with too many feedback loops can be just as risky as the alternative.

Read More

Isn't a 'Minimal Viable Product' Kinda Lame?

by Charlotte Jepps


Forgive me for being biased by the fact I'm currently wrestling with distilling what I think is a great idea down into a 'minimal viable' form, but I can't help but find this concept inherently frustrating...

Is it realistic that many online business concepts can truly be attractive to users when presented in a simplified form?

Whilst I think this is indeed true for some platforms and businesses, which need to 'train' their users before adding complexity (eg a social game), I think there is a risk for other sites that the 'lean startup' methodology drives them toward stunting themselves - to an extent that great ideas might be deemed unattractive and thus abandoned too early.

I understand that definitionally every site has a 'minimal viable' version (which lies on a sliding scale between something incredibly simple, and the full ‘vision’ for the site), but is the term still useful if 'minimally viable' and ‘full site’ are in the majority of cases the same?

As an example, let's consider retail. Zappos is a great business, which focusses on providing a pretty spectacular level of service to shoe shoppers. As the creator of such a site I might decide to apply 'lean' principles and create the minimal possible form to test my vision for my service capabilities with early users.

But how can one really expect to usefully test the level of interest in a site if you only partner with a couple of shoe retailers?

As a user of Zappos, my minimal requirement is a sufficient level of selection – so the ‘minimal’ site really does require some pretty substantial up-front investment

Perhaps the possibility of creating a truly ‘minimally viable’ version of an online business depends on whether you’re testing something that people ‘know’ they need or not. If you’re providing a product or service that targets a clear gap in the market in the totally conventional (‘offline’) sense then you may simply be wasting time and should launch a full site, but if you’re testing dropbox you go for the MVP.

There are therefore a great majority of ecommerce ventures that should steer well clear. Who in their right mind would take advice or make online purchases from a skeleton site? How many times have you googled travel recommendations or tried to purchase products and immediately disregarded all the sites that have partial information or functionality. To a certain extent MVPs (like vark.com) get around this by plugging in to your social network to attain early credibility, but with the proliferation of Facebook Connect I question how much longer users will really apply any weight to this feature.
In sum, I think sites need to think long and hard before overly trimming down their launch version - into something that fails to stimulate a following because it is too simple to be cool. Put yourself out there and take a risk, or the only thing you’ve validated is that you’re lame.

Read More