Product Market Fit in Health IT Training

By Dan Gebremedhin, M.D.

Several studies have shown that when healthcare workers implement new forms of Health IT (Electronic Health Records and other systems) their productivity drops as they are learning to use these new systems. (Click here to learn more about EHRs.) 

In attempting to solve this problem, our company has created a robust online training platform. The platform has proven effective in returning users to full productivity faster than the industry standard. Despite this evidence, customers have been reluctant to purchase access to the platform. In this blog entry, I’ll outline the evidence for the effectiveness of our product and begin to examine whether or not there is product market fit.


The Research
There have been a handful of studies that have looked at productivity drop with Health IT adoption. Bhangrava et al, found that providers uniformly dropped in full productivity by 25-33% for at least one month after EHR implementation. After 3 months out, internal medicine physicians returned to full productivity but a large subset of physicians never returned to full productivity during the remainder of the study.


A report by Clayton et al. out of Intermountain Health corroborates this data, by showing that their subset physicians did not return to full productivity until 6 months. Despite the drop in productivity, the physicians in the study increased their level of data entry significantly. This supports the argument that although EHR implementation decreases productivity, EHR use leads to more comprehensive care. 


iHIT Results
With the use of our platform technology and methodology, our customers return to full productivity within 4 weeks. Some customers have achieved full productivity within 7 days. From this superior margin of improvement compared to the industry standard, our clients have achieved an ROI of 4:1 and savings of > $1M. These results have been published separately at HIMSS 2011 and ACE 2011. (Click here to learn more.) 


Proven Results = PM FIT?
It would seem natural that with proven, published results, there would be significant demand for our product. With 13 healthcare organizations using our product, we have achieved initial traction. In reality, many of these sites were beta-sites that were upgraded to paid subscriptions to our platform. These existing customers did not have to face the same purchase decision that our prospective customers are currently facing. 


From our initial market launch in Q32010, we’ve had to adjust our pricing to find the willingness to pay (See below for a chart of our pricing trajectory over the last 10 quarters). We quickly learned that the ROI calculation alone was not enough to convince HCO’s to pay > $150k/year for access to training software. 


In focusing on unit economics, we’ve found that HCOs are willing to invest between $100 - 200 per physician per year in total IT online training spend. Further, we know we need to communicate the long term benefits and ROI of the product as opposed to implementation only. As this message is crafted, careful attention will be focused on our conversion rate which sits around 15%. In my estimation, if we can double this to > 30%, our CAC will drop significantly. I think then, we can say that we’ve hit PM Fit and are ready to scale the business. 

Source: Interal iHIT research

References
Bhangrava, et al. Electronic Medical Records and Physician Productivity: Evidence from Panel Data Analysis. December 2010. http://www.news.ucdavis.edu/search/news_detail.lasso?id=9665

http://www.healthcareitnews.com/news/study-emrs-effect-docs-productivity-depends-needs-workflow

Clayton et al. Physician use of electronic medical records: Issues and successes with direct data entry and physician productivity. AMIA Annu Symp Proc. 2005; 2005: 141–145. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1560588/

Read More

Are We There Yet: Thinking Through Product/Market Fit (LTV)

by Colin Barry

"Startups occasionally ask me to help them evaluate whether they have achieved product/market fit. It’s easy to answer: if you are asking, you’re not there yet.” – Eric Ries, “The Lean Startup” (pg 220)

The Context

Hypothesis-driven entrepreneurship — epitomized by Eric Ries’ Lean Startup methodology — has become all the rage among aspiring tech founders. It’s not hard to see why. Ries’ focus on customer discovery and iterative development addresses a dangerous problem in the two established paradigms for building software.

Waterfall product development presumes that the problem and the solution are known, and we just have to build the solution in an efficient, staged manner. Agile product development admits that the solution is unknown, but still presumes that the problem is known — the “voice of the customer” (usually the product manager) will recognize useful software when she sees it. But actually, Lean Startup tells us, the problem is usually unknown, too: it takes ingenuity, guts, and contact with customers <shudder> to determine whether software is useful or not.

The Problem

Okay, so we admit that we’re in the land of the blind, and we need to be disciplined and systematic about figuring out what problem we should be solving. But how do we know when we’ve got the right problem paired with the right solution?

Ries tells us that the watershed condition is product/market fit — we’ve looped through the Build-Measure-Learn process, improving all the while, until we’ve built an “engine of growth” (by which Eric appears to mean a product with rapidly accelerating user acquisition). Then, we just add fuel and watch the engine go.

The trouble is that a whole bunch of phenomenally successful and (purportedly) Lean startups slowed down rapid product development iteration — basically, decided that the product was mostly done — well before they had anything resembling an engine of growth.

Why It Matters

The lack of rigor around what constitutes product/market fit makes successfully applying Lean methodology much harder. One of our class guests, David Skok, has written brilliant expositions on constructing a repeatable, scalable sales model. He opened our class discussion with a graphic (roughly): 

 
But wait, my business only has product/market fit if I already have a sustainable user acquisition model — ideally an engine of growth so powerful that just a small quantity of “fuel” causes users to crash my webservers and beat down the front door to my startup’s office in desperate mad dash to consume my product.

But it seems to me that my venture has probably already figured out the marketing and sales part of the equation if that is occurring.

And many (if not most) successful startups do appear to delineate building a product from building a sales model.

Two examples in brief (one cribbed from David Skok):

- Airbnb enters YCombinator with a product they have already trialled at SXSW and the Democratic National Convention. They expect to do a ton of development in YC. Paul Graham tells the founders to stop building product, hop a plane to New York City (where they have the most adoption), and personally snap prettier photos of the apartments currently listed on Airbnb.
Scalable? No way.
Did the market tell them to do this? Nope (at least I don’t think so).
Did it work? Yes. Prospective renters and couch-surfers had been turned off by ugly descriptions of the listed apartments. Success.

- Constant Contact completes product development (in the words of David Skok, has found product/market fit). But prospective customers aren’t buying.
In fact, prospective customers (small businesses in Atlanta) aren’t sure why they would need mass-mailing software in the first place. Constant Contact distributes wire-bound books for small businesses to use as visitor books to record shoppers’ e-mail addresses.
A month later, Constant Contact’s prospective customers have hundreds of e-mail addresses, and now they need mass-mailing software. Success.

We can tell similar stories for startups like JBoss, RentJuice, and Dropbox.

To be clear, I don’t think David is wrong in making a distinction in the startup lifecycle between product development and marketing/sales. I think Lean Startup makes it difficult to tell when to stop focusing primarily on building product and start focusing primarily on selling product.

Conclusions

One possibility (maybe what Eric would argue) is that I’ve defined the “product” part of “product/market fit” too narrowly. The product is not just a piece of software but rather the startup as a whole — a product-marketing-sales-PR conglomeration that must be tuned and operating in harmony.

I’m not so sure. There is a natural tension between Lean-style product development and marketing/sales.

In Lean product development, we tirelessly labor to determine what the customer will use and then build something viable and desirable. If the customer tells us (by his or her behavior) that a feature should be different, we change. We pull the product out of the market.

In sales and marketing, we tirelessly labor to convince the customer that they should use or buy what we already have. If the customer tells us that a feature should be different, we generally try to convince him or her that we know better. We push the product onto the market. And ideally, the customer doesn’t require much convincing.

Lean Startup methodology needs a better definition of product/market fit than “I know it when I see it.” Maybe arbitrary standards like 40% net promoter score make the most sense. Perhaps we just admit that user acquisition growth is not the right metric for many startups, especially those that require higher-touch sales models. Regardless, I think we need to take a deeper look at what goal we’re actually chasing if we’re running lean.

Read More

Quantitative Tests For Product-Market Fit

by Josh Sandberg

Is the product-market fit concept useful for entrepreneurs and managers? How can you know in the moment (rather than in retrospect) that you’ve achieved PMF?

Knowing when you’ve achieved Product-Market Fit (PMF) is vital for those following the lean startup methodology: it tells you when to stop pivoting and start scaling. In a influential early post advocating the importance of PMF, Marc Andreessen claims that “You can always feel when product/market fit is [or isn't] happening”. However, several of the startups we’ve analyzed so far this semester have shown that reaching PMF is not always the lightning-bolt moment Andreessen claims: they’ve spent time stuck in a gray area where they’re having some success, but not as much as they’d hoped for. Do they continue to pivot in search of that magical state of PMF where ‘everything just works’, or accept that they’ve found a market that is ‘good enough’?

I believe that two slightly more specific concepts are core to what it means to achieve PMF: first, that you solve a significant pain point for users; second, that you create passion for your company and your solution. The first is key to demonstrating and capturing the value of what you have created; the second is key to spreading the word about your service and generating growth. Each of these can be tested quantitatively to help indicate PMF.

To test for user’s pain, my favorite question comes from Sean Ellis: “How would you feel if you could no longer use [my product]”? Based on his experience, achieving product/market fit requires at least 40% of users say they would be “very disappointed”. While the exact threshold is debatable, the underlying validity of the metric is not: it’s a great way of testing whether or not you are solving a significant pain point. If you are, fantastic! But if a significant proportion of your users are not seeing demonstrated value to the point where their lives would be worse without your product around, it’s fairly clear you do not have PMF – you’re going to struggle to sell, retain, and monetize customers. A useful further analysis is to identify the commonalities among those “very disappointed” users: this is your current core demographic. If you capture a reasonable share of this segment, is it a big enough market? If not, it may be time for a pivot.

To test for customer passion, my favorite question is the Net Promoter Score: “How likely is it that you would recommend [Company X] to a friend or colleague”? There’s a significant amount of research linking high NPS scores (driven by lots of strong promoters and few detractors) to higher repeat purchase rates and long-term growth. Intuitively, the concept seems even more valid for high-tech firms hoping to leverage user referrals and viral distribution. Your customers may love you because of your economic value delivered, your customer service, your elegant design, your quirky copy, or any combination thereof – at the end of the day, what matters is that they care. Eric Reis commented in class that “the opposite of loving a product is usually not hating it… it’s indifference”. Indifference is dangerous: it means you’re going to be doing all the hard lifting, so throw all those viral growth models out the window and get ready to deal with a high customer acquisition cost. On the other hand, being able to create passionate users is a powerful indicator of PMF.

Neither of these two metrics provides a precise measure of when PMF has been achieved, but I believe that taken together they incorporate many of the core concepts of PMF, and can help provide some quantitative guidance as to whether your team should keep pivoting or hit the accelerator.

Read More

Product Market Fit – Useful or Theoretical?

by Shavi Goel

Of course, entrepreneurs want to make product which customers need/want. I would actually stretch the argument and say the deep desire and egoistic satisfaction of entrepreneurs lies in creating solutions to unmet needs/ markets. Then why is it that a large percentage of start-ups fail to create something which serves any market.

Mad rush called entrepreneurship

To my mind, issue at hand is very much ingrained in DNA of entrepreneurship.

Any Entrepreneur has to be wildly passionate (read borderline crazy) to invest himself in the start-up based in garage. On top of it the fact that in most cases he is trying to challenge existing paradigms / business models implies there is no existing market / ready customer segment to test the product concept early on. Hence, he obviously believes strongly in his Product Market Fit (PMF) from the word ‘go’, well, in the future PMF.  

Risk of Cognitive Dissonance

Yes, I appreciate when Steve Blank asks to focus on early customer validation. But I can also imagine how hard it will be to listen to the customer feedback when you are so close to the product. All the objectivity can kiss itself good-bye. The justifications can find home in ‘product is not ready enough’, ‘not enough or right sample size’. Some might just side with Steve Jobs and say ‘customers don’t know enough’.

Despite all the reasons above on why it is not done. Can it be done in the moment?

At times, a new start-up can stir up entire ecosystem of customers, investors etc and feedback comes very much rushing in – usually positive sometimes negative. Those are easy calls, so strong is external validation that you know you have PMF or not.

Issue is for the majority who doesn’t shake things early on. How can PMF test be done for them in the moment? I would argue it can be done. The metrics need to be very unique to every start-up, but can be defined upfront to avoid the risk of listening to customer feedback with happy ears stated above. One of the best ways to quantify PMF can be found in the companies projections. Validating milestones which company set for itself and measuring against those is good way to confirm PMF. Are we getting enough eyeballs, conversion rate, new customers or new partnerships – right question will depend on what’s at the core of start-up. Providing for a lot of ambitious plans being reflected in the projections – I would target for a xx% hit rate on key metrics. Problem is xx% that’s unknown.

Can all this be defined up-front and continue to have inviolable sanctity in the evolving environment?

Things that can help avoid pitfalls

When doing the PMF dipstick, its important to measure right things. Can’t stress enough to steer away from ‘nice to have’ to ‘must have’ for the company. Also important to check ‘willingness to adapt’ and ‘pay’ both. Also, setting time frame ahead in the original road map, essentially like timelines for doing a beta or a prototype test. Keep channels of informal feedback open and developing as many mentors who you can trust for more than fair critique. Right investors can play a big role by asking the right questions and bringing in objective outlook.

Having given the prescription above, I doubt if entrepreneurs need it. I can see many lit-up eyes in a room when someone makes the pitch on PMF and many deeply committed entrepreneurs still repeating in abated voices –‘wonder if everyone who made it, follow this theory ..Should I be here or go back to getting product out’ 

Read More