No innovator fails because they couldn’t build their product. They fail because no one found value in what they built.
Here’s what usually happens when someone gets a new idea:
Sadly, this is the typical cycle of innovation. Most new initiatives fail.
But successful innovators know that they have a very powerful tool at their disposal to significantly decrease the risk of innovation: experimentation.
Experiments are small bets that you make to see if what you believe to be true is actually true - to see if your predictions about the customer and the market are right. It’s something small that you do today to prove that you are spending time and money on the right things . . . on building something that people will buy.
In All in Startup, the reader meets Owen after he has followed the very process outlined above and wasted hundreds of thousands of dollars and a year of his life building something that wasn’t meeting any of his projections. What could he have done before he committed his available resources to executing his plan, to make sure customers would be waiting for him when the product was ready? He should have run some experiments!
But the hardest thing about experiments is running them correctly.
Here are some guidelines about what a good experiment looks like.
To run a good experiment, you need to determine and document these five elements before you begin:
What is it that you’re trying to learn? Or what are you trying to prove? What are the riskiest assumptions you’ve made about your idea? For Owen, his riskiest assumption was that people would buy half-priced, used bicycles online.
Your high school science classes taught you what you need to know for this part. This is a statement that you’re trying to prove true or false through the experiment. The result will be a “yes” or a “no” so you need to phrase your hypothesis appropriately.
For example, a good hypothesis would be based on this setup: “If I do this action, then this outcome will happen.”
The key is to make sure that your Hypothesis helps you get closer to the Goal you outlined above and reduce the risk of your riskiest assumption. The hypothesis can help you test whether you have identified the right customer segment, whether your target customers actually have the pain point you think they do, whether they perceive enough value in your solution to buy it, whether your solution actually solves their pain point, whether you’ve identified the right channels to target your customers, whether your supplier cost estimates are accurate, whether you’ve chosen the right price point for your product, etc.
The biggest mistakes in putting together a hypothesis include:
Some example hypotheses Owen could have created:
Who are you targeting with the experiment? How are you filtering who will participate and who won't? For instance, if Owen puts up a landing page to see if people interested in road bikes will want to buy his bikes, what kind of information is he gathering on the landing page to make sure that the right people are seeing his messaging before he decides whether it's working or not?
Tip: If you are having trouble limiting your target subject for the experiment, try first listing out people who wouldn't fit into your target subject. I.e., for Owen it's people who want to buy a $100 bicycle at Walmart or Target or perhaps people who are interested in roadbikes but have never actually purchased one because they think they are too expensive.
How are you going to conduct your experiment? What’s the time period? How do we know when the experiment has started and has finished? How many people will you target? Who will carry out the experiment?
A key question to ask yourself here is: is this the least amount of time and effort I can spend to test this hypothesis? Remember, this is supposed to be a small bet you can afford to lose. Too many people think their experiment is building a lighter version of the final product – taking 6 months to put together.
You should be able to run your experiment in under 2 weeks. I will frequently push my innovators to come up with a hypothesis and figure out a way to start the experiment within 24 hours.
Something of value that the subjects of your experiment have to give you in order to prove whether your hypothesis is true.
The key is that it be something painful for them to give up in order to demonstrate their sincerity. This can include anything from money to time to a commitment of certain resources. Basically anything that demonstrates customers will be willing to part with something they value in order to obtain the product or service you’re offering. You want to make sure that they aren’t just trying to be nice to you or lying about their intent for some other reason.
When you are trying to de-risk an idea, the worst kind of evidence you could gather is a false positive (i.e., perceived interest from people who don’t actually see value in your product) because it gets you excited about moving forward and going All In on the idea. Simulate a world where your product already exists, and see if your customers will give you the currency you think you deserve.
In the Owen hypothesis examples above, Owen is asking for specific actions, a commitment of money or commitment of time to demonstrate whether he’s identified the right customer segments, marketing channels and distribution channels.
Before you begin your experiment, it’s important to define what success and failure will look like. If success is having 25 percent of customers give you currency, what does it mean when only 15 percent provide it? Has your experiment failed?
You need to set up these parameters before you begin the experiment, so that you’ll objectively understand the outcome and not be forced to debate what the results mean with your team.
Additionally, a friend of mine, Justin Wilcox, suggests that innovators write out two separate plans of action to pursue depending on whether the experiment reveals the hypothesis to be correct or incorrect. He emphasizes that this should be done before conducting the experiment. Some people have such a hard time deciding what to do if an experiment doesn’t go as they had planned that they end up making up a justification of why it was a success and allowing themselves to keep moving forward on their idea.
OK. Those are the elements you need to determine and document before you begin your experiment. And while this experiment framework can seem cumbersome, remember, it is your single greatest asset in reducing the risk that goes with creating something new.