The data you choose to measure to test the success of a project is very important. If you measure vanity metrics instead of measuring real value based on the strategic goals of the organization, you could end up with a warehouse full of cupcake supplies.
The metrics you choose to measure the success of any project are very important. If you measure the wrong information, you might end up with a false positive. You might think a project is going well only to figure out later that it was a waste of time, money and effort.
But, if you measure the right metrics and approach Value Metrics like a scientist, you'll be able to weed out wasted effort much faster and more efficiently.
When Quantitative Strategist Brandon Ritzo began working for a home loan company in 2012, each customer was receiving a custom cupcake to remind them to fill out their loan paperwork. For $25 a pop, the cupcakes were made and shipped overnight to their destinations. Brandon was tasked with figuring out if the cupcakes scheme benefitted the company's bottom line.
You have to choose Value Metrics very carefully because if you end up using the wrong ones, you're going to see value where there isn't any. For example, in the cupcake experiment, people responded very positively to the cupcakes on social media by posting pictures and sending virtual thank you notes. Although interacting with customers on social media is a form of engagement, it's also a vanity metric. “Likes” and sales are two very different things.
The real Value Metrics of the cupcake scheme, Brandon thought, should be whether or not the cupcakes resulted in more customers closing loans. After all, closed loans directly added value to his employer's bottom line.
To know if an initiative is working, you have to have a control group. When the cupcake creators thought up their idea, they decided to buy the supplies in bulk and rent a warehouse to store the excess. Brandon put some of the extra cupcake supplies to good use by differentiating the cupcake group from the non-cupcake group. Without a control group, you can't test the results of your efforts.
When Brandon's team started to compare the control group to the cupcake group, they discovered that the cupcakes were resulting in more customers turning in their paperwork! To a less scientific person, this might indicate it's time to throw an office party and rent another cupcake warehouse.
But even though signed paperwork was the Value Metric the group originally set—the cupcakes came with a note reminding people to send in their paperwork—it turned out to be a Vanity Metric, a piece of data that makes you feel really good, but is actually concealing bad results.
Brandon's team found that the cupcakes had no effect on the real Value Metric: closing loans.
Not only were the cupcakes, shipping and warehousing costing the company money, but processing the increased amount of paperwork resulting from the cupcakes was costing the company money without any benefit to the bottom line.
“If we get extra paperwork, but we aren't getting to the final stage, everyone is worse off. We've wasted everyone's time and created more work,” Brandon said. “When you apply that logic, you're not only spending money on cupcakes but you're also costing the company money in overhead.”
After discovering the real cost of the cupcake experiment, Brandon's employer stopped shipping cupcakes. Even if it's hard to admit that an idea may be a bad one, the only other option is to trudge through bad ideas, accept sub-par results, and ultimately have to admit an even bigger failure down the road.
After shutting down the cupcake scheme, Brandon began looking for other projects that weren't going well. The project managers knew something was broken, so Brandon's team would come in, break the process apart entirely and see where they could deliver more value.
For example, the company also provides free credit counseling for people who don't qualify for a home loan. “We were drowning in clients for this service,” Brandon said. People would sign up, but then many would almost immediately stop participating. For this example, the Value Metric was whether or not the customer was able to improve their credit score to the level required by most banks to get a mortgage.
The conventional wisdom was to work with wealthier customers who would have the resources to dig themselves out of the hole, or to work with customers closest to achieving their goals because there was less distance to cover. But Brandon's team was able to isolate a much more influential factor: how well these customers lived below their means. So, they developed some methods to identify these customers and encourage counselors to focus on these customers first.
From there, Brandon and his team streamlined ways to test ideas as quickly as possible. From the initial results, they could slowly scale up successful projects before putting the full weight of the company behind something that might not add value to the organization.
Even if Brandon's team has improved an initiative, they never stop measuring Value Metrics. For example, after the credit counseling experiment, 2 percent of all call volume continued to follow the previous standards. Even improved initiatives aren't perfect. You should still measure the right Value Metrics, have a control group, and keep experimenting and refining to avoid perpetuating errors from the past.