Hey Ryan: I’d Measure for Experimentation

So in our ongoing online nerd-out, Ryan posed the following question around measuring innovation:

If you had to measure one thing, just one thing, for (growth and) innovation, what would it be?

Holy hard questions, how am I supposed to answer that? Never mind the fact that every major management consultantency and their boutique spin-off siblings are manically running around trying to figure out how to measure innovation, now you’re only going to let me chose one thing? Ok, fine, I like a good unsolvable problem, lets do it.

The nice thing about really hard questions is that it forces you to really break things apart, which is sort of the whole point of this exercise. I started to think about how most metrics actually measure multiple things, it’s a synthesizer of the organization. GE, for example, tracks innovation through measuring organic growth. In that case you have find new revenue and the assumption is that to capture that revenue, you must be innovating to create that value. Diego’s Mileage Metric is a readiness measure that looks for people that have operated in a certain environment AND those products have gone to market. It’s not enough to just be near innovation, you need to have soldiered through the morass.

The nice thing about both those metrics is that they’re designed to suggest what you want as an outcome as much as they can help you track how you’re doing. Good innovation measure should be pretty transparent in order to encourage specific activity. Measures and Metrics are in some ways self-fullfilling prophecies when executed right. You do yourself a disservice when you get too clever with your metrics. I think that’s the reason why there’s so much anxiety and hand-wringing going on about metrics. Simple metrics give the illusion that we aren’t running a sophisticated operation, and complicated metrics cause us to think about some dark corner-case that sends the entire organization down the wrong path. For those reasons, simple, transparent metrics seem to be the most appropriate.

If I wanted to make sure innovation was happening in my organization, I’d try to measure average experiments per person. I think a foundational principle of innovation is trying small tight experiments to understand how things work (or could work better). Additionally, I think you’re not learning till things start breaking. If you do it right, innovative organizations are ultimately just as much learning machines as they are execution machines, so fostering that continuous exploration and learning seems important. Average experiments per employee is a signal to the organization that on every level you must trying, evaluating, and incorporating new methods as you go.

An experiment is pretty relative to the job you’re executing. People with more operational/executional job are probably performing more incremental experiments. People who develop offerings and services are encouraged to iterate on what they have.

Measurements should help manage tensions. Ultimately, every business is vectoring and managing among multiple goals. I’m assuming you’ll have other measures that track revenue, efficiency, customer satisfaction, etc. So I would hope measuring experimentation against those traditional operational metrics would help create the necessary tension propel innovation. The efficiency measures hopefully keep your experiments tightly scoped, the revenue measures make sure that your experiments will serve the business.

I have no doubt measuring experimentation would be hard. It’s a highly ambiguous concept, open to all types of interpretation. I think this is where a companies culture will have to come in. I think the organization will have to define what an experiment means for the business, but that seems like an important step in evolving an innovative organization.

What do you think?

Leave a Reply