I managed the team to a set of numbers that did not fully capture what I wanted. I wanted a great product that customers would love with high quality and on time — in that order. Unfortunately, the metrics that I set did not capture those priorities. At a basic level, metrics are incentives. By measuring quality, features, and schedule and discussing them at every staff meeting, my people focused intensely on those metrics to the exclusion of other goals. The metrics did not describe the real goals and I distracted the team as a result.

One thing in life is that there are parameters to measure success.

When a measure becomes a target, it ceases to be a good measure.” Measurement is only useful when it guides you and adds context to a larger picture, not when it consumes you. Each number is simply one piece of feedback in the overall system.

What gets measured gets managed. — PETER DRUCKER, management theorist, author

Measuring requires, first and foremost, analytical ability. But it also demands that measurement be used to make self-control possible rather than abused to control people from the outside and above — that is, to dominate them. It is the common violation of this principle that largely explains why measurement is the weakest area in the work of the manager today. As long as measurements are abused as a tool of control (for instance, as when measurements are used, as a weapon of an internal secret police that supplies audits and critical appraisals of a manager’s performance to the boss without even sending a carbon copy to the manager himself) measuring will remain the weakest area in the manager’s performance.2

When performance is measured, performance improves. When performance is measured and reported back, the rate of improvement accelerates.” — Pearson’s Law

I measure success by one metric: Can we positively impact 1B+ people? Everything else is noise... Prizes, funds, communities, books—they're all just containers for scale.

We design performance measures to cover-up error. They are called benchmarks. We make decisions based on: Will we be found out? Discovered? Identified as the wrongdoer? Do we really want to have to explain this stuff to someone who spent his or her life in sales or marketing?

It is easy to see that there are many ways for leaders to be misinterpreted. To get things right, you must recognize that anything you measure automatically creates a set of employee behaviors. Once you determine the result you want, you need to test the description of the result against the employee behaviors that the description will likely create. Otherwise, the side-effect behaviors may be worse than the situation you were trying to fix.

optimizing for engagement metrics is a great way to hide

FOCUSING TOO MUCH ON THE NUMBERS In the second example, I managed the team to a set of numbers that did not fully capture what I wanted. I wanted a great product that customers would love with high quality and on time — in that order. Unfortunately, the metrics that I set did not capture those priorities. At a basic level, metrics are incentives. By measuring quality, features, and schedule and discussing them at every staff meeting, my people focused intensely on those metrics to the exclusion of other goals. The metrics did not describe the real goals and I distracted the team as a result. Interestingly, I see this same problem play out in many consumer Internet startups. I often see teams that maniacally focus on their metrics around customer acquisition and retention. This usually works well for customer acquisition, but not so well for retention. Why? For many products, metrics often describe the customer acquisition goal in enough detail to provide sufficient management guidance. In contrast, the metrics for customer retention do not provide enough color to be a complete management tool. As a result, many young companies overemphasize retention metrics and do not spend enough time going deep enough on the actual user experience. This generally results in a frantic numbers chase that does not end in a great product. It’s important to supplement a great product vision with a strong discipline around the metrics, but if you substitute metrics for product vision, you will not get what you want.

What's measured improves

The drive to win is not, per se, a bad thing. Problems arise, however, when the metric becomes the only measure of success, when what you achieve is no longer tied to WHY you set out to achieve it in the first place.

MANAGING STRICTLY BY NUMBERS IS LIKE PAINTING BY NUMBERS Some things that you want to encourage will be quantifiable, and some will not. If you report on the quantitative goals and ignore the qualitative ones, you won’t get the qualitative goals, which may be the most important ones. Management purely by numbers is sort of like painting by numbers — it’s strictly for amateurs. At HP, the company wanted high earnings now and in the future. By focusing entirely on the numbers, HP got them now by sacrificing the future. Note that there were many numbers as well as more qualitative goals that would have helped: Was our competitive win rate increasing or declining? Was customer satisfaction rising or falling? What did our own engineers think of the products? By managing the organization as though it were a black box, some divisions at HP optimized the present at the expense of their downstream competitiveness. The company rewarded managers for achieving short-term objectives in a manner that was bad for the company. It would have been better to take into account the white box. The white box goes beyond the numbers and gets into how the organization produced the numbers. It penalizes managers who sacrifice the future for the short term and rewards those who invest in the future even if that investment cannot be easily measured. CLOSING THOUGHT It is easy to see that there are many ways for leaders to be misinterpreted. To get things right, you must recognize that anything you measure automatically creates a set of employee behaviors. Once you determine the result you want, you need to test the description of the result against the employee behaviors that the description will likely create. Otherwise, the side-effect behaviors may be worse than the situation you were trying to fix.