Sir Arthur Eddington, an English astrophysicist, told a short story involving a scientist studying fish by pulling them up with nets. After checking all the fish hauled up, the scientist concludes that there is a minimum size of fish in the sea. But the fish seen were determined by the size of the holes in the net, the smaller ones having slipped through, unmeasurable. The instrument you use affects what you see. Or as Richard Hamming puts it: "You get what you measure."

This analogy provides a nice concrete example of a phenomena that affects us routinely in more subtle ways. What and how we choose to measure affects the conclusions we draw. So, a website may easily measure sales and bounce rate for its pages, while things like trust, authority or satisfaction, which may be more significant longer-term metrics, go unmeasured.

"There is always a tendency to grab the hard, firm measurement, though it may be quite irrelevant as compared to the soft one which in the long run may be much more relevant to your goals. Accuracy of measurement tends to get confused with relevance of measurement, much more than most people believe. That a measurement is accurate, reproducible, and easy to make does not mean it should be done, instead a much poorer one which is more closely related to your goals may be much more preferable. For example, in school it is easy to measure training and hard to measure education, and hence you tend to see on final exams an emphasis on the training part and a great neglect of the education part."

What gets measured gets better (an earlier take on this)

Quote from Chapter 29, You get what you measure, of Richard Hamming's, The Art of Doing Science and Engineering: Learning to Learn.

If you happen to be in the business of doing research or science Richard Hamming's Bell Lab's talk, You and Your Research (pdf), is an excellent read.

You get what you measure

There are lots of questions you can ask of data. I like this model as it gives a framework for the different types of questions you might ask. It was introduced to me as the data science pyramid with descriptive analytics — measuring what’s happening — being the foundation on which others are built. You don’t have to go in order, but it’s a nice approach to consider how the questions you’re asking of your data might evolve.

To take our school Summer fête as an example, for a long time takings were all in cash and put into one bucket at the end for counting. You end up with one number for how much you raised. But you might move to collecting more descriptive data on the takings for each stall — whether the bar earns more than the tombola, for example. You then might be able to ask diagnostic questions like figuring out why one fête earned more than another with a different mix of stalls. If you know how many people came you might be able to make some predictions on the takings of a future fête based on number of people and stall types. And if you get fancy you might start to use your data to figure out how you could raise more money for the school in future.

There are many ways to slice and dice data and the questions you ask, but I like the clarifying simplicity of this one.

As far as I know, the framework is the Analytics Ascendancy model, or Analytics Value Escalator, or other such business sounding name from Gartner.

Analytics maturity

Pitched as the one number you need to grow your business, the Net Promoter Score (NPS) is a way to find out from your customers how many love what you’re doing enough to tell their friends vs how many are ambivalent or negative about your business. The resulting number, your NPS, ranging from +100 to -100 has been shown to be an indicator of companies’ future success.

To calculate your NPS ask your customers how likely they would be to recommend your business to a friend—or variations thereof—and give them a scale of 0 (not likely) to 10 (very likely) to answer with. Those who answer 9 or 10 are your biggest fans or Promoters. Those who answer 7 or 8 are passively satisfied—they like your business but are unlikely to shout about it. And those answering 6 or below are your Detractors. If they talk about your business it could well be negative. Your NPS is your % of Promoters minus the % of your Detractors. For example, if you received 100 responses, 35 as 9s or 10s, 37 as 7s and 8s, and 28 as 0–6s then your NPS would be 35 minus 28 = 7.

Keep track of your NPS over time and you’ll get a sense of how your business is doing. Follow up with qualitative feedback, for instance, “Why did you give this rating?”, and get a recipe for improving your business. Simple.

Learn more in the original Harvard Business Review article, “The One Number You Need to Grow” by Frederick F. Reichheld.

Net Promoter Score

Campbell’s law, rather pessimistically suggests:

"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."

Campbell's law is kind of a Goodhart’s law with more explanation.

It pretty much seems to work with everything, unfortunately. If you measure the performance of an employment office by the number of cases handled you may get quick but ineffective placements and interviews. If, instead, you measure them on number of persons placed in jobs you may unintentionally drive a focus on placing the simplest cases at the expense of those who most need the service.

When customer support is measured by number of tickets dealt with I’ve heard of writing quick answers that don’t fully solve a question in the knowledge that the customer will soon come back with another ticket to solve.

When policing effectiveness is judged by crimes reported and the number of unsolved cases you may get both under reporting of crimes, and a form of plea bargaining where a criminal may be encouraged to confess to extra crimes — therefore fewer unsolved cases — in exchange for more lenient sentences. Campbell notes that:

It seems to be well documented that a well-publicized, deliberate effort at social change–-Nixon’s crackdown on crime–-had as its main effect the corruption of crime-rate indicators, achieved through underrecording and by downgrading the crimes to less serious classifications.

Measuring schools on attendance encourages creative reasons for children being taken out of school. Assessing learning can quickly become teaching to, and cramming for, the test. When reviews come to dominate purchase decisions pressure increases for fake reviews.

And then there’s elections and voting…

Campbell, Donald, Assessing the Impact of Planned Social Change, Dec 1976. (pdf)

Also see: what gets measured gets better and Seth Godin’s take on Campbell’s law: Speedometer confusion

Scott Carter gave a pithier paraphrasing of Campbell’s law: “What gets measured, gets corrupted.”

Campbell’s law

The story goes something like this. Back in colonial India the top Brit in charge decided there were too many cobras around Delhi. To reduce the population they put in place a cash reward, or bounty, for anyone who brought in a dead cobra. The intention was clear. Legend has it that people did bring in the cobras reliably because some enterprising souls had started breeding cobras for the very purpose of getting the bounty. When the authorities realised this they scrapped the scheme, the cobra farms closed and the bred cobras were released into the wild significantly increasing the cobra population by a few orders of magnitude. Hence, the cobra effect: when a well-intentioned measure can have the opposite effect to that desired.

It’s not clear how true this story is, but the cobra effect has been demonstrated many times. The Freakonomics podcast on the cobra effect discusses some excellent examples (with references), including:

A bounty on rats in French colonial Hanoi — where the bounty was earned by bringing in a rat tail — which went the same way as the cobra story

Removing wild pigs at Fort Benning, Georgia, at $40 a pigs tail that didn’t make a dent. It could be the hunters would bait the pigs with all sorts of tempting scraps. Many pigs learned to avoid the hunters and instead bred like crazy while well-fed on scraps.

The capital of Colombia, Bogotá, tried to reduce traffic congestion by restricting each car from driving one day during the week. The result: people started buying more than 1 car and did more driving on weekends.

Publication of proposed areas of habitat to be protected for endangered species meant land developers responded by building in those areas before the land was protected.

People are smart. Incentives work—in the sense that you tend to get what you incentivise. So be careful what you are incentivising. As Charlie Munger said: "Show me the incentive, and I'll show you the outcome."

The cobra effect

Goodhart's law states that when a measure becomes a target, it ceases to be a good measure. In other words, if you pick a measure to assess performance, people find a way to game it.

To illustrate, I like the (probably apocryphal) story of a nail factory that sets "Number of nails produced" as its measure of productivity. The workers figure out they can easily make tons of tiny nails to hit the target.

Yet, when the frustrated managers switch the assessment to "weight of nails made", the workers again outfox them by making a few giant heavy nails.

And there's the story of measuring fitness by steps from a pedometer only to find the pedometer attached to the dog.

Some strategies for helping this are to try and find better, harder-to-game measures, assess with multiple measures, or allow a little discretion. More detail in this nice little article.

I also liked an idea I read in Measure What Matters of pairing a quantity measure with a quality measure, for example, assessing both the number of nails and customer satisfaction of the nails.

How strongly Goodhart's Law applies varies. John Cutler shared the Cutler Variation of Goodhart's Law:

"In environments with high psychological safety, trust, and an appreciation for complex sociotechnical systems, when a measure becomes a target, it can remain a good measure because missing the target is treated as a valuable signal for continuous improvement rather than failure."

Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure.

Tricky topics - validity trickier than reliability. At it’s core I believe reliability is about getting the same results given the same conditions and validity is about measuring what you intend to measure. Harder than it sounds.

Understanding reliability and validity

In both a “great! we’re improving kind of way,” and also a “teaching to the test” kind of way. From Richard Hamming’s, The Art of Doing Science and Engineering.

Metrics sketches