Metrics are like an onion: full of layers.

A simple question to ask can often be complicated to answer. Or we may find that we need to ask a different question. While it'd be lovely to have one simple metric to understand our business the reality is usually a lot more nuanced and interesting than it first seems.

Averages will deceive. Vanity metrics may wow —  by, say, quoting a total user count and conveniently ignoring whether those users are actually paid, active, or leaving. Growth overall can hide declines in other areas or product lines and vice versa. The people on monthly subscriptions may be doing great, while revenue from add-ons may be declining. New users may be signing up in droves while existing users are churning. Mobile traffic may be different from desktop, online sales different from store sales... In practice, it's the hard questions we're usually asking even if we don't know how to ask them.

It pays to carefully consider what you're trying to learn and to understand all aspects of a business before putting faith in the numbers. And it may make sense to start with simpler metrics first.

The metrics onion

There's an old story about the statistician who drowned after seeing that the average depth was 3ft. Averages, or in this case the mean, necessarily hide some data, but very often they also hide what's really going on.

Let's say you run a delivery service and have an average delivery time of one day, it could be that most deliveries are actually a few hours while a few particular ones have people waiting for a week. Or a figure of average incomes might hide that most people have low salaries and a few are millionaires. But just looking at the average you can't tell. Sometimes a different measure of central tendency, like the median, can provide a clearer picture.

Sheldon Zedeck, a UC Berkeley psychology professor who taught me about the design of experiments, gave wise advice: spend time with your data. Sometimes it's the only way to know what's happening for sure.

Sneaky averages

When we notice two things change with each other — sunny days and ice cream sales, or player height and goals scored — it's natural to think that one thing is the cause of the other. And it's only a small step to consider that if you do more of one, you'll get more of the other. But this instinct and our natural sense-making behaviours can easily lead us astray. Just because two things change together — are correlated — doesn't mean that one causes the other: correlation is not causation.

You might reasonably notice a correlation between the number of children in a household and the number of storks nests on the roof of houses in Denmark or Holland. Fortunately, while the two may be correlated it's not hard to see that it's most likely not the storks that are causing the larger families and instead another factor such as the size of the house that is the more likely cause. As such it's a handy reminder that correlation is not causation.

This is also the subject of my favourite xkcd on correlation, and another on cell phones to drive the point home.

For more see the excellent spurious correlations.

The stork nest example is from Darrel Huff's How to Lie with Statistics, as is the example on Sampling Bias.

Correlation is not causation

If you wanted to know who would win an election, then the only reliable (most of the time) way to do that is to ask everyone to vote for real and count them all. Short of doing that, if you want to get an idea who would win you can pick a group that you hope is representative of the whole of the voting population, ask them what they plan to vote instead, and then multiply up to the full group. But it turns out, whenever you ask people things, it's jolly hard to really have confidence in what you're extrapolating up to as all sorts of bias can creep into your sample.

If you ask people what they plan to vote they may lie, they may tell you what they plan to do then change their mind, they may tell you what they think you want to hear depending on who's asking, they may say the first option because it's easiest, and you may ask in the places where it's easy or cheap to do so, or hear from only the people who are happy and not too busy to talk with you.

In the example in the sketch, if you hope to receive responses to your questions by surveys then you get responses from people who happen to not mind doing surveys — from anyone who throws them in the bin you won't receive a single data point.

All of these, and others, can lead to a bias in your sample and, at times, to drawing wholly inaccurate conclusions.

Extreme example from How to Lie with Statistics by Darrell Huff.

Sampling bias

I grew up with the idea that there are developed countries and developing countries. There was the ‘rich’ world and the ‘poor’ world. I didn’t actively consider it. It’s just always been there. And it’s wrong.

In his excellent book Factfulness, Hans Rosling shows that wealthier countries tend to have smaller families and a low child mortality rate. In contrast, poorer countries are more likely to have larger families and a high child mortality rate. Plotting the spread of countries in 1965 gives a reasonable approximation of a cluster of ‘developed’ countries with small families where most children survive and a cluster of ‘developing’ countries with larger families where more children die. But that was in 1965.

The world has changed a lot since then, and that model of dividing the world into two buckets no longer holds true with the data. Most people in the world are somewhere in the middle.

And more helpful than dismantling my worldview was providing a new model to replace it: a model with 4 income levels instead of 2 and the number of billions of people living at each level. It’s a model with most people somewhere in the middle.

For more, check out the Factfulness book, or Gapminder. And you could do much worse than watch Hans’ entertaining TED talk: the best stats you’ve ever seen.

A world of 4 income levels

In my everyday life I’d have been hard pushed to put my finger on the difference between accuracy and precision but, in engineering at least, you learn that there is a clear difference between the two.

Precision describes how variable an outcome is, or how close repeated measurements or actions are to each other.

In contrast, something may be very precise but not be accurate to your intent or to the true value of a thing. That’s accuracy — how close you are to the true or intended value. If you’re hitting near your intended target then you’re accurate.

Hmm…one of those things where a sketchplanation works a whole lot better than words.

For the more spiritually inclined: Everything is aiming.

Or for the more statistically inclined: Measures of central tendency (mean, median, mode), and Common distributions (normal, skewed, pareto).

That accuracy is true to intention and precision true to itself I learned from Simon Winchester in Exactly: How precision engineers created the Modern World.

Accuracy and precision

Very easy to forget the difference between the mean, the median and the mode.

Median: the middle value of a list in ranked order.

Measures of central tendency: mean, median, mode

Normal (the weight of apples, measurement errors, IQ…), skewed (Number of children in a household, age at retirement…) and Pareto (80/20 rules, long-tails, size of meteorites).

Sketchplanations tagged with statistics