Chihuahua syndrome refers to messy data from variations in spelling or input—Chihuahua is easy to misspell. The quality of your data matters—errors can creep in anywhere, particularly when people enter data. Garbage in, garbage out.

Here's Chris Groskopf quoted in Seeing with Fresh Eyes—Meaning, Space, Data, Truth by Edward Tufte:

"There is no worse way to screw up data than to let a single human type it in, without validation. I acquired a complete dog licensing database. Instead of requiring people registering their dog to choose a breed from a list, the system gave dog owners a text field to type into, so this database had 250 spellings of Chihuahua. Even the best tools can't save messy data. Beware of human-entered data."

Capitals, spaces, misspellings, hyphens, numbers stored as text, numbers entered as letters (I, O), accents, straight/curly apostrophes, dates out of order, languages, dialects, abbreviations, and more are all routes for misleading your analysis.

The name The chihuahua syndrome is from Edward Tufte.

Adding these terms below to help anyone searching for the sketch ;o)

che hua hua, che-hua-hua, che-wara, chee-hua-hua, cheehuahua, chehuahua, chehaha, chehawa, chewawa, chihaha, chihuahua, chihuawa, chiwawa, chiehuaua...

Chihuahua syndrome

So does the willingness to seek out data. If none of us has data to share then we can't help but bring our own personal biases, preferences, and opinions to the table — anyone might be right, but who's to say (often the HiPPO: the Highest Paid Person's Opinion)? It's why I love this quote from W. Edwards Deming:

"Without data you're just another person with an opinion."

Perhaps you've been in a car with several backseat drivers who all seem to know better than the driver. Or maybe you've been part of a review for a new initiative where everyone has their own views of what will make it succeed. Or a discussion on what's driving some unexpected behaviour. We can't know who's right until we bring some data. It's amazing how even a little data can avoid an "I'm right, you're wrong" situation.

And where there isn't data, framing a discussion on how to quickly gather data or experiment to find out which approach may be best so often focuses a team, removing ego and opinions.

After all, as Jim Barksdale, former Netscape CEO said:

"If we have data, let's look at data. If all we have are opinions, let's go with mine."

Without data you're just another person with an opinion

Metrics are like an onion: full of layers.

A simple question to ask can often be complicated to answer. Or we may find that we need to ask a different question. While it'd be lovely to have one simple metric to understand our business the reality is usually a lot more nuanced and interesting than it first seems.

Averages will deceive. Vanity metrics may wow —  by, say, quoting a total user count and conveniently ignoring whether those users are actually paid, active, or leaving. Growth overall can hide declines in other areas or product lines and vice versa. The people on monthly subscriptions may be doing great, while revenue from add-ons may be declining. New users may be signing up in droves while existing users are churning. Mobile traffic may be different from desktop, online sales different from store sales... In practice, it's the hard questions we're usually asking even if we don't know how to ask them.

It pays to carefully consider what you're trying to learn and to understand all aspects of a business before putting faith in the numbers. And it may make sense to start with simpler metrics first.

The metrics onion

There's an old story about the statistician who drowned after seeing that the average depth was 3ft. Averages, or in this case the mean, necessarily hide some data, but very often they also hide what's really going on.

Let's say you run a delivery service and have an average delivery time of one day, it could be that most deliveries are actually a few hours while a few particular ones have people waiting for a week. Or a figure of average incomes might hide that most people have low salaries and a few are millionaires. But just looking at the average you can't tell. Sometimes a different measure of central tendency, like the median, can provide a clearer picture.

Sheldon Zedeck, a UC Berkeley psychology professor who taught me about the design of experiments, gave wise advice: spend time with your data. Sometimes it's the only way to know what's happening for sure.

Sneaky averages

When we notice two things change with each other — sunny days and ice cream sales, or player height and goals scored — it's natural to think that one thing is the cause of the other. And it's only a small step to consider that if you do more of one, you'll get more of the other. But this instinct and our natural sense-making behaviours can easily lead us astray. Just because two things change together — are correlated — doesn't mean that one causes the other: correlation is not causation.

You might reasonably notice a correlation between the number of children in a household and the number of storks nests on the roof of houses in Denmark or Holland. Fortunately, while the two may be correlated it's not hard to see that it's most likely not the storks that are causing the larger families and instead another factor such as the size of the house that is the more likely cause. As such it's a handy reminder that correlation is not causation.

This is also the subject of my favourite xkcd on correlation, and another on cell phones to drive the point home.

For more see the excellent spurious correlations.

The stork nest example is from Darrel Huff's How to Lie with Statistics, as is the example on Sampling Bias.

Correlation is not causation

Sankey diagrams are handy ways to visualise material or energy flows through usage and time. They are excellent for displaying energy source and usage, for example the UK government publishes a Sankey diagram of UK energy each year (pdf).

Perhaps the most famous example is Charles Joseph Minard's remarkable visual showing Napoleon's invasion of and retreat from Russia in the Winter of 1812–13. In that chilling example the thickness of the lines represent the remaining size of Napoleon's army.

I learned about Sankey diagrams from Edward Tufte but perhaps people see them most commonly today in Google Analytics showing the different paths visitors take on a website before leaving.

Sankey diagram

Lagging metrics are usually the ones we care about most. In business, they're things like revenue, sales, or profit. The trouble is, we can't act always act on those directly. By the time we assess our revenue everything we did to get there has already happened, so if we don't like it, it's too late.

Leading metrics are what we need to focus on to drive the lagging metrics. If we want to lose weight we might be better off focusing on eating less and eating healthier, working out more, and sticking to our workout schedule. To increase our revenue we may want to look at deal size, ease of sign-up, or our referral rate.

Paying attention to and improving the leading metrics gives us leverage over the lagging metrics we may really care about.

Leading and lagging metrics

If you wanted to know who would win an election, then the only reliable (most of the time) way to do that is to ask everyone to vote for real and count them all. Short of doing that, if you want to get an idea who would win you can pick a group that you hope is representative of the whole of the voting population, ask them what they plan to vote instead, and then multiply up to the full group. But it turns out, whenever you ask people things, it's jolly hard to really have confidence in what you're extrapolating up to as all sorts of bias can creep into your sample.

If you ask people what they plan to vote they may lie, they may tell you what they plan to do then change their mind, they may tell you what they think you want to hear depending on who's asking, they may say the first option because it's easiest, and you may ask in the places where it's easy or cheap to do so, or hear from only the people who are happy and not too busy to talk with you.

In the example in the sketch, if you hope to receive responses to your questions by surveys then you get responses from people who happen to not mind doing surveys — from anyone who throws them in the bin you won't receive a single data point.

All of these, and others, can lead to a bias in your sample and, at times, to drawing wholly inaccurate conclusions.

Extreme example from How to Lie with Statistics by Darrell Huff.

Sampling bias

Where is the Life we have lost in living?
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?

In these short lines he touched on an intriguing differentiation between wisdom, knowledge and information. Add data at the beginning and people have proposed various models such as a hierarchy, chain, or pyramid to help understand their relationships.

It's perhaps like a skier who's experienced the data points of snow and conditions in the mountains most days of their life. They begin to observe predictable patterns and then gradually distill these into knowledge of the relationships between the weather and avalanche risk. And over a lifetime of building and applying such knowledge they may develop instincts and behaviours for predicting risk and living with the mountains that may even be called wisdom.

In an unlikely scenario, I once wrote a class paper, that somehow people still find to read, on The Data, Information, Knowledge, Wisdom Chain: The Metaphorical link (pdf). It touches on some of the subtle differences of how we think about each of these, such as how I might complain of information overload, but never of knowledge overload.

Data Information Knowledge Wisdom

Sir Arthur Eddington, an English astrophysicist, told a short story involving a scientist studying fish by pulling them up with nets. After checking all the fish hauled up, the scientist concludes that there is a minimum size of fish in the sea. But the fish seen were determined by the size of the holes in the net, the smaller ones having slipped through, unmeasurable. The instrument you use affects what you see. Or as Richard Hamming puts it: "You get what you measure."

This analogy provides a nice concrete example of a phenomena that affects us routinely in more subtle ways. What and how we choose to measure affects the conclusions we draw. So, a website may easily measure sales and bounce rate for its pages, while things like trust, authority or satisfaction, which may be more significant longer-term metrics, go unmeasured.

"There is always a tendency to grab the hard, firm measurement, though it may be quite irrelevant as compared to the soft one which in the long run may be much more relevant to your goals. Accuracy of measurement tends to get confused with relevance of measurement, much more than most people believe. That a measurement is accurate, reproducible, and easy to make does not mean it should be done, instead a much poorer one which is more closely related to your goals may be much more preferable. For example, in school it is easy to measure training and hard to measure education, and hence you tend to see on final exams an emphasis on the training part and a great neglect of the education part."

What gets measured gets better (an earlier take on this)

Quote from Chapter 29, You get what you measure, of Richard Hamming's, The Art of Doing Science and Engineering: Learning to Learn.

If you happen to be in the business of doing research or science Richard Hamming's Bell Lab's talk, You and Your Research (pdf), is an excellent read.

You get what you measure

There are lots of questions you can ask of data. I like this model as it gives a framework for the different types of questions you might ask. It was introduced to me as the data science pyramid with descriptive analytics — measuring what’s happening — being the foundation on which others are built. You don’t have to go in order, but it’s a nice approach to consider how the questions you’re asking of your data might evolve.

To take our school Summer fête as an example, for a long time takings were all in cash and put into one bucket at the end for counting. You end up with one number for how much you raised. But you might move to collecting more descriptive data on the takings for each stall — whether the bar earns more than the tombola, for example. You then might be able to ask diagnostic questions like figuring out why one fête earned more than another with a different mix of stalls. If you know how many people came you might be able to make some predictions on the takings of a future fête based on number of people and stall types. And if you get fancy you might start to use your data to figure out how you could raise more money for the school in future.

There are many ways to slice and dice data and the questions you ask, but I like the clarifying simplicity of this one.

As far as I know, the framework is the Analytics Ascendancy model, or Analytics Value Escalator, or other such business sounding name from Gartner.

Analytics maturity

Pitched as the one number you need to grow your business, the Net Promoter Score (NPS) is a way to find out from your customers how many love what you’re doing enough to tell their friends vs how many are ambivalent or negative about your business. The resulting number, your NPS, ranging from +100 to -100 has been shown to be an indicator of companies’ future success.

To calculate your NPS ask your customers how likely they would be to recommend your business to a friend—or variations thereof—and give them a scale of 0 (not likely) to 10 (very likely) to answer with. Those who answer 9 or 10 are your biggest fans or Promoters. Those who answer 7 or 8 are passively satisfied—they like your business but are unlikely to shout about it. And those answering 6 or below are your Detractors. If they talk about your business it could well be negative. Your NPS is your % of Promoters minus the % of your Detractors. For example, if you received 100 responses, 35 as 9s or 10s, 37 as 7s and 8s, and 28 as 0–6s then your NPS would be 35 minus 28 = 7.

Keep track of your NPS over time and you’ll get a sense of how your business is doing. Follow up with qualitative feedback, for instance, “Why did you give this rating?”, and get a recipe for improving your business. Simple.

Learn more in the original Harvard Business Review article, “The One Number You Need to Grow” by Frederick F. Reichheld.

Net Promoter Score

The data prison is the default table where the bars of the cells crowd out your data with non-data-ink and lock your data into a prison.

When data is locked into cells, their relationships with other data are harder to see, and the reader has to work harder to learn from your table.

When making a table of data, consider if, by adjusting spacing and alignment, you can make the columns or rows clear enough as they are without the prison bars. And look for opportunities to remove the visual noise to let your data and the relationships between them shine out.

The data prison is a concept from Edward Tufte. Check out the beautiful Envisioning Information

The data prison

I grew up with the idea that there are developed countries and developing countries. There was the ‘rich’ world and the ‘poor’ world. I didn’t actively consider it. It’s just always been there. And it’s wrong.

In his excellent book Factfulness, Hans Rosling shows that wealthier countries tend to have smaller families and a low child mortality rate. In contrast, poorer countries are more likely to have larger families and a high child mortality rate. Plotting the spread of countries in 1965 gives a reasonable approximation of a cluster of ‘developed’ countries with small families where most children survive and a cluster of ‘developing’ countries with larger families where more children die. But that was in 1965.

The world has changed a lot since then, and that model of dividing the world into two buckets no longer holds true with the data. Most people in the world are somewhere in the middle.

And more helpful than dismantling my worldview was providing a new model to replace it: a model with 4 income levels instead of 2 and the number of billions of people living at each level. It’s a model with most people somewhere in the middle.

For more, check out the Factfulness book, or Gapminder. And you could do much worse than watch Hans’ entertaining TED talk: the best stats you’ve ever seen.

A world of 4 income levels

Goodhart's law states that when a measure becomes a target, it ceases to be a good measure. In other words, if you pick a measure to assess performance, people find a way to game it.

To illustrate, I like the (probably apocryphal) story of a nail factory that sets "Number of nails produced" as its measure of productivity. The workers figure out they can easily make tons of tiny nails to hit the target.

Yet, when the frustrated managers switch the assessment to "weight of nails made", the workers again outfox them by making a few giant heavy nails.

And there's the story of measuring fitness by steps from a pedometer only to find the pedometer attached to the dog.

Some strategies for helping this are to try and find better, harder-to-game measures, assess with multiple measures, or allow a little discretion. More detail in this nice little article.

I also liked an idea I read in Measure What Matters of pairing a quantity measure with a quality measure, for example, assessing both the number of nails and customer satisfaction of the nails.

How strongly Goodhart's Law applies varies. John Cutler shared the Cutler Variation of Goodhart's Law:

"In environments with high psychological safety, trust, and an appreciation for complex sociotechnical systems, when a measure becomes a target, it can remain a good measure because missing the target is treated as a valuable signal for continuous improvement rather than failure."

Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure.

The half-life of caffeine is about 4 hours.

The data vary, but for a medium-sized adult, you can probably expect half of the caffeine from that 3 pm cup of coffee to still be doing half of its thing about 7 pm, 4 hours later. Apparently, smokers deal with caffeine quicker, and caffeine sticks around in children much longer.

You can expect about 10 minutes before it first kicks in, which, I suppose, is why if you’re feeling tired while driving, it’s a decent plan to take a break and drink a coffee, then have a 15-minute nap. Once it starts to kick in, you might hit peak caffeine any time from 45 minutes to 2 hours later.

Caveat: The chart visualisation is extrapolated from reported data, so the shape is only approximate.

Some references say longer, and some say from 3-9 hours or so. My source, which seemed pretty comprehensive, was:

Bertil B. Fredholm, Karl Bättig, Janet Holmén, Astrid Nehlig and Edwin E. Zvartau, Actions of Caffeine in the Brain with Special Reference to Factors That Contribute to Its Widespread Use, Pharmacological Reviews March 1999, 51 (1) 83-133;

The half-life of caffeine

There’s a famous photo series by Eadweard Muybridge that shows, unequivocally, that while galloping all four of a horse’s legs leave the ground. I’d always heard that’s what they trying to find out at the time, but from what I’ve read it was more a question of at what points do a horse’s legs leave the ground. The photos show the somewhat surprising state of leaving the ground while the horse’s legs are tucked, not when they are outstretched. It’s a lovely example of what Edward Tufte calls small multiples.

Do horses legs leave the ground at a gallop?

Best travel choices in London

Data graphics can lie in many ways, but the most common, I expect, is the error of using area or volume objects to represent one-dimensional data.

More data visualisation principles from Edward Tufte.

Watch for the lie factor

From the video Wealth Inequality in America

Wealth inequality in America

Principles from Edward Tufte. Remove chart junk. Erase redundant and non-data-ink. Above all else show the data.

Of course the present infographic trend laughs in the face of these principles in the interests of trying to encourage people to actually look at the data in the first place. But, assuming you actually want people to seriously infer some meaning from the data, these are not bad principles to follow when constructing your data graphics. Some basic Excel templates could do with a clean up for example.

Maximise data-ink

Tricky topics - validity trickier than reliability. At it’s core I believe reliability is about getting the same results given the same conditions and validity is about measuring what you intend to measure. Harder than it sounds.

Understanding reliability and validity

In both a “great! we’re improving kind of way,” and also a “teaching to the test” kind of way. From Richard Hamming’s, The Art of Doing Science and Engineering.

Data sketches