Episode 82: Statistics Don't Lie

Announcer:
This is The WealthAbility® Show with Tom Wheelwright. Way more money, way less taxes.

 

Tom Wheelwright:

Welcome to The WealthAbility® Show where we're always discovering how to make way more money and pay way less taxes. This is Tom Wheelwright, your host, founder, and CEO of the WealthAbility show.

So statistics in today's world are like, “What in the world do I believe?” China gives us statistics and we go, “Do I believe those?” We have Russia. They give us statistics. “Do I believe those?” And then we have our own country, wherever we are, and we go, “Do I even believe the statistics I'm getting? And should I believe them?”, when it seems like everybody's got their own set of statistics to prove their own point.

I know this is not new. It's just in the news right now, and we have a very special guest. Our guest is Tim Harford, who is The Data Detective, and really the expert on explaining how statistics work and how we interpret statistics.

So I'm very excited to have you on the show, Tim, and if you don't mind, just give us a 30 to 60 second background, what you've been doing, and why you do what you do.

Tim Harford:

Sure. I mean, I'm happy to give you the full one hour version of my bio, if you like, but who has time for that? So yeah, my name is Tim Harford. My most recent book is The Data Detective: Ten Easy Rules Rules to Make Sense of Statistics. I'm also known for a book I wrote a few years ago, called the Undercover Economist. I write a column for the Financial Times, called The Undercover Economist, all about economic ideas in everyday life.

And I also present various BBC radio shows. One is about numbers. It's called More or Less. One is about vaccinations. It's called How to Vaccinate the World. And I present a podcast, called Cautionary Tales, made by Malcolm Gladwell's company, Pushkin, and that's all about things going wrong and the lessons that we can learn.

Tom Wheelwright:

Oh, I like that. So there's an old accounting joke, Tim. It says, “You ask an accountant what's two plus two?”, and the answer is, “Well, what would you like it to be?” So that seems to be the way that statistics work, because it seems to me, I mean, in our country, the Republicans use statistics to prove that it was, they were using it to say, “Well, look, statistics show that it's improbable that there wasn't fraud.” Right? I mean, they were using those statistics showing that, “Wow, there's probably fraud here, so we've got a problem with the election.” And the Democrats, they're using their own statistics for their own reasons, and we always seems to use statistics for our own reasons from a macro level.

So how do you make sense of it? How do you actually get through the statistics and say, “What do I believe? What don't I believe? Or how do I use that?”

Tim Harford:

It's a great question. I mean, I think the first thing to point out is that we tend to do a lot better in thinking clearly about statistics when it's not in an atmosphere of political polarization. But what we believe and what we disbelieve is really overwhelmingly governed by what our friends think, what our political leaders think, what the media sources that we're following tell us, by our preconceptions, by what we expect.

And you get to this extraordinary situation, where you have a smart Democrat, who pays attention and is curious and is thinking, comes to a completely different conclusion to a smart Republican, who pays attention and studies the sources and is thinking, and that just goes to show how overwhelmingly our emotions and our preconceptions play a role.

So in the book, The Data Detective, the very first piece of advice I give to people is, 10 pieces of advice, and you might think I'd be straight in there with, “Oh, check your sources,” or, “Correlation is not causation,” or some really complex technical argument. But actually, my very first piece of advice is, “Notice your emotional reaction.”

You're looking at a graph, you're reading a social media post, there's a media headline, these things are designed to produce some kind of reaction, fear or triumph or disgust or anger, something. So just notice that, and after you've noticed it, after you've gone, “Oh. Oh, I seem to be a bit triggered by that,” then go back and have another look, and immediately, I think you'll be thinking more clearly.

Tom Wheelwright:

Oh, I like that. I always like to say that opinion is where knowledge stops and learning stops. Right? Once we get an opinion, then all of a sudden, it doesn't matter what statistics are, doesn't matter what anybody else says, because that's our opinion. So I love that, that test that you're using there, look at your emotions, because emotions really tell you there's an opinion in there somewhere that is driving this, as opposed to the actual data. So, okay. So once you've done that, what are you looking at?

Tim Harford:

So the next thing that I would suggest is, you want to get the context, and by context, there's all kinds of questions you could be asking. So that could be things like, “Well, who's telling me this? And why are they telling it to me?” But it can also be sometimes really simple questions like, “Is that a big number? Or is that a small number? How can I compare this to historical precedent? Is it going up? Or is it going down? Also, what exactly is being measured?”

So very often, when you hear a statistic, you could do what I call premature enumeration. So you're straight in there, slicing and dicing and taking averages and plotting trendlines and such, and you're not taking a step back and going, “What are they actually measuring here? What are they actually counting here?”

So you think back to the financial crisis, 2008. There are lots of different reasons why that happened, but I think one of the reasons is that you had complex models that were supposed to be making very finely tuned bets about risk. Well, when we say risk, what do we actually mean by risk? And where are we getting our measure of risk? And by not asking that question seriously enough, you have amazing mathematical models that produce junk answers. So all of these things, I would put under the label of context.

Tom Wheelwright:

Well, to me, it's a little like under COVID, and it depends on how are you measuring that? Right? I mean, I'm an accountant, so I am always looking at garbage in, garbage out. So if I don't have, like you say, what am I measuring? And if I'm looking at, for example, infections, or even deaths from COVID, for example, what constitutes a COVID death? Right? Because, I think that it's been clear that different countries are taking a different approach on that. Is it solely COVID that caused the death? Or was it really that COVID complicated some other disease that I had?

And the U.S., for example, is taking a very broad approach on that. So anytime anybody had COVID and they died, that was a COVID death. And there's an argument on both sides of that, but the reality is, the tough part is then how do we compare that? And what I come back to is, is I like to look at statistics as a movie instead of a photograph. So to me, if I'm seeing the story as it moves along, that's much more interesting to me than if I look at a point in time. So is that a fair analysis, my question for you?

Tim Harford:

No, I think it is. I think it is. I think one can take it too far, because all of the key points I'm making in my book is that we are often invited to be excessively skeptical of statistics and go, “Oh, it's all just junk. It's all fake news.” Whereas actually, when you look at it, a lot of the most important discoveries in human history could only have been made through the use of statistics.

So we can't get into this situation where we just go, “Oh yeah, that can't be right,” so you've got to take them seriously. You've got to look at what you can learn from them. But you're absolutely right. So a COVID death can mean different things, and different countries use different definitions.

So I've looked at this quite carefully for my own work. And so, if you look, for example, at the U.K., we have, by some measures, the worst COVID mortality of any country anywhere in the world, with the possible exception of Belgium.

So then let's look at that a little bit more closely. Is that really true? And the answer is, it's not not true. It's not far off, but it's not exactly true. Belgium, for example, has a more expansive definition of COVID deaths than the U.K. So they were trying very hard, especially in the first wave, to make sure anything that could possibly have been considered a COVID death was included, and they paid the price for that in terms of a very high death toll. But if you adjust for that, if you recalibrate, do they still have a very high death toll? Well, yes, they did.

But what about countries that are under-counting? So one example I would say is Russia. Russia, it needs a postmortem to say, “Yes, this was a COVID death.” Now, I don't know what the pathologists and the coroners are saying in Russia, but I think it's quite possible that they're underplaying it.

Or another example, Peru. If you look at just excess deaths, where we have good data on this, like just a load of people dying and we don't know why, way more than usual. Well, they're very, very high in Peru. There aren't very many official COVID deaths. Now, the most plausible explanation for that, there are different explanations, but the most plausible explanation is, it's probably COVID and it's probably undiagnosed COVID.

So when I look at the statistics in the U.S., it's clear that a lot of the people who are being registered as COVID deaths, a lot of them had other conditions, a lot of them were very elderly, but I think the fairest view of that is to say, “They would not have died at that point, if it had not been for COVID.”

So very often, the doctors will say, “Oh yeah, they died of COVID plus heart failure,” or, “COVID plus respiratory failure.” And then you go, “Well, why was their respiratory failure?” And the answer is, “Well, because of COVID.” So COVID definitely killed them, but you would also say, actually, I mean, in the U.K., the average age of the people who've died is about 83.

So it's this thing, when you're in a political argument and you're trying to make some case that says, “Oh, COVID is the worst thing that's ever happened,” or, “COVID is all just fake news. It's just nothing. It's just a plot,” when you're in those arguments, no one's getting any smarter.

But when you take it seriously and you look, you're able to hold different things in mind, then to say, “These COVID deaths are real, or they are mostly being caused by COVID, but at the same time, the people who are dying are also very elderly and they're highly vulnerable,” and both of those things can be true.

Tom Wheelwright:

Well, it's that. That's an interesting point, because they say that the hardest thing for any human to do is hold two opposing views at the same time and actually look at both opposing views, and I think that's a lot of what we're seeing right now.

Let's bring this down to a little bit of a micro level. I'm a micro guy. I love the macro stuff. I love to talk about it, but there's not a lot I can do about it.

Tim Harford:

Yeah, I'm a micro guy too. I'm with you.

Tom Wheelwright:

With the micro stuff, in my business and my investing, I can do something about it. And I actually find, when I look at numbers, I look at all numbers in the form of, they're really statistics, because unless it's an absolute number, like how many dollars do I have in the bank, how many pounds do I have in the bank, it's more of, “Okay. So what's going on? What's the pattern? And what do I need to do to improve that pattern or change that pattern or maintain that pattern?” And so, what are some things we can do statistically actually to use statistics in our investing and our business activities?

Tim Harford:

Oh, it's a great question. I'm smiling to myself, because just this morning, I was looking at an amazing history of an accountant, like you, but it's from near Florence in 1396, and it's just his letters, writing to his business associates, and the insults he uses, because they're getting confused about the numbers and they don't know what's going on. It's stuff like, “You could lose a crow in a bowl full of milk. What a psycho, or, “You could lose your way from your lows to your mouth.” I love this. So these classic 14th century Italian insults from an accountant, because his associates are just bungling the numbers, this was making me smile.

But so, what advice can I give? Well, one piece of advice that I would give is that you need to try to combine the statistics with your personal intuition and the wisdom comes when you've got both. So very often, the statistics are giving you the bird's eye view. They give you the overview, they show you everything that's going on. At the same time, it's often quite distant, it's quite thin. You miss a lot of detail.

Your personal experience, talking to a customer, talking to an employee, just walking around, looking at the shop floor, or just examining the details of a contract, you can learn a lot more. But at the same time, it's a very particular slice of what's going on. It's a biased slice, it's a narrow slice, so you need both the bird's eye view and the view up close. Combine the two, and you get wisdom.

The second thing I would say is, be really careful when you're using statistics as performance monitors of other people. That's when things get really tricky. I'm not saying don't ever do it. I'm not saying you can't do it, but it's dangerous. The reason it's dangerous is, because the statistics are always missing something, and if you're just trying to understand the world, that's fine. The statistics will give you a good sense of what's going on. But the moment you try to control someone else's behavior through statistical means, they'll find ways to wriggle out.

So an example I really like comes from the U.K., when the British government decided it wanted ambulances to show up faster, and it said, “Okay. What we want is, in emergency cases, we want ambulances there within eight minutes of the call, or rather we want a first responder there within eight minutes of the call.”

And so all kinds of things started happening. One thing was, it was amazing how many ambulances showed up at exactly seven minutes and 59 seconds. It was kind of like the dispatcher, because they're all trying to hit these performance targets, and the dispatcher's like, “Oh, it's nearly there. Stop the clock. We'll just say it's there.” So you just have people lying to you.

Another thing that happened is, they started going, “Well, hey, we could sell the ambulances. We could split up the ambulance crews and we could buy motorbikes, or even pedal cycles, and we could send out twice as many first responders on bikes instead of in ambulances,” and it really works. I mean, then you do get people there faster, and sometimes that's really what you want, like one expert paramedic with a bike there really fast, but it's not much use if you actually need to take someone to hospital.

Another thing that happened is, people would go, “Oh, okay. So this target only applies for critical calls. This case doesn't sound that critical actually,” so then people would start redefining cases. And worst of all, this is the thing, this is the real crime, you would blow through the target and it's, “Oh, the ambulance didn't get there within eight minutes,” so this person's now been waiting nine minutes with a heart attack or whatever.

This didn't happen that often, but there is evidence it sometimes happened. The dispatchers would go, “Okay. Well, who's been waiting five minutes? Because we'll reroute the ambulance to them.” So once people miss the eight minute slot, then it's like, well, you could be waiting an hour.

Tom Wheelwright:

That's one of the problems that I've seen, and I actually made this mistake in my business years ago, where we were using numbers for performance, and what it did was, it does change the performance. So people will perform to reach the numbers, and be careful what you ask for, because sometimes, I mean, what happened was, we're very much about how much are we billing and how much time are getting to the clients, and how much time are we wasting, all that kind of stuff, so very much an efficiency model.

And what we found was is that it became a very selfish, self-centered statistic, because they got bonuses based on their performance and so forth. And what we ended up doing is, because I'm going, “This is not working,” there goes the intuition. Right? It didn't take much intuition. It was pretty clear. And I said, “Well, wait a minute. What do we really want? What do we really want? What we really want is a team effort.” And so, then we started rewarding the teams. The rewards, it completely changed the environment.

So what I'm hearing you say, and tell me if I'm hearing this right, is that, sometimes, you've got to be careful about using the numbers too much, because at that point, you can actually drive behavior that you don't really want.

Tim Harford:

Absolutely. It's one thing to try to measure people's performance by looking at the numbers. That'll give you a good sense of what's going on. But the moment you go, “Oh, we are actually going to tie your pay or your promotion to these numbers,” well, at that point, you can forget the idea that the numbers will tell you the truth. Because the moment they're paid to get whatever it is, whether it's the five star review on Amazon or whatever it is, there's another way to get it without actually delivering the performance that you wanted. They will find a way to hit the metric instead.

This is well known. I mean, there's a classic management text, On the Folly of Rewarding A, While Hoping for B. There's a classic text in sociology. There's a guy called Donald Campbell. Campbell's law is all about this, that the moment you use this as a target, it's useless. And there's a similar idea in economics, Goodhart's law, and the fact that this same idea exists in sociology and economics and management tells you this is a real widespread problem.

Tom Wheelwright:

Hey, if you like financial education, the way I do, you're going to love Buck Joffrey's podcast. Buck's a friend of mine, he's a client of mine. He's a former board certified surgeon, and he's turned into a real estate professional.

So he has this podcast that is geared towards high paid professionals. That's who he's geared towards. So if you're a high paid professional, you're going, “Look, I'd like to do something different with my money than what I'm doing. I'd like to get financially educated. I'd like to take control of my money and my life and my taxes,” I would love to recommend Buck Joffrey's podcast, which is called Wealth Formula Podcast with Buck Joffrey. I hope you joined Buck on this adventure of a lifetime.

I'd like to go back, just for a second, to your first point, which was the combination of statistics, numbers, and intuition, because I found that, when I make the most mistakes, it's not because of the numbers that I've made the mistakes, because I didn't follow my intuition, and it's that gut feeling that, “Wait a minute.”

Just last night, literally just last night, I get this feeling, “Something's going on here. I need to look at this in my business.” And so the very first thing I did was, I sent out emails to my directors, said, “Okay. What's the pattern? Is there a pattern here? If there's no pattern, it's not a problem. But if there's a pattern, that's something we need to think about.” So sometimes, to me, the intuition actually even drives the need for the statistics.

Tim Harford:

Yeah, absolutely. And there's no silver bullet here. There's no single way, “Oh, you should always rely on your intuition,” or, “You should always look at the numbers.”

Tom Wheelwright:

Right.

Tim Harford:

There's no guaranteed way. So when you think about this, I mean, one of the classic ways of improving your intuition is to go to the data and use what's sometimes called the Outside View. So this is an idea from the psychologist, Danielle Kahneman. So I'll give you a specific example, and this isn't a business example. This is like an everyday example, but may be slightly in bad taste. But imagine you're sitting at a wedding and you see the happy couple are there, and it's the wedding breakfast. Everyone is kind of enjoying the happy day, and the guy sitting next to you turns to you and says, “What's the chance they're going to be divorced in five years?”

And you think to yourself, “Oh, that's such a tasteless thing to ask. How can you ask that?” And you start thinking, “Well, yeah. But I mean, I do see them argue.” Then you start thinking, “What is the chance?” But you're immediately thinking, “What do I know about this couple? What do I know about their relationship?”

Whereas actually, the first thing you should be asking is, “How often do couples split up within five years?” That's the first question. Until you know that, until you know what's called the base rate, then you're going to have all the personal intuition you like, but it's going nowhere, because you haven't got the right box to put it in. So you go to the data, and that gives you a starting point.

Tom Wheelwright:

So it's almost like you have to look at the macro before you get to the micro.

Tim Harford:

Yeah, but the common sense that-

Tom Wheelwright:

Yeah. I mean, because for example, if I look at the market, I'm going to look at the market before I look at my own personal investment, because the market data might tell me, “Well, you know what? This is the way this market performs, and it's unlikely you're going to way out-perform that market.” So maybe the real answer, I need to be in a different market or a bigger market.

Tim Harford:

Absolutely. Absolutely. So you need that overall data. You need the statistical context to inform your intuition. Now, you might go, “Okay. Well, maybe it should just all be statistics. Maybe we hand the whole thing over to the robots.” Right? And we don't. Well, I mean, maybe. I mean, there are trading strategies that sometimes seem to make sense, but without that, just that sense check from the human, I think it is very easy for the algorithms to get things wrong.

I just saw a wonderful example today. So you know algorithms now are really good at looking at photos and just figuring out what's in the photo. It's amazing. You just send them a photo of a dog and they're like, “It's a dog.” You show them a photo of a wolf and you say, “It's a wolf,” and they'll figure it out.

Well, somebody realized that, let's say you show somebody a photo of an apple and the algorithm will go, That looks like an apple.” But if, instead, you write with pen and paper, “iPod,” and you just put that label, “iPod,” next to the apple, and then you show that photo to the computer, the computer just immediately goes, “Well, you know what? Whenever something has writing next to it that says iPod, that's a really strong signal that it's an iPod.”

It's amazing. The computer will 99% go, “Must be an iPod.” It says iPod. There's no sense check at all. It's just going, “Statistically, whenever I look at a photo and it says a word next to the thing, then the word is the thing. That's what always happens.” So you can fool computers in the most straightforward way. The superpower is to get the data together, algorithmic inside, but is to combine it with human intuition as well. It's not easy, but you can do it.

Tom Wheelwright:

As you know, financial education is critical when it comes to creating wealth, passive income, and saving on taxes. Gain expert knowledge, as Marco Santarelli shares valuable insights and proven strategies for making money with real estate on the Passive Real Estate Investing Podcast.

Over the last five years, his guests have included Robert Kiyosaki, Mark Victor Hansen, Garrett Sutton, Brendon Burchard, Jim Rogers, and yours truly, Tom Wheelwright, just to name a few. If you are a busy person looking for actionable advice on the road to financial freedom, then this is the podcast for you. Available on every podcast platform, or simply visit passiverealestateinvesting.com. That's passiverealestateinvesting.com.

So really, my big takeaway from our discussion here, Tim, thank you for this, is really using statistics and numbers with my intuition. So doing that combination, seems to me, it makes total sense to me that that would be very powerful. Any final words you'd like to share with our viewers?

Tim Harford:

Well, let me try and boil down the advice in The Data Detective to the the three Cs, because I give you 10 rules. Actually, I give you 11 rules, because there's a bonus golden rule at the end. But to keep it really simple, when you're thinking about numbers, the three Cs are calm, context, and curiosity.

So calm is that thing we discussed. Notice your emotional reaction and try and interpret all the data calmly, rather than in the light of some argument or some preconception. The second thing is context, just asking questions like, “Is it big? Or is it small? Is it going up? Or is it going down? What comparisons make sense? And what's the definition behind the number?”

And the final thing is curiosity, just always asking yourself, “Well, what is this telling me about the world?”, rather than “How can I use this to win an argument with my wife or with my coworkers or with my friends or on Twitter?” Instead of using numbers as weapons, using them as windows onto the world t
help us make more intelligent decisions. Calm, context, curiosity. You don't need to buy The Data Detective now, but I suggest that you do anyway.

Tom Wheelwright:

For sure, for sure. We need to buy The Data Detective. It's not easy to make statistics interesting and fun. You've done a great job. Where else? You talked about your podcast. People want to know more about what you're doing in your work, Tim. Where would they go?

Tim Harford:

Sure. So my name is Tim Harford, and my website, timharford.com. I'm on Twitter and the other podcast. Check it out wherever you get your podcasts. It's Cautionary Tales.

Tom Wheelwright:

Cautionary Tales. You have it, so here's the great thing. When we look at statistics and are able to interpret them and combine them with our intuition, I think we're always going to be sure, so we're always going to make way more money and pay way less taxes.

Tim Harford:

Yeah.

Tom Wheelwright:

Well, thanks.

Tim Harford:

Well, I'll-

Tom Wheelwright:

Thanks, Tim.

Tim Harford:

I endorse that message. Sounds great, Tom.

Tom Wheelwright:

Thank you.

Announcer:
You've been listening to the WealthAbility Show with Tom Wheelwright. Way more money, way less taxes. To learn more, go to wealthability.com.