Episode 100: Noise Cancellation with Olivier Sibony

Announcer:
This is The WealthAbility® Show with Tom Wheelwright. Way more money, way less taxes.

Tom Wheelwright:

Welcome to The WealthAbility Show, where we're always discovering how to make way more money and pay way less taxes. Hi, this is Tom Wheelwright your host, founder and CEO of WealthAbility. So one of the most difficult things we do as business owners and investors is making decisions. So today you're going to learn how to get through all the noise and make those decisions easier. And actually, we're going to talk about a simple pattern and simple ways to make good decisions instead of making bad decisions. And I'm thrilled to have as our guest today, Olivier Sibony. And Olivier has [inaudible 00:01:00] studied this area of bias and noise and judgment for many, many years. You've got new book out called Noise, which I love the title just to begin with. So if you would just give us a little bit of your background on why you studied this topic, why is this topic so important right now?

Olivier Sibony:

Well, this topic has always been important. My background to an industry of [inaudible 00:01:22] is that right now I'm a Professor of Strategy in a business school called HEC Paris and at Oxford University. And before that, for the prior 25 years, I was a management consultant with McKinsey where I was ultimately a senior partner. And I was in a privileged position to see how the very, very senior business people for whom I had a lot of respect and a lot of admiration making very important decisions. And when you're in that position, it must strike you if it doesn't strike you, you are not looking that sometimes these incredibly skilled, incredibly talented, incredibly driven and highly motivated people are nevertheless making fairly surprisingly bad decisions. And that's what always intrigued me and that's what led me to study biases and now to study noise, which is the topic of the latest book I wrote together with Danny Kahneman and Cass Sunstein.

Tom Wheelwright:

So let's start with that. Bias I think people have a pretty good idea of what bias means. We see bias to information, we see biased press. Bias is a major word right now that we are all concerned about. How do you distinguish noise from bias?

Olivier Sibony:

It's very simple. The easiest way to distinguish it is to take the simple example of measurement. Suppose that your bathroom scale on which you step every morning to figure out how much weight you've gained since yesterday. Your bathroom scale is on average a little bit too generous. It understates your weight by one pound on average. And you know that, that's the bias of your scale. Your scale is biased. It's a little bit too kind. You probably like that bias in the big scheme of things, and you sort of know what you would need to do to get an accurate reading. You just need to add one pound to whatever the reading you're getting is. But if you step on your scale several times in quick succession, you're also going to find, unless you've got a really good precision scale, you're probably going to find that you don't get exactly the same reading every time. There's a seemingly random variation.

Olivier Sibony:

It's actually not random in the cosmic sense of the term. It depends on whether you stepped quickly or slowly, whether you stepped in the corner of the scale, in the middle of the scale. There is a reason for everything, but for practical purposes, it's a random variation. That random variation is the noise in the measurement instrument. So you can see that a scale like any measurement instrument can have a bias, that's an average error, and it can also have noise, which is a random error. And that the two add up, they may correct each other but in absolute terms, they add up. The same is true for judgements. Our judgements can be predictably wrong, and that's what we call a bias. If you are making an acquisition, for instance, we can predict with a pretty good degree of certainty that you're going to over overestimate the synergies you hope to get from the acquisition, that you're going to underestimate the challenges of integrating the acquisition and that you're going to overstate the price that you're prepared to pay.

Olivier Sibony:

We know that the direction of the error is always going, not always, but is generally going to be in the direction of that kind of optimism. Now that doesn't mean that everyone who prizes an acquisition or estimates synergies is wrong by exactly the same amount. There is going to be variability in the judgments of different people and that variability is noise. So in an important judgment that your listeners might do from time to time, just like in the measurements of their bathroom scale, we have predictable errors, average errors called bias, but we also have random errors called noise. It's a very simple distinction, it's just that we have talked a lot about bias in recent years, and we have talked very little if at all about noise. What we're trying to do in this book is to redress the balance.

Tom Wheelwright:

So typically when we talk about noise, we talk about cutting through the noise and making sure that we're not making decisions based on that noise. So when you look at that noise and what it does to a decision and how do you actually deal with that noise, in your studies and analysis, what have you come up with?

Olivier Sibony:

What does it do to your decisions? Basically it creates error, and here are two ways to think about it. There's at least two types of noise that you should be concerned about. The first type is in your organization, whatever it is, there are many different people who are paid to make judgements and to make decisions and who make judgements on behalf of the organization. One example we give is a study we did in an insurance company where there are underwriters and claims adjusters. And these are very skilled professionals who make important judgements. For instance, the underwriters decide how much you will pay for insuring a particular property or a particular risk. Not when you are buying insurance for your car, that's standardized, but for something complex and unusual, you need an expert to give you a judgment, an estimate about how risky this policy is, and therefore how much the insurance company should charge you as a premium to ensure the risk.

Olivier Sibony:

You expect if you are the head of the insurance company or the head of the underwriting departments, you expect that if you picked two underwriters to make the same assessment, you would get something pretty close. Not exactly the same number to the second decimal point, because, Hey, they're human. They're not machines, right? But you expect that the difference wouldn't be very large. It turns out when you actually do the experiment, that the difference between two randomly chosen underwriters who are supposed to produce the same judgment, because they are applying the same methods to look at the same data is in fact much larger than people expect. On average, they say, “We wouldn't expect any more than 10% difference.” It turns out that the correct answer is closer to 50%. It's five times larger than people expect. So that's the first type of noise that creates trouble in organizations. Depending on the person who happens to be in charge when the decision is made, you are going to get a very different answer.

Tom Wheelwright:

Okay. Let's start with that right there, Olivier. So how do you break through that? How do you keep that from happening? In other words, is there a way to standardize that decision making process ahead of time so that those two people will give you a more consistent result?

Olivier Sibony:

Well, standardize is a very interesting word, and we're going to come back to this one in a second. But before we do that, let me just point out the other type of noise, because the same remedies that we're going to talk about are going to help with both. The other type of noise is that the one underwriter that you happen to pick is not always the same person at all times. He's not going to give you the same decisions on the Monday morning when his football team lost the game the day before. And on a Friday afternoon, when it's sunny and he's planning an exciting weekend. He's not going to give you the same number when he's just looked at a very difficult case for which he set a very high price. And when he's looked at a very easy case for which he set a low price.

Olivier Sibony:

So this is within person noise, this is our own variability. Now you want to put in place measures what we call decision hygiene, to answer your question, that will protect you against both types of noise, against the variability between people in your organization who are supposed to be interchangeable. You want to actually make them more interchangeable, make the decision a little bit more standardized, as you say, that's an interesting term. And you also want to make sure that because the decision is standardized, it is going to be less dependent on the moods, the hunger, the fatigue, the sequence of decisions made prior to decision that any given individual is making. The same remedies are going to tackle those two types of noise, so you need to think through these remedies.

Tom Wheelwright:

So let's pursue-

Olivier Sibony:

What are they? Yeah.

Tom Wheelwright:

What are these remedies? I have some but I'd like to hear you first. And then I'd like go to through some of the things that we do and I'd really like to get your feedback on them. So [crosstalk 00:09:40] from your perspective, what do you see as these remedies?

Olivier Sibony:

There are quite a few. Let me give you a couple of examples. And in fact, to make them very concrete, let's apply them to a decision that all your listeners are making at one point or another, the hiring decision. We all know that hiring the best people is super important. It might be the most important decision that business people make. We all know that it's difficult, that it's actually not guaranteed ever, that you will get the best people and that we will all make mistakes in making hiring decisions. That's the fact of life. And we all have strong view based on our experience about how to do that. So here's what noise means in this situation. It means that first, the person you're interviewing will look different to you, depending on whether, again, it's Monday or Friday, sunny day or rainy day, et cetera.

Olivier Sibony:

It also means that the person who looks great to you might look not so great to one of your colleagues who is also interviewing that person. Which is by the way, why almost universally, when we have to make a hiring decision, we have several people in the company meet the candidates who are going to interview. Which is a form of recognition of the fact that there is noise and we need to do something about it. So here is the first thing that we can do and that we try to do, but we usually do wrong. Use the diversity of those points of view to reduce the noise in your judgment about the candidate. You're going to have different people meet the candidate and have a judgment on the candidate. Your people are diverse or we hope they are, they have different points of view, let's leverage that. The average of their judgment is less noisy judgment than any individual judgment, because noise will cancel out over a large number of judgments.

Olivier Sibony:

So you're probably thinking, “Yeah, sure. That's obvious we already do that.” Except most organizations, I don't know about you, Tom, but most organizations usually do not do that right. Here's what they do. They say, “We've met a candidate. We need to come to a shared point of view about that candidate, let's get together and talk about him.” And in fact, before they even do that, they communicate about the candidate. So maybe you've met the candidate first, Tom, and you knock on your colleagues door and you say, “Hey, Bob, here's a candidate I would like you to meet. Try to make time to meet him.” And maybe on another day, you say, “Hey, Bob, I've met this guy. Well, if you have the time, can you try and meet him and see what you think.” Right. You haven't actively tried to influence Bob here, but you've actually telegraphed what you're thinking about the candidate. Now, it could be much worse. You could actually be actively trying to influence Bob. You could be Bob's boss.

Olivier Sibony:

You could be someone who Bob respects a lot for his insight into people. So you could be influencing Bob either intentionally or unintentionally in a lot of ways before Bob makes his decision about the candidate. Bob's judgment has now lost a lot of its potential value. It's not independent from your judgment anymore. If you want to leverage the diversity of your people, the collective wisdom of your people in making less noisy decisions, you need to keep their opinions, to keep their judgements independent of each other before you get to the time when you're actually going to come to a shared conclusion. And that means taking active steps to prevent them from sharing their views about the decisions that they're making. Because if you don't take active steps to make sure that doesn't happen, it will.

Olivier Sibony:

Organizations are designed to make people communicate and to get people to converge, to get people to agree on what to do together and how to move forward. If you don't take steps to protect the independence of judgments, they will not be independent. And in most organizations they're right. So that's the first idea, use the average of multiple judgments by different people, but to do that, make sure that the judgements remain independent and are expressed in writing separately before you actually start discussing that. First idea, is that something you are doing?

Tom Wheelwright:

I'm going to find out, I hope the that's what we're doing, but you make a good point is that we are always influencing other people, right? I guess the other day that was her point is that we are always influencing other people, whether we-

Olivier Sibony:

Exactly.

Tom Wheelwright:

Believe it or not. And particularly when it's the boss, because they… I had this experience this morning. I've got a piece of software that we're beta testing. Well, I just want raw data, I don't want people to talk to me about it. I want raw data. I want to know what do you think? And their bias is, well, what you want me to think. Right?

Olivier Sibony:

Yeah. And we do this with the best intentions. So we're not aware of how damaging it is to influence each other when we should be trying to aim for independent judgements. And once you explain to people that this is actually not a good thing to do, and that they should do it differently, it's actually not very hard to keep their opinions any better. So that's the first idea that we can all relate to. Here's the second idea, which I'm going to try to illustrate again with the example of hiring. When you're hiring people, you are not, I hope you're not just saying, “Hey, I like this guy,” or “I like this person, let's hire this person.” You have a set of criteria that you're applying to a hiring decision. You have a job description, you have some things that you care about a lot, and that you have decided are the tests against which you need to evaluate your candidates.

Olivier Sibony:

You have criteria. Now, all these dimensions of assessments are exactly like the different people who meet your candidates. You ought to keep them independent of each other, but it's hard. Why is it hard? Because when you are having an interview with a candidate, you're forming an initial impression and that initial impression is coloring your judgment of the candidate on all the dimension. Your first impression is, “Great lady I'm meeting. I really like this person, I relate to her in a great way.” And then the question in your form says, will she fit the corporate culture? Well, yes, I like her. So she will fit the corporate culture. And the next question is, is she really smart? Well, yeah, must be really smart. And the next question is, how relevant is her experience? Well, extremely relevant now that you ask the question. So what you're seeing here is what is called the halo effect.

Olivier Sibony:

And it's been well known for a long time. The evaluations that we make of people on different dimensions, even when we try to keep them separate bleed into each other. Contaminate each other, influence each other in the same way that the different people interviewing the same candidate are influencing each other if you let them communicate. How do you overcome this problem? If you are evaluating a candidate on multiple dimensions, and you should be, define those dimensions as clearly as you can, and try to get different sources of input on each of the dimensions. If you want to know if the candidate is really smart, well, you can try to evaluate in an interview, but it's probably a better idea to give them a general mental ability test, if that's what you're interested in. Now, is that the only thing that you care about? Absolutely not.

Olivier Sibony:

You care about a lot of other things. Do they have the relevant skills and the relevant experience? Well, again, you could interview them and try to find out, but that's going to be influenced by a lot of other things you observe in an interview. Why don't you give them a job sample test, if they are supposed to write code, give them some code to write and have someone who isn't you, who hasn't met the candidate, grade the quality of that code. If you are hiring a marketing executive whose tasks are going to include writing copy for your ads or writing copy for your website, give them a little assignment where you ask them to do just that. And again, have someone evaluate the quality of the end product, who is someone who hasn't met the candidate and who is not influenced by the other dimensions. The more you can get inputs on the various dimensions that are independent from each other, the more uncorrelated they will be, the better your final decision is going to be because your multiple dimensions are not influencing each other. So that's the second idea, structure your decisions.

Tom Wheelwright:

Hey, if you like financial education the way I do, you're going to love Buck Joffrey's podcast. Buck's a friend of mine. He's a client of mine. He's a former board certified surgeon and he's turned into a real estate professional. So he has this podcast that is geared towards high paid professionals. That's who he's geared towards. So if you're a high pay professional, you're going, “Look, I'd like to do something different with my money than what I'm doing. I'd like to get financially educated. I'd like to take control of my money and my life and my taxes.” I would love to recommend Buck Joffrey's podcast, which is called Wealth Formula Podcast with Buck Joffrey. I hope you join Buck on this adventure of a lifetime.

Tom Wheelwright:

You brought out the idea of having clear criteria. And I actually think that's something a lot of people miss. They miss it when they're investing. I like to say that the difference between a professional investor and amateur investor, an amateur investor makes a new decision every time and a professional investor makes the decision once and just applies those criteria over and over again.

Olivier Sibony:

Yeah. That's an excellent way to describe it.

Tom Wheelwright:

Because that's what I've seen over the years. I've seen thousands of different investors in my career and that's what I see. So I think establishing those criteria, how do you establish criteria that are objective? Particularly, we're talking about job candidates now. Objective criteria, pretty easy in an investment, little more difficult when it comes to people. So how do you go about establishing those more objective criteria?

Olivier Sibony:

Well, I'll give you… That leads us to a third decision, hygiene technique, which actually applies just as well to investments as it goes to hiring candidates. Suppose that one of your investment criteria before you invest in say a new venture, suppose you're a venture capitalist and you're making investment decisions. Turns out that in fact, I am a part-time venture capitalist, but I'm part of the venture capital firm where we make those kinds of decisions. One of our criteria is what's the quality of the management team? And that's as difficult an assessment as when you are hiring someone, it's in fact the same kind of assessment. What we found and what is supported by the data and the research that we share in Noise.

Olivier Sibony:

Is that it's actually a lot less difficult to make those decisions, if you make them relative rather than absolute. What does this mean? If you ask me, “Is this management team great, good, or okay?” Well, that could mean a lot of things to a lot of different people. Even if you try to describe this with little sentences, which a lot of evaluation systems will do. It will be things like great means that it has a lot of relevant experience of creating ventures with success and it has a good dynamic where the founders are working well with each other. But what does relevant mean? What does a lot mean? Et cetera. If you want to make these judgments less noisy, more rigorous in terms of being interchangeable from one person to the next, you have to make these judgements comparative.

Olivier Sibony:

So here's what we do in that venture capital firm, where I work. We've got a list of the founders that we've met before and who are the founders of the companies we've invested in. And on each of the criteria on which we evaluate the management teams, we're going to say, “Well, on this one, our absolute benchmark, the person who stands for great is Susan.” Is the person we're looking at now, as good as Susan who defines the A standard? And we all look at each other and we say, “Well, he's great, but no, he's not as great as Susan.” Okay. Let's move on to a B. B is defined by Peter. Is this person as good as Peter? Yeah, actually on this dimension, he's just as good as Peter. And now we have complete consensus on each of the dimensions, we can of course have some divergence, but a lot less divergence between people than we otherwise would have. A lot less noise between us than we otherwise would have when we make those judgements relative. When this is-

Tom Wheelwright:

I think that's a much easier process, frankly, because it requires less judgment. And really all we're doing when we compare people, we do all day long, everybody's comparing people all day long anyway. We ought to be pretty good at it, right?

Olivier Sibony:

We're very good at it. We are hardwired to make comparisons. We are much better at comparing the length of two lines than at saying this line is 4.2 inches. It's very, very hard to measure something. It's very, very easy to compare two things. And this is a simple psychophysics observation, which we're applying here to other fields, but it's the same idea. It's much easier to make comparisons. Now it's easier, it'll requires a little bit of investment at the beginning, of course, to create those scales and to keep updating those scales, because maybe Susan and Peter who have been my benchmarks for a while, well have evolved in different directions. And Peter is nearly as good as Susan or Susan is even better than she was before and Peter isn't quite as good anymore. So we need to keep updating this.

Olivier Sibony:

And in our firm, we do. And we also need to make sure that the dimensions are the relevant ones and that we've got the right benchmarks and the right dimensions. Takes a little bit of work, but once that work is done once and for all, and I really like your observation, that professionals are people who define their criteria once and for all, and then apply them. That's an excellent way to think about it, as opposed to amateurs who think each case is a different one. Once you've done that you got a much less noisy judgment. So this is a third decision hygiene technique, try to make judgements relative, not absolute. We've talked about averaging judgements with multiple people. We've talked about structuring judgements, and we've talked about using case scales as we call them to make judgements relative, not absolute.

Tom Wheelwright:

I love it. This has been terrific Professor Sibony. I love this. Where can we find more about your book Noise and about what you're doing here in this what I think is just a critical field of study?

Olivier Sibony:

Well, there are lots of books on this topic. Of course, we recommend Noise. We think everybody should read this book, which is what every author will tell you, at least I hope. And you will find lots of references in Noise of other material that we think is relevant to this. Just to give a little bit more of background. We came across this topic in various different ways. We discovered that there isn't a lot of research specifically on the topic of noise that takes this angle, but there is a lot of research that has measured noise and has suggested the ways to reduce it. The decision hygiene techniques I've mentioned to you, plus many others that are described in the book in great detail. And we described those tools and those principles in some detail, trying to make them as practical as we can, and to give our readers the practical tips and tools that they need to tackle noise in their organization. So that would be a good place to start we hope.

Tom Wheelwright:

That's awesome. Olivier, great to have you with us. And remember everyone that it's really taking the noise out the decisions and thank you for the three simple steps. Simplify things so that you've got this good hygiene for making decisions. What you'll always end up with is way more money and way less tax. Thanks and we'll see you next time.

Announcer:

You've been listening to the Wealth Ability Show with Tom Wheelwright. Way more money, way less taxes. To learn more, go to wealthability.com.