Goodhart's Law

Jono Hey:

They gave them all pedometers, but very quickly, they were impressed about how much these kids were running around.

 

Of course, they found they'd stuck it on the dog.

 

Tom Pellereau:

Yeah.

 

Rob Bell:

And then you've got the example of Volkswagen.

 

They were programmed their cars to recognize that they're having a test done, and so you go into test mode and cut down your emissions, but it's not representative of what that vehicle would be doing out on the roads.

 

That's Goodhart's law, isn't it?

 

Jono Hey:

100%.

 

Tom Pellereau:

Do you remember during COVID at the beginning, where the government set themselves a target of 100,000 tests a day, and it turned out that they just changed the way that they measured a test, and so their definition of a test being done was that it had been put in the post.

 

Rob Bell:

You could call it gaming, thinking outside the box, or even manipulating the system, and the likelihood is we've all had a go at some point.

 

Hello and welcome to Sketchplanations, The Podcast, where each week we select a sketch from sketchplanations.com that explains something from the world around us.

 

We chat about it and see where it takes us.

 

Think of it as your weekly guided adventure through the Forest of Facts, across the Lakes of Enlightenment, around the Islands of Illumination, over the Peaks of Perspective, back through the Forest of Facts.

 

Oh, who's supposed to be map reading, boys?

 

Come on, this is basics.

 

I'm Rob Bell, broadcaster and son of a scout leader, and joining me once again, our senior ranger, Jono Hey, and still working on his Duke of Edinburgh Bronze Award, it's Tom Pellereau.

 

Hello, chaps, how are we doing?

 

All right?

 

Tom Pellereau:

Just thank you.

 

Jono Hey:

Very good.

 

Rob Bell:

Scouts, DV, Duke of Edinburgh, did any of you guys do any of that?

 

Jono Hey:

I wish I'd done some of that.

 

Tom Pellereau:

Me?

 

Jono Hey:

I didn't.

 

Tom Pellereau:

Me too.

 

I didn't do scouts.

 

Rob Bell:

Did you not?

 

Tom Pellereau:

I didn't do DV, I did cadets.

 

Rob Bell:

Ah, so that's another one, cadets, yeah.

 

Tom Pellereau:

I did the RAF, because I thought I'd get a chance to fly planes, and we didn't at all.

 

And the Army had a lot more fun, and I always, and all my other mates seem to do the Army, so I missed out on that.

 

But my son and daughter love scouts, and they do it every Wednesday, so they've really, really enjoyed it.

 

Rob Bell:

Yeah, it's good.

 

And Jono, you never did scouts or Duke of Edinburgh Award or any of these things?

 

Jono Hey:

No, I was always a bit jealous of the Duke of Edinburgh, going out, what it seemed like, going out on cool expeditions.

 

My niece recently joined the Air Cadets.

 

You get the little waiver for them to sign, and it says, are you happy for your child to, you know, handle a knife, be out without supervision at times, and fly a fighter jet?

 

And you're like, oh, that's the sort of thing that goes on in there.

 

It's brilliant.

 

Rob Bell:

It is brilliant.

 

Jono Hey:

That's what you hope for, isn't it?

 

Rob Bell:

In fact, exactly, and you say that, so my dad's been in and around Scouts for 60 odd years, and he also kind of is involved in Duke of Edinburgh as well.

 

And the bane of his life is the health and safety that he's getting more and more restricted around what kids can do.

 

Because I remember doing Scouts, my dad was a Scout leader, and I mean, it was really fun because we were doing all sorts.

 

It was great.

 

You're attacking your mates with sticks in the woods in the middle of the night.

 

You're shooting fireworks at them.

 

It's great fun.

 

Great fun.

 

Jono Hey:

Basically, you're still allowed to run them.

 

Rob Bell:

But one thing that all of this kind of area brings up, in my mind, which is just a beautiful piece of design and illustration and graphical representation is maps.

 

I think all of those things use maps quite key, the expeditions and the hikes and whatever it is you go on, you're orienteering.

 

I love maps.

 

I think they're just beautiful documents.

 

I don't know how you boys feel about them.

 

Jono Hey:

I could spend hours.

 

Tom Pellereau:

Jono has got one behind him.

 

I mean, look at that.

 

Jono Hey:

Yeah, there's the world map there.

 

But we were in the Lake District recently and just studying all the maps, particularly the 3D mountain maps with all the contours and stuff.

 

Just unbelievable.

 

I'm a big fan of Edward Tufte, who's essentially a professor in information design.

 

And he brings up the Swiss mountain maps multiple times in his book.

 

And it's amazing.

 

If you just look at a square of those maps, the amount of information, the density of information there is just remarkable.

 

And the Ordnance Survey ones in the UK are the same.

 

They're just fascinating.

 

You can look at every little shed and trickle of water anywhere.

 

It's incredible.

 

Rob Bell:

If you look at archive footage of Ordnance Survey maps being drawn, hand drawn on great big drawing boards, and the guys are their rulers and different inks and pens and colours, it's such a real skill.

 

It's an art.

 

It is an art.

 

Beautiful, beautiful creations.

 

Jono Hey:

You can be saved by maps as well, right?

 

They make you go down the right valley, that kind of thing.

 

I was also recently, like OS have an app now, and the game changing thing for it, obviously, is that it shows you your location exactly where you are on this map, which is always the hardest thing to do with a map is go, where the heck am I?

 

I get the picture, but where the heck am I?

 

And so I'm still like dreaming of the time where you could, you know, roll out like a metre by metre map and have a little dot where you are.

 

You know, like the Marauders map in Harry Potter or something.

 

Tom Pellereau:

We were watching that this weekend with the kids and the Marauders map, you see where everyone is and where your friends are, that'd be cool.

 

Rob Bell:

The second best thing about having OS maps on your phone is that you've got all of the maps that you could possibly want right there in your pocket, and you don't have to do that unwieldy unfolding and then refolding back on the page that you want and the winds lashing across and half of it's down the mountain.

 

Tom Pellereau:

But that was also half the fun.

 

Rob Bell:

I agree.

 

Jono Hey:

It's built some character.

 

Tom Pellereau:

It's a big old map and you're fighting with it to fold it in the right direction and the rains coming in.

 

Jono Hey:

Swearing at each other.

 

Tom Pellereau:

Blowing away.

 

Nowadays it's like, oh, I've got no signal or I've got no battery.

 

Rob Bell:

Should we keep going?

 

Yeah.

 

I prefer a paper map.

 

I'm going to put that out there now.

 

Even though it's very handy.

 

I am such a lover, Jono.

 

Proudly.

 

Jono Hey:

I'm smashing up your phone.

 

Don't ever hand rob your phone.

 

Tom Pellereau:

I'll put parts of it on his wall.

 

There's a piano up there.

 

Rob Bell:

I'm not afraid to dismantle.

 

Brilliant.

 

Brilliant.

 

Yeah.

 

And to your point as well, Jono, countryside maps, especially in mountainous areas, we've got lots of lovely contours and the colours that are used.

 

There's another range or there used to be another range of maps over here called Bartholomew maps, which I think were based on the same data as Ordnance Survey.

 

I think they nicked the data from Ordnance Survey, but they just made their colours more vibrant.

 

I use those all the time on Walking Britain's Lost Railways series I did on telly, because they just pop on telly so much more.

 

They're beautiful, beautiful maps.

 

And I always had the original prints from them.

 

So they're all from like the 60s and 70s, these maps, and they've been well worn and well thumbed.

 

They're lovely documents to have with you.

 

It's really nice.

 

Jono Hey:

History of many walks.

 

It is.

 

Rob Bell:

And probably arguments as well.

 

Jono Hey:

Swearing at each other in the rain, trying to get them open to the right bit.

 

Rob Bell:

Well listeners, it's my job to navigate us safely through this half hour or so that we have together.

 

Thank you, first of all, for all the correspondence that we've had on email and social media.

 

It's really lovely hearing all of your own perspectives on the stuff that we cover here on the podcast.

 

Please do keep them coming in.

 

We read every single one of them and we'll pick out as many as we can get through at the end of this episode.

 

We'd also love it if you could follow or subscribe to the podcast.

 

You can like and rate us.

 

All of that really helps us to keep putting these podcast episodes together.

 

Right then, before we get going, please make sure you have enough water with you, as well as all of the things that I put on the checklist that were sent out of a couple of spare compasses, but there's not enough to go around.

 

The weather looks fair.

 

It looks like it's going to play ball and stay dry, but do remember, layers are the key to comfort.

 

Right then, let's get going.

 

This week, we've chosen to explore something called Goodhart's Law, an observation around a trait of human behavior towards performance targets, observed and formalized in 1975 by a British economist called Charles Goodhart.

 

Now, you'll be able to see Jono's sketch that beautifully summarizes this phenomena, Goodhart's Law, as the artwork for this episode on your screens.

 

You can also find it in more detail at sketchplanations.com.

 

Right then, Jono, where did you first hear about Goodhart's Law and why did you think it worthy of a sketchplanation?

 

Jono Hey:

Yeah, I first read about it in a magazine article, actually, a sort of science magazine.

 

And it was one of those ones, well, first of all, I'd never heard of it.

 

And, you know, I was in my 30s.

 

And secondly, it just immediately spoke to me as very applicable in lots of different areas.

 

I think the article at the time, and I couldn't find the original, was about Uber drivers gaming their ratings and actually also having trouble because the company was measuring them on the stars.

 

And if they dropped below a certain amount of stars, then they get kicked off the platform and they didn't like that.

 

And so they figured out ways to get stars, despite the fact, the normal ways to get them.

 

Anyway, so, but as soon as I saw it, I thought, you know, that's really interesting.

 

And you can start to see different places in your life, in my professional life, where this has come up.

 

And over a long, slow and quite important way in many cases.

 

Rob Bell:

So let's kind of nail down, what is Goodhart's law?

 

What is proposed in this observation of human behavior?

 

Jono Hey:

Yeah, so Goodhart's law is really simply put, is when a measure becomes a target, it ceases to be a good measure.

 

As a smaller side, it's also a good example of why Sketchplanations is not really about the sketch.

 

It's about the idea, because it's not a great sketch, but it just gets across the point so well.

 

And it's been one of the most popular ones that people keep coming back to.

 

Rob Bell:

Has it?

 

Jono Hey:

Because it's such an interesting law, yeah, absolutely.

 

Yeah, they've been like shared on, reshared on Twitter and things like that.

 

Tom Pellereau:

If you put Goodhart's law into Google, Sketchplanations is very top.

 

Absolutely the first thing you see.

 

Rob Bell:

Well, that's it.

 

Jono Hey:

I mean, I'd never heard of it before.

 

You know, it was quite interesting.

 

It was a little known, I would say.

 

Rob Bell:

So what is the sketch?

 

Let's explain that.

 

Jono Hey:

So the sketches, it was really when I was reading about it, there were a few examples that were given.

 

And the most common one was this one about, I think it was a nail factory.

 

And if you've got some central planners trying to optimize for getting as much productivity out of this nail factory as possible, they said, okay, well, we're gonna measure people on the number of nails that's made.

 

And then of course, what you get was people delivering thousands and thousands of teeny tiny nails, because they all count, because that's what you're being measured on.

 

And so they're like, right, we'll change it up and we'll actually, we'll measure it on weight of nails, because then you can't stitch us up like that.

 

And then of course, they got a few giant heavy nails, which are also no use at all.

 

So that's the example.

 

And it just like, it's kind of absurd.

 

I'm sure it's fictional, but it kind of gets across the point that of course, if you're gonna be measured on that, you're gonna figure out a way to make it work for you.

 

Rob Bell:

And the phrase that's used a lot about making it work for you is gaming that target, right?

 

Gaming that deliverable.

 

Tom Pellereau:

That's a kind way of putting it, maybe.

 

Rob Bell:

Yeah, yeah.

 

But I like it because it comes back to this trait of human behaviour.

 

Because these targets are made for, you'd build a target like that for something that wasn't sentient, right?

 

For something that is a machine that can't think outside of the parameters that it's been given.

 

Whereas we're humans, and we're gonna try and do the best for ourselves as we possibly can.

 

And so, if you were incentivised by that target to do something, to game it, then you're gonna do that.

 

Jono Hey:

Yeah, I think the other example that really sticks with you, just making it so clear, was about, I think it was measuring life fitness of kids in cities, and they gave them all pedometers.

 

And they were impressed about how much these kids were running around, of course they found they'd stuck it on the dog.

 

Because that's what you're being measured on.

 

So yeah, that's it.

 

You will figure out a way to game the measure is basically Goodhart's Law.

 

And I think as soon as you start looking around, anytime that a measure is a target, you sort of see that this really does happen.

 

Rob Bell:

And there are loads of examples that I've read about that are kind of, some of them are anecdotal and others are real examples.

 

I mean, Tommy, have you, any examples that you can think of, or you know of, of Goodhart's Law?

 

Tom Pellereau:

Yeah, I've got some sort of serious sort of business ones, but kids just make me laugh so much because they are the masters of gaming overall.

 

I'll never forget watching, watching tennis, a group of kids playing tennis and there's loads of balls around.

 

And the coach goes, okay, whoever can get the most balls and bring them back to me wins.

 

And one kid just ran straight to the bucket where all the balls were and poured them on the floor and then picked them, you know, and basically put the balls into his shirt.

 

And the whole point of obviously the exercise was the coach wanted the balls picked up and put back, not the bucket emptied out into someone's shirt.

 

And they were just so chuffed with themselves at this point, like that.

 

And then I've done similar things with Jack, you know, in the garden, you're kind of like, right, okay, got to show them, teach them grit and help out in the garden.

 

So it's like, okay, well, there's some sort of rubbish or there's bits and pieces around.

 

Jack, you know, how many of the bits of like paper, or you know, after you've done some building work, so there's lots of bits out in the garden that needs sort of picking up and they're quite close to the ground so they can pick the bits up.

 

And you go, you know, who can get the most bits or you sort of go, or maybe I can sort of pay them for bits of paper, you know.

 

So what they do is they go over and they find a bit and then they rip it up into lots of little bits.

 

So this is no longer one bit of rubbish, it's now 50 bits of rubbish.

 

And actually all they're doing is sitting there ripping bits of rubbish up to make them smaller.

 

So there's more bits here.

 

Look, Danny, I've made, look, I've made, that's not the body.

 

Rob Bell:

Three hundred quid later and you've questioned how good a measure that was.

 

Jono Hey:

I remember doing litter picks in secondary school.

 

I don't know if you ever did that.

 

And I do remember them counting the bits of litter that you had to bring back.

 

Rob Bell:

Really?

 

Tom Pellereau:

And what are you gonna do?

 

Jono Hey:

With the scissors.

 

Rob Bell:

Exactly.

 

There's a really good example I think that is again, it's one that's widely written about or spoken about and it was the Cobra Effect.

 

Did you read about that one?

 

So, and I don't know if it's factual or if it's anecdotal or what, but I suppose the British colonial government in India in the early 1900s, decided that there were too many loose cobras, I was gonna say running, slithering around the streets of Delhi.

 

So the government issued a policy offering a bounty for every dead cobra that was brought in to try and rid the streets and make them a safer place for the citizens of Delhi.

 

And it worked for a while.

 

So they started breeding cobras.

 

And they started breeding cobras.

 

And so then the government decided, well, this is not working anymore.

 

So they scrapped the policy.

 

So then everyone who'd been breeding cobras just let them out into the streets because they were a danger to have at home.

 

That's the Cobra Effect.

 

That's one I read about.

 

Jono Hey:

I don't suppose you read about that on the Sketchplanations website.

 

Rob Bell:

Did I read about that on Sketchplanations?

 

Maybe I did.

 

Jono Hey:

I don't know if you did, but there was a Sketchplanation for the Cobra Effect.

 

Rob Bell:

Have you done a Sketchplanation?

 

Do you know what?

 

I didn't, actually.

 

I didn't read it there.

 

Jono Hey:

I think, because I was thinking of it when you said about the tennis ball example, Tom, because all you want is the tennis ball's in the bucket.

 

And so you measure that and they tip them out of the bucket.

 

Rob Bell:

It's like, boom.

 

Jono Hey:

It's a great...

 

Rob Bell:

Balls, cobras.

 

You say potato.

 

Jono Hey:

A great example of the Cobra Effect I heard about in Bogota, where they were trying to reduce traffic.

 

And so their idea was that you could only drive cars with a certain license plate on certain days of the week.

 

Rob Bell:

Oh yeah.

 

Jono Hey:

And so you could only drive on Mondays, Wednesdays, this week and Tuesdays, Thursdays and next week.

 

And so what do people do?

 

Tom Pellereau:

They got extra cars.

 

Jono Hey:

They got extra cars, exactly.

 

Tom Pellereau:

Well, of course you would.

 

Jono Hey:

There's some great examples of the Cobra Effect out there.

 

I mean, often there are quite awkward ones, you know, like quite awkward, like problematic ones.

 

You know, you bring something in to like, like get rid of cobras, but actually like genuinely to get rid of like invasive pests in a country and the thing you brought in has other effects that you didn't expect.

 

But yeah, Cobra Effect.

 

Yes, it's fascinating.

 

Tom Pellereau:

Do you remember during COVID at the beginning where the government set themselves a target of 100,000 tests a day?

 

Do you remember all that?

 

Because at the beginning of COVID, we didn't have enough tests and the government got really sort of punished, ridiculed for this.

 

So they were like, we're going to get, and they put this massive target in.

 

And then they somehow managed to miraculously hit it.

 

And it turned out that they just changed the way that they measured a test.

 

And so their definition of a test being done was that it had been put in the post, as in they'd sent a test out.

 

Wasn't the fact that someone had actually received it or done it.

 

Rob Bell:

Or done it correctly.

 

Tom Pellereau:

Or done it correctly.

 

They were like, yeah, we put 100,000 into the post office, into the post box yesterday, so we met our target.

 

So that's not the definition of a test being done.

 

Jono Hey:

Yeah, it's a good one.

 

Big consequences, isn't it?

 

Rob Bell:

One I thought of talking around cars is, you know, CO2 emissions becoming a measure of how clean or efficient your car is.

 

And, you know, you see that advertised, loads and loads and loads now, because there's quite a lot at stake on it.

 

You know, your congestion charging in big cities, company car, taxes, tax bans are based around it.

 

And then you've got the example of Volkswagen and what they were programmed their cars to do, to recognize that they're having a test done.

 

And so you go into test mode and cut down on it.

 

You're cutting down on your emissions because some, they've got the engine to act more efficiently, but it's not representative of what that vehicle would be doing out on the roads.

 

And I think that's Goodhart's law, isn't it?

 

That's banging in there.

 

Jono Hey:

100%.

 

I mean, I was going to say, one of my examples from my life, which we can probably all relate to, was cramming for exams.

 

There were definitely some subjects where we got good enough to pass the exams.

 

And then literally a week later, I probably couldn't have hardly told you any of it, which is obviously not the intent of the exam.

 

Rob Bell:

And you're right, exams is something that probably everybody or almost everybody has experienced.

 

And it's not necessarily the best means.

 

It's a measure that you can game through cramming and just learning the exam process.

 

I think it's a really, really classic example.

 

But then what are you supposed to do?

 

Because you're trying to find a measure across a massive population, that you're trying to standardise it as best you can.

 

And ultimately, the thing that strikes me with exams and with many of the other things that we've talked about here is how unfair it can be.

 

Jono Hey:

Yeah, I mean, if you're a school and your budget is dependent on the number of people who pass an exam, then you're gonna try and make sure that most people pass it.

 

If you're a teacher and your promotion or whatever depends on them passing it, you're probably gonna do your best there to get them over the line, even though you might think that's not the best, the very best use of the time you had together.

 

I mean, I'm not an expert on this at all, but there are some other ways to do this.

 

And I think some other countries just do stuff in different ways.

 

There's much lower emphasis on exams, more emphasis on trust, more respect for letting qualified teachers do what they think is best.

 

And using, you can still use assessments as a way of gauging progress in where you are and informing what you're doing, but you don't necessarily have to have consequences at the end of it.

 

But these things are tricky, which is why Goodhart's Law is everywhere.

 

Rob Bell:

And why we're talking about it.

 

I mean, one area that's a little bit meta for the podcast is trying to get the podcast viewed, listened to as much as we possibly can.

 

And so trying to understand the algorithms that will promote it as much as we can.

 

That's why we tell people to like and subscribe and rate and five-star us and follow all this kind of stuff.

 

Because we're trying to understand the algorithms that will kind of pop that up higher into, I don't know, whatever it is they're looking at, their recommendations that come at them.

 

Tom Pellereau:

And is that why you've been listening to it on Spotify and Apple and other platforms simultaneously?

 

Rob Bell:

Yeah, I've got all my old phones.

 

Jono Hey:

Just got it on repeat.

 

And my music on Spotify, I get a tiny fraction every time somebody listens to it.

 

So if I leave it on repeat all day long, you know.

 

Tom Pellereau:

Does it know your ISP?

 

Very popular.

 

Jono Hey:

I don't think it touches Spotify's bottom line.

 

Rob Bell:

But I think the likes of, aren't the likes of Instagram and Spotify and Google and TikTok and all this, aren't they constantly trying to change or develop the algorithms so that they can be more reflective and so that they're not being gamed as much?

 

Jono Hey:

In some ways, I think Goodhart's law is responsible for like ruining quite a bit of our experience of the web.

 

If you go back to the very original web, which wasn't good, but it was fairly honest.

 

Whereas Goodhart's law is partly what gives you clickbait titles.

 

Some people just click on it, whether or not the article is good or not.

 

Rob Bell:

Click farms.

 

Jono Hey:

Yeah, people just clicking on stuff for whatever, emails with subject lines that just draw you in, even if it's not great.

 

And I think the original, I don't know if you guys were doing web stuff this year, but the original page rank algorithms that Google was doing, right?

 

So search engine optimization, SEO, it makes or breaks your business if you're at the top of the list.

 

So everybody's trying to second guess what Google was doing.

 

And I remember the days where you would put tiny text in exactly the same color as the background on the bottom of the page or on the top of the page, just so it was on the page, because it wouldn't really know.

 

And then of course, they have to then get more sophisticated and go, okay, well, the text has to be big.

 

The text contrast has to be enough.

 

But you know, because they're trying to do this.

 

And so everybody's like, they take one step, people figure out a way.

 

They take another step, people figure out a way.

 

I do have to say, generally, I think, like my approach to this, and I would do it for the podcast too, is if you make good stuff, things will rise up.

 

Like you should do things in a respectful way.

 

And I've always tried to do it with Sketchplanations.

 

That's probably why it's not a massive commercial success.

 

It's just an interesting project.

 

But if you try and just do stuff, honestly, I think PageRank and stuff like that, they're always trying to find the good stuff.

 

And so I trust, as much as possible, that they're going to figure out all the people trying to game it.

 

And mine go to the top, but maybe that's not the case.

 

Rob Bell:

And I'm 100% with you on this, with anything to do, not just on the web, so with the podcast, but with this podcast, but also with stuff generally.

 

Just have trust in what you're doing and trust in yourself to do the best that you can.

 

And let that be the incentive and have faith that that will rise, if it is of decent quality.

 

Jono Hey:

You know, some of the guidelines for articles, like, okay, this is how Google ranks stuff, you know, like titles that reflect the content of the article, structured titles that help break up your article into readable chunks, interesting content that will help people stay on the page longer.

 

And you're like, that's just writing a good article.

 

That's the idea of the whole thing.

 

So in some ways, I feel like a lot of places, we've maybe got there, but probably a lot of places we haven't.

 

Maybe I'll add some interesting things around like trickier things to measure, like trustworthiness or if you're in finance, like fraud.

 

And so, for example, if you're trying to assess, is this a real account and you're trying to use a lot of signals to do it, in a way, you can't tell people how you're measuring it.

 

Because if you tell people how you're measuring it, people will figure out how to game it.

 

But, of course, it doesn't feel good to not know how you're being measured.

 

It feels a bit sort of creepy, a bit irky, like, oh, I don't know, they're in control.

 

It's all a black box.

 

But actually, the fact is, if you said, this is how we determine if your account is real or not, people will figure out how to do fake ones using those guidelines.

 

So it's a tricky thing.

 

Right now, I think that's a tricky thing.

 

And fraud as well.

 

Bank won't tell you every way that they are assessing whether or not your transaction is likely to be fraudulent, because otherwise people will just do all the things that pass all the tests.

 

Rob Bell:

Yeah, it's tricky.

 

Tom Pellereau:

And often the things that seem to be the things that you should be measuring, like for a business, profitability, right?

 

It's the most important bit of a business, how profitable it is.

 

But profitability is usually led by many other things down the line.

 

And you can be kind of at your profitable, or actually currently, where in the future, it's actually the business is not going very well.

 

Do you understand me?

 

So for my business, our lead measures are how many good new products we have coming.

 

And that can be quite independent of actually how profitable we are in any given month.

 

Rob Bell:

Okay.

 

Tom Pellereau:

So it's new products coming that we know could be really good ones, and also how many new customers we've got.

 

And it takes about 12 to 24 months for us to bring on a new retailer, right?

 

So if you are measuring profitability this month, that actually has no gauge really on where your profitability will be in the future, unless you start looking at how many new products you've got coming, Tom.

 

What's your pipeline like for new products?

 

What's your pipeline like for new customers?

 

And those are way away from profitability, but your current profitability.

 

Yeah.

 

But if you're not, but ultimately profitability is the most important bit about business.

 

So it's, I think they're called leaders.

 

Leading metrics.

 

Leading metrics.

 

Thank you, Jono.

 

I'm rubbish at remembering the terms for these, but those are the thing.

 

And those can be actually really, really difficult to understand within a business what they are.

 

But also sometimes they're just bleedingly obvious, right?

 

It's like, oh yeah, you invent products, Tom.

 

Probably the lead measure is how many new products you've got coming.

 

And how well they sell in the first couple of months.

 

You're like, oh yeah, that's actually really obvious.

 

Oh, thanks.

 

Jono Hey:

Tom, if you keep forgetting, you can always check out the sketch on leading and lagging metrics.

 

Rob Bell:

I'm going to have to make a note of these so that I can include them all.

 

Jono Hey:

There's a really nice philosophy I really like.

 

I think it's by a guy called Bill Walsh, which is the score takes care of itself, which is an American sports coach.

 

But it's that idea.

 

And actually, I really like right now, Man City have just won the Premier League in the UK.

 

And I like Pep Guardiola as a manager, because when his team loses or draws, doesn't do as well as they should, he can be very complimentary of his team and everything that they do, because he knows that they're doing the right things, such that their overall success will take care of itself.

 

And I like that idea, the score takes care of, like focus on the products that you're bringing out.

 

Profitability will follow.

 

Rob Bell:

And it comes back to that trust, right?

 

It comes back to the trust in the product or the service or the performance that you're giving, and that you need to have ultimate trust that you are doing the best that you can and that that will prevail.

 

Tom Pellereau:

And as Jono said, it's about having content, having the right content and the right of the right products, doing the right thing, ultimately, all of us.

 

Jono Hey:

So if you take that same mindset, sorry, Tom, and you do the same thing, but not for business, but for education, right?

 

Rob Bell:

Yeah.

 

Jono Hey:

Which was exactly what you said, like have trust.

 

The score takes care of it.

 

The success of the students will take care of itself.

 

Trust the teachers, trust the education, people do their job.

 

That's kind of what you're saying there, isn't it?

 

Rob Bell:

Yeah, it is.

 

Jono Hey:

It's hard at those big scales to do those things.

 

Rob Bell:

Yes.

 

Yeah, it really is.

 

When so much external factors or so many other factors, I should say, then based on that, like budgets, like your school rating, like whatever else it might be.

 

Jono Hey:

Where I work, like building products on the web, I've seen a really common one, I think, that comes up is marketing is typically about bringing people in.

 

So they're often measured on like leads or sign ups, for example.

 

Rob Bell:

Yes.

 

Jono Hey:

And then product and ultimately the success of the business is obviously those people not just signing up, but becoming customers, becoming paid customers and doing the things on the site.

 

And so there's always this little tussle.

 

It's like, well, we're bringing loads of leads, but none of them are converting.

 

You're not bringing the right people.

 

Well, but the site isn't doing its job at converting the right people.

 

And so there's always a little back and forth.

 

And of course, there are obviously there are solutions to that.

 

But I think it's a very common thing.

 

It's like, okay, as we specialize in all these areas, you're going to focus on bringing them in.

 

I brought loads of people in.

 

Yeah, but they were the wrong people.

 

Or no, you're not doing your job when they get there is the problem.

 

You know, the course isn't good enough, you know, or the program or whatever.

 

Rob Bell:

So then for it to be useful, it has to be a combination of all these measures.

 

But then you get into this complexity of what that measure is and how to analyze it.

 

Jono Hey:

Speaking of complexity, I think, Rob, you're with Vitality as well, right?

 

Rob Bell:

Yeah, health insurance.

 

Yeah.

 

Jono Hey:

Yeah.

 

And so I think Vitality is a really good example, back to the putting the pedometer on a dog.

 

And like you said, Tom, the intent of these things is good.

 

Like they are trying to keep people healthy so that they don't have to fork out money for insurance for medical procedures in the future.

 

But they have to decide, like the ones like to decide, you get money off your insurance for safe driving.

 

They have to decide, are you being healthy?

 

And so they do various things, you know, like if you wear a watch, you can count your steps and you get points for workouts and mindful minutes or whatever.

 

But then there are rules to those.

 

And I don't know about you, Robert, every now and then I'm like, I don't know, it counts a 30 minute workout.

 

So if I'm on 28 minutes, I'll probably walk up the road for two minutes.

 

Rob Bell:

I do the same.

 

I just did it tonight.

 

I just did it tonight.

 

Tom Pellereau:

Yeah.

 

Jono Hey:

And of course, and of course you would.

 

But, you know, like, you know, was that extra few minutes worth it?

 

Or like, this is where you know, people jogging in their living room or something to get over the 10,000 steps.

 

Tom Pellereau:

But at least that's good.

 

I thought you were going to say, so I give my watch to Rob and I let him carry on for the next 20 minutes.

 

Jono Hey:

That's a good idea.

 

Rob Bell:

Well, you could do that.

 

Jono Hey:

You could run a business taking everybody's smart watches for their vitality health insurance.

 

Rob Bell:

This is brilliant.

 

Jono Hey:

I don't mean that vitality.

 

Rob Bell:

No, nor do I.

 

But there's another way.

 

Tom Pellereau:

I bet there are people doing that.

 

Rob Bell:

Oh, definitely.

 

There used to be a thing where it might have been for vitality or something else, where it was before they had the complexity of linking it to your smart watch.

 

It was the fact that you went to the gym.

 

So if your membership registered on the gym system, that you'd gone through the turnstiles.

 

That's it.

 

Cool.

 

You've gone to the gym.

 

Great.

 

Well done.

 

We'll tick you off on your points.

 

I know someone who would go there in the morning, go through the turnstile or straight out the exit and get in the car and take the kids to school.

 

Well, I've done it.

 

Jono Hey:

Just going to pop to the gym.

 

Rob Bell:

I've been there.

 

But then, but the other thing with this as well.

 

So one of the rewards for being active, and this isn't necessarily Goodhart.

 

I'll get your guys' impression on it.

 

So with Vitality Health Insurance, with the points that you gain through being active in the week, one of the rewards you get is to go to Nero Coffee and you can get free coffee and you can get whatever coffee you like.

 

So I go in and get the most unhealthy coffee.

 

The most expensive, which is the one with, you know, your full fat, oat milk, caramel latte with squirty cream and all the rest of it, and a large one.

 

I wouldn't do that normally.

 

Jono Hey:

A cake, maybe.

 

Rob Bell:

Yeah, whilst I'm at it.

 

I wouldn't do that normally.

 

I wouldn't go and get that coffee.

 

So is there an element of Goodhart's law in that?

 

Jono Hey:

That sounds a bit more like the Cobra Effect, doesn't it?

 

We're having heart attacks with coffee.

 

Rob Bell:

I've been bitten in the neck by a Cobra.

 

Yeah, it's interesting though, isn't it?

 

All these systems in place, and we're all guilty.

 

I think a lot of people can relate to gaming systems in some way or another, or measurement systems, as is specific with Goodhart's law.

 

Jono Hey:

When you said about the kids, I thought about, it's not a target, but we're at the table, and they always want to get up and go off around the room.

 

Look, please stay in your chair.

 

And of course, then they sit down in the chair, pick up their chair, walk around the room.

 

That's what you do when you're a kid.

 

This is what law is like, right?

 

Like a whole of tax stories, so blooming complicated.

 

Because, you know, if you make more money as a contractor, then if you're employed, so everybody starts contracting.

 

So you have to change the rules to close the little loophole.

 

Rob Bell:

Yes.

 

Jono Hey:

And then some people find some other loophole.

 

That's why I like the British legal system is incredibly complex in a way.

 

It's Goodhart's law.

 

Rob Bell:

That's how good lawyers make their money, right?

 

Is finding loopholes.

 

Jono Hey:

I don't know if that's the only value they provide.

 

Rob Bell:

That's how...

 

Jono Hey:

Knowing the rules of the law is part of the expertise of a lawyer.

 

Rob Bell:

Are there ever examples where Goodhart's law...

 

Tom Pellereau:

You might need to.

 

Jono Hey:

You're going to hear from some.

 

Tom Pellereau:

Lock your front door.

 

Rob Bell:

Yeah, I've gone a bit warm now.

 

Are there...

 

I'm going to change the subject.

 

It's certainly interesting.

 

I hadn't considered this as something that's happening in the world around us before looking at the sketch and before thinking about it with a bit more detail and talking about it on the podcast, but it's everywhere.

 

It's happening all around us.

 

Tom Pellereau:

It certainly is.

 

It's so enlightening to know that it exists because otherwise you can go about your business thinking you need to make up targets and things and not think through the unintended consequences of them kind of thing and to be careful and probably needing to understand that you probably need to be dynamic and moving with your targets.

 

Otherwise they will probably get gamed after a while.

 

Not moving, but flexible.

 

Like this is not always, unless you find that perfect one, which might take a few iterations to find it.

 

Rob Bell:

Yeah, but even then, once you think you've perfected it.

 

Jono Hey:

I think, you know, as a general philosophy for life, it's nice to trust more and measure these things, but don't use them to like force your actions.

 

Just use them to inform and guide your actions.

 

Keep track of how many times you turned up at the gym, but don't get hung up on that being the thing.

 

And you just go to the gym and go to the bar.

 

Tom Pellereau:

Is that maybe a thing that kind of the reason we game these things is because they were ridiculous, as it were.

 

So if you kind of try to impose something really rigid on someone, they will probably just be like, no, this is ridiculous.

 

Whereas if you show trust in them and their ability and work with them to try and make everything better, then they probably won't be trying to game you the whole time.

 

Rob Bell:

I see two incentives.

 

One is personal reward and the other is to be bloody minded and make a point about something.

 

But my big takeout from this is about trusting in the quality of what it is you're doing or trusting in the quality of a system that has been thought about in some way that justice will prevail and that anything that is unfair about it, if you stick to your principles, which are hopefully positive, then it should come out in the end.

 

Jono Hey:

It's nicer to live your life that way.

 

Know all the rules, but live your life with a little trust.

 

It's funny about you saying about the bloody minded.

 

I was just thinking of it was the vote for player of the season recently, which of the team we support, and it just got completely bombed by the local rival team voting for the goalkeeper.

 

Rob Bell:

Did it.

 

Perhaps listeners, there's something within your control at the moment that's suffering from Goodhart's Law, again, at work or in your community or at home with your family, is the voice we talked about.

 

And we'd love to hear about it if so.

 

You can send us an email to hello at sketchplanations.com or you can leave us a message or a comment on social media.

 

Please do continue to like and rate and subscribe to the series because we want to get behind those algorithms.

 

We're going to game it, guys.

 

No, we're not.

 

Jono Hey:

Just trust in the process.

 

Rob Bell:

Trust in the quality in the process.

 

Jono Hey:

If you're saying interesting things, people will subscribe.

 

Rob Bell:

And we will trust in the brilliance of our listeners to keep coming, keep telling their family and friends.

 

A bit of both.

 

A bit of both.

 

Jono Hey:

You can't help it, can you?

 

Rob Bell:

Why not?

 

Next week, we'll be talking about our senses, including cross modal perception.

 

It's something I knew about, but didn't know I knew until I saw Jono Sketch and learned the name for it, as with so many different things.

 

It's already up there on Sketchplanations if you want to take a sneak peek, or you can wait until next week for the podcast, which will be episode 12, can you believe?

 

But with my trusty compass in one hand and an awkwardly folded OS map in the other, the three of us now set off towards the sunset in search of our next adventure.

 

Stay tuned as we go through this week's post bag in just a second.

 

But for now, thank you all very much for listening.

 

Stay well and go well.

 

Cheers.

 

Goodbye.

 

Right, let's get into the post bag.

 

So this is...

 

Well, last week's episode was our quick fly around on sketches we found surprising.

 

We got through 12 of them, boys.

 

Did well there, I'll tell you what.

 

And had an email from Dan who...

 

This was about the owl, the twit-to-woo sketch, saying, do you ever just hear a twit sound from one owl?

 

I don't have the answer to that.

 

But he was suggesting that that was the obvious question that we didn't ask in the podcast, or just a two.

 

Jono Hey:

Let's listen out.

 

Rob Bell:

Yeah, let's listen out.

 

Let's listen out.

 

We need a naturist, naturist, naturist, naturist.

 

Tom Pellereau:

Naturist, very different.

 

Rob Bell:

Very different naturist.

 

Actually, this is also from Dan.

 

He says he loves the idea of a sneeze switch, which we talked about in the podcast.

 

This was one you brought up, Tommy, about looking up to bright lights in order to help with a sneeze.

 

And he goes on to say in his email that he's also an advocate of the sit-down wee, which was a reference to the keeping one eye closed when you go to the toilet in the middle of the night.

 

Jono Hey:

Definitely wasn't the topic of the podcast.

 

Tom Pellereau:

And too much information.

 

Rob Bell:

No, it wasn't.

 

Yes, yes.

 

The idea of the t-shirts folded in the drawer as the kind of filing cabinet, which I've done for years because of the sketch.

 

My girlfriend, she came to me in the week and said, is that why you do that?

 

Yes.

 

So that was a realisation of her as well.

 

Jono, we had an email from Niall.

 

Jono Hey:

Yeah, we had a quick note from Niall.

 

He mentioned at the time of the Abla reduplication sketch, which is the one which is about zigzag, flip-flop, they actually shared it with the advertising guru who he believes originally coined Bish bash bash, which is pretty great, isn't it?

 

Rob Bell:

A chap called Dave Trott, is it?

 

That's right.

 

An advertising guru.

 

And this is off the back of a conversation I was having with a friend of mine at the weekend.

 

So she told me that she was sat listening to the podcast about optimism bias a few weeks ago, where the three of us were talking about how we've often gone into things, maybe slightly over optimistically perhaps, but overconfident of our chances of success.

 

We went through a number of examples of where that was the case in our own lives.

 

And she told me that she was sat there listening to it, thinking, well, no, that's not how I think at all.

 

And then later on in the podcast, when we queried about whether our joint perspectives, the three of us on this were perhaps typically male, and that maybe a more female perspective would be slightly more measured, maybe a bit more risk averse.

 

And she said that suddenly she was saying to herself, yes, that is me, that is how I operate.

 

So whether that's a male, female thing or just different individual perspectives, I don't know.

 

But it was interesting to hear a very different perspective on optimism bias.

 

Yeah, so loads of correspondence.

 

Thank you very much, guys.

 

But that's it, so keep them coming in and we'll be back with you next week.

 

Bye for now.

 

Tom Pellereau:

Cheerio.

 

Rob Bell:

All music on this podcast series is sourced from the very talented Franc Cinelli.

 

And you can find loads more tracks at franccinelli.com.