We all make many decisions every single day. From little ones like what to eat for breakfast, to big ones like whether to take a new job. Given how regularly we’re deciding, we certainly have a vested interest in getting better at this skill. But how do we do so? How can we get better at making big choices, and spend less time dithering over the insignificant minutiae that often overwhelms our mental bandwidth? And why didn’t anyone teach us how to do this stuff to begin with?
My guest today has written a book that offers an education in a subject matter many of us missed out on. Her name is Annie Duke, she’s a former professional poker player and decision-making expert and strategist, and her latest book is How to Decide: Simple Tools for Making Better Choices. Today on the show, Annie shares many of those practical tools, beginning with how to overcome hindsight bias and “resulting” — our tendency to judge decisions based on their outcomes — by doing something called “knowledge tracking.” We discuss how to figure out the probabilities for things that seem difficult to predict and the importance of embracing an “archer’s mindset.” When then get into when you should make decisions slowly, when you can speed up, how to employ the “only option” test when making a choice, and why when a decision is hard, it’s actually easy.
If reading this in an email, click the title of the post to listen to the show.
Show Highlights
- Why don’t we teach decision making to kids and students?
- What is resulting? How does it get in the way of our making better decisions?
- How to fight hindsight bias with “knowledge tracking”
- Making a decision tree that actually works
- How to assign probabilities and make predictions when you have little to no information
- Why you need an archer’s mindset — or, why hitting the bull’s eye is overrated
- How better decision making can make you more resilient against poor decisions
- Applying the “only option” test as a way to choose between choices that don’t fundamentally matter
Resources/Articles/People Mentioned in Podcast
- My first interview with Annie on how to think like a poker player
- What Poker Can Teach You About Luck, Skill, and Mastering Yourself
- The Role of Luck in Success
- Can You Learn to Be Lucky
- 5 Tools for Thriving in Uncertainty
- Meditations on the Wisdom of Action
- How to Make Better Decisions by Thinking Like a Rocket Scientist
- How to Get Better at Making Life-Changing Decisions
- Using Mental Models to Make Better Decisions
- Assault Your Assumptions Through Red-Teaming
- How to Wrestle With a Difficult Decision: Advice From Sergeant Alvin C. York
Connect With Annie
Alliance for Decision Education
Listen to the Podcast! (And don’t forget to leave us a review!)
Listen to the episode on a separate page.
Subscribe to the podcast in the media player of your choice.
Listen ad-free on Stitcher Premium; get a free month when you use code “manliness” at checkout.
Podcast Sponsors
Click here to see a full list of our podcast sponsors.
Read the Transcript
If you appreciate the full text transcript, please consider donating to AoM. It will help cover the costs of transcription and allow other to enjoy it. Thank you!
Brett McKay: Brett McKay here and welcome to another edition of The Art of Manliness podcast. We all make many decisions every single day. From little ones like what to eat for breakfast, to big ones like whether to take a new job. Given how regularly we’re deciding, we certainly have a vested interest in getting better at this skill. But how do we do so? How can we get better at making big choices, and spend less time dithering over the insignificant minutiae that often overwhelms our mental bandwidth? And why didn’t anyone teach us how to do this stuff to begin with?
My guest today has written a book that offers an education in a subject matter many of us missed out on. Her name is Annie Duke, she’s a former professional poker player and is now a decision-making expert and a strategist, and her latest book is How to Decide: Simple Tools For Making Better Choices. Today on the show, Annie shares many of those practical tools, beginning with how to overcome hindsight bias, and “resulting,” which is our tendency to judge decisions based on their outcomes, by doing something called “knowledge tracking.” We then discuss how to figure out the probabilities for things that seem difficult to predict and the importance of embracing an “archer’s mindset” when making decisions. We then get into when you should make decisions slowly, when you can speed up, how to employ the “only option” test when making a choice, and why when a decision is hard, it’s actually really easy. After the show’s over, check at our show notes at aom.is/howtodecide. Annie joins me now via clearcast.io.
Annie Duke, welcome back to the show.
Annie Duke: Thanks for having me back. I’m so excited. This is… When we talked, what was it? Gosh, it was two years ago, right?
Brett McKay: Yeah, when your first book… 2019.
Annie Duke: Yeah, that was absolutely one of my favorite podcasts that I did last time, so I’m so excited to be back.
Brett McKay: Well, same here. Thinking in Bets is the book we talked about previously. It’s a book that I keep thinking about even though I read it several years ago and think about the ideas. You got a new book out though, it’s a follow-up called How to Decide: Simple Tools for Making Better Choices. And this book is, basically, it’s a workbook, I would describe it, of the tools you talked about in a very, I think, theoretical way in Thinking in Bets, but showing people how to be more explicit and in how to use them. And as I was reading this, I was thinking, “How come no one ever told me this stuff before?” Because we make decisions all the time, small ones, really big ones, but no one ever sits you down and be like, “Here’s how you can make a good decision.” Why is that? Why don’t we get taught explicitly how to do something we do every day.
Annie Duke: Yeah, so I think this is a very deep question. So just as background, I co-founded an organization called the Alliance for Decision Education, and we’re actually trying to tackle exactly this conundrum that you’re pointing out, which is, why don’t we teach decision education to K-12 students? When I talk to people and I ask them, “Did you ever have an explicit class on decision-making?” If anybody has, it would have been in college, and only if you were pursuing certain types of majors…
Brett McKay: I took a philosophy of decision-making class in college. Yeah.
Annie Duke: Exactly. So nobody really teaches you how to make a good decision, which is kind of strange, so I have some theories about it, so I’ll just throw a couple of things out there. One is, our educational system is set up actually, from way back when, from when England, obviously, was very colonialist, they had people who were far and wide and everybody needed to sort of learn very specific skills, like how to sail, so everybody had to be taught the same thing. And decision-making, actually, wasn’t something that they were trying to teach because they wanted people… People needed to all be doing kind of the same thing and have the same skills, so for example, trigonometry is in there, not just because it’s really good for sailing, but also because it’s hard and it doesn’t directly feel like it’s practical or connects to anything, so it was meant as a screener that would tell people you’re gonna go on to great things because you had the grit to be able to get through trigonometry, and all of you aren’t. And then that translated into American education with tracking. So trigonometry is literally a way to clear out the people who aren’t willing to work really hard at things that don’t have any purpose, which is kind of a weird thing to have in the school system. So our curriculum today is not really designed for today, it’s designed for a long time ago, I think that’s problem number one.
Problem number two, I think, is that it’s kind of like walking. You’ve been walking your whole life, and it would never occur to you that you should take a class on how to walk, and everybody has been making decisions their whole lives. And so the idea, I think, that you would maybe be bad at that, that it would be really good if you had a class that really taught you how to make a good decision, I don’t think that it’s really intuitive, I don’t think parents, in general, think they’re poor decision makers, I think that they probably think they’re pretty qualified to teach their children, and even when you look at the history of science, it wasn’t until Kahneman, Tversky and Richard Thaler came along and people like Barry Star, where they started saying people aren’t perfectly rational. If you give people the information that they need and let them make decisions, they actually aren’t going to make decisions that are necessarily really rational, and the ways that they’re irrational are actually quite predictable. And this was in the ’70s, this was heresy within science and economics. Up until then, the assumption was a rational actor. So we didn’t really even start to figure out the ways in which people are bad at decision-making until the ’70s, and then it wasn’t really widely accepted until, gosh, it started to gain some traction into the ’80s and ’90s.
And then, obviously, in the last two decades, people have really become wise to this and then you have… I think it was 2014, that Thinking Fast and Slow came out, and then the general public really started to get it. So we’ve been pretty behind the curve on this, and so the fact that it hasn’t gotten into the school system is maybe that, not that surprising. So that’s kind of what we’re trying to do at the Alliance for Decision Education, is kind of catch K-12 education up with where the science is, and actually, where business is. Because business has really accepted that this is something that they need to work on.
Brett McKay: Well, let’s talk to some of these tools you highlight in the book, and the first one, we talked about this a bit in Thinking in Bets, but I think it’s a really… When I learned about this concept, it’s changed the way I think about how I interact with the world and how I think about the world, and it’s this idea of resulting. So what is resulting, and then, how does that get in the way of us making good decisions?
Annie Duke: Yeah. So that is a concept that has really taken hold from Thinking in Bets and I’m quite pleased ’cause I think it’s a really important concept for to start understanding where our decision-making goes wrong. So what we wanna think about is how do we actually learn to become good decision makers, and it seems obvious that the way that we do that is from experience, so you make decisions, you get outcomes of the decisions and then you sort of tie those feedback loops together, then that helps you become better at making decisions. That would be what one would hope. But resulting actually really gets in the way. So this is what resulting is, it’s basically what it sounds like. You look at an outcome and depending on the quality of the outcome, was it good, was it bad, did you win, did you lose, you then use that outcome, the quality of the outcome to work back to the quality of the decision. So the decision that I open Thinking in Bets with is Pete Carroll in the Super Bowl in 2015, he’s obviously not playing, he’s coaching, and he’s against the Patriots, and there are 26 seconds left in the game, so obviously, it’s fourth quarter.
They’re on the one yard line of the Patriots, it’s second down, they have only one timeout. This is actually a really difficult situation for them ’cause they’re down by four, so they need to be able to score a touchdown, they can’t just kick a field goal, they obviously have three downs that they could do that in, second, third and fourth down, but they only have 26 seconds left, so this is quite a hard problem here, ’cause you have a clock management problem, given that you only have one timeout. So everybody expects Pete Carroll to have Russell Wilson hand the ball off to Marshawn Lynch, great running back. He doesn’t do that, he has Russell Wilson pass. Russell Wilson passes to the right corner of the end zone and the ball is intercepted by Malcolm Butler, and everybody goes nuts that this is absolutely the worst play in Super Bowl history.
In fact, USA Today, the headline that they had the next day was that it was the worst play call in NFL history, in all of NFL history. Now, this is a really classic case of resulting, because what you can do, if you go look at any of the articles that were written at the time, like the USA Today article, for example, it’s pretty statistics-free, so it doesn’t tell you what you need to know in order to determine whether that was a good decision or not, things like how likely was it that Marshawn Lynch was gonna score, or more importantly, how likely was it that the ball was gonna get intercepted. But I could tell you those things. Marshawn Lynch was gonna score about 20% of the time, it’s actually lower than people think it was. The ball was gonna get intercepted less than 2% of the time, but I don’t really need to do that. All I need to do is do a thought experiment with you, which is, imagine it’s the same situation, 26 seconds left in the Super Bowl against the dreaded Patriots, they’re on the one yard line, down by four, Pete Carroll has one timeout, does this really unexpected thing and he calls for a pass play, and the ball is actually complete for the game-winning touchdown. So they catch the ball, game-winning touchdown. I’ll just ask you, “What do the headlines look like the next day?”
Brett McKay: Greatest play. Gutsy. Amazing.
Annie Duke: That’s right. So all of a sudden, weirdly, USA today doesn’t say it’s the worst play in Super Bowl history. So in both cases, whether the ball is complete or intercepted, people make an assumption about what the decision quality is, if it’s intercepted, they say the decision quality is terrible, if it’s complete for the game-winning touchdown defeating the Patriots and denying them their fifth Super Bowl ring at the time. Then it’s the greatest play in Super Ball history, and that’s why he’s gonna go to the Hall of Fame. But here’s where we can see that this is an error, because the decision is the decision. There’s math that goes into it, I told you a little bit about it. Marshawn Lynch is only gonna score about 20% of the time, remember, he’s in a compressed part of the field on the one yard line, so there’s a lot of Patriots there to stop him, the ball is only gonna get intercepted less than 2% of the time. There are some other things that go into that, like if you pass, you’re more likely to get three plays off instead of two, which someone would assume you’d like against the Patriots, and those are the things that we should care about as we’re trying to determine what the quality of the decision is.
But the problem for us as decision-makers is that if I were to go through that, and I just went through a little bit of it, but if I were to go through the whole thing, it’s very complicated. You have to understand probability, statistics and probability, what that does in terms of win probability, depending on the choice that you make, you need some options theory in there so that you can understand what the value is of having the extra play and how you might actually get to that, so there’s a lot of conditionals in there as well. It’s just, it’s complicated. And this is sort of what we face when we’re trying to look back on our decisions. It’s complicated, that’s whether it’s the Super Bowl and the last play of the Super Bowl or trying to hire somebody based on just a CV, a few interviews and some references. These things are very complex. So in order to simplify, what we do is we say, “Well, I know what the result was, the ball was intercepted, so therefore it must have been a terrible play.” “I know what the result was, it was the game-winning touchdown, therefore it must have been a great play.”
And you can see why it has to do with that sort of how complex getting to the decision quality is because we don’t do this kind of resulting as much when the decision quality is really clear. Like if I go through a green light and I get in an accident, you don’t tell me going to a green light was a bad decision, but that’s because it’s super… Like we’ve already decided that, it’s the rules of the road. This is getting into two plus two equals four as opposed to some kind of strange like linear algebra or something like that. So it’s a way that we kind of simplify the world when we shouldn’t that really messes our decision-making up.
Brett McKay: Gotcha, so what resulting does, it prevents you from learning whether you’re actually making good decisions, you could be making the decision that’s terrible, like the process, what you’re thinking about is just absolutely terrible, but you get good outcomes because of just plain dumb luck, but you think to yourself, “Well, I’m making a great decision,” and really, eventually it’s gonna bite you in the butt, but you’ll never know that because you’re just looking at the outcome.
Annie Duke: Yeah, it’s actually, I would say, it’s actually worse than that. So you do some things, you get some great results from it, you decide that the decisions that led to that great result were amazing, then you do those things again, and maybe now you don’t get such great results from it, but now you get caught in motivated reasoning, where you start to say, “Well, I know the decision quality was good, so this must just be bad luck,” because it would be really, really hard for you to think that the success that you’d had in the past from the decision-making that you’d done previously was not actually because you make great decisions. That doesn’t really fit well with the way that we wanna think about ourselves, and we wanna think about ourselves in a positive way, so what kind of happens is that then part of what happens with motivated reasoning is that we’ll start to sort of look for reasons that we can maintain that our decisions were good, and this becomes really problematic ’cause we’re thinking about ourselves. So when we look at other people, we do pretty straight resulting, which is if there’s a good outcome, it’s from a good decision, if there’s a bad outcome, it’s from a bad decision. But when we’re actually thinking about ourselves, we have this real need to maintain a positive self-image.
And part of that, obviously, is, “I’m a good decision maker. I do good things and bad things aren’t my fault,” and because of the presence of luck, now we get into trouble, which is, we have some success, we believe that our decisions were good, maybe they’re very low quality though, and eventually they’re gonna, as you said, bite us in the butt, but when they start to bite us in the butt, we start to blame other factors.
Okay, so with resulting, we judge the quality of our decision based on its outcome, and part of overcoming that is trying to objectively separate out what was actually luck and what was skill in the decision. But then, there’s another bias connected to resulting, which is hindsight bias, and that’s where you think the outcome of something was more predictable than it was, and your memory actually can get distorted, so when you’re looking back on something, you think you really knew all along how it was gonna turn out, and there’s a tool you suggest using called knowledge tracking, that can help with both of these things. So can you walk us through knowledge tracking?
When you’re thinking about a decision, actually think to yourself, “What did I know at the time of the decision? What revealed itself after the fact? Those things that revealed themselves after the fact were they knowable beforehand?” If they were knowable beforehand, you still aren’t done though. You wanna move on to two other questions: Could I afford to get it? So as an example, I think I have an example in the book. You’ve only lived in the south your whole life, you get offered a great job in Boston and you’re trying to decide whether you’re gonna move there because you’re concerned about whether you’ll like the weather. You go up for a couple of days in February just to kind of check it out. It doesn’t seem so bad. The job’s a great opportunity, so you move there and it turns out that you hate it. Your first winter, it’s just like brutal and you end up moving back to the south. So this would be a good example where knowledge tracking would be really helpful, like, “What did I know at the time?” “Well, I knew that New England had bad winters. I wanted to try to figure out if I liked it or not. So I went up there for a couple of days in February, it didn’t seem so bad. And so I took the job.”
What revealed itself after the fact? “Well, it turns out that when I had to endure a whole New England winter, I hated it.” So now you can say to yourself, was that knowable beforehand? The answer is yes, I could have gone and spent a winter up in New England before deciding, except that I couldn’t afford to do that because the job wouldn’t have still been available to me. So that’s just kind of like, then you just sort of shrug your shoulders and notice that that’s a case where hindsight bias really happens, where your friends are gonna be like, “I knew you’d hate it there.” And you’re gonna say to yourself, “I should have known I would have hated it.” But this is actually a really helpful tool to sort of get you away from that, because what you can see is, “Well, of course, I couldn’t have known that I hated it because I couldn’t have been up in New England for a whole winter to be able to find it out.” Then you can ask yourself the next question, which is, “Even if I couldn’t have known it beforehand, either because it wasn’t knowable or because I couldn’t afford to go find this information out, could I use that in my decision process going forward?” And then another tool that you can use, and this is in retrospect, is actually to try to recreate for yourself what are the possible things that could have occurred.
So if you’re thinking about, for example, like the job in Boston, to actually go back and try to get yourself away from that feeling of inevitability that resulting creates and hindsight bias creates that, obviously, it was gonna work out horribly, and so therefore, I should have known it, which is kind of what ends up happening, and instead say, “Let me try to remember at the time that I was thinking about the job, what were all the different ways it could have turned out.” And you can think like, “I could have loved the job and become a winter nut who then goes skiing.” “I could have hated the job, but loved Boston so much that I ended up staying in Boston and I found another job.” “It could have been okay, the job could have been okay, and Boston would have been fine, and I could have spent a few years there, and ended up moving back on my own terms.” “I could have loved the job but hated the weather so much that I left,” or, “I could have loved the job, but hated the weather, but felt it was worthwhile to stay.” And we can see that once we start to do that, we start to realize like, no, the thing that happened wasn’t inevitable.
There were all sorts of ways that this could have turned out, so that’s kind of like… Those are the first three chapters of my book, of How To Decide, the follow-up, or trying to help you with these retrospective problem. How do I look back on a decision and actually start to dig down into the decision quality without falling into these traps where I start to do resulting or hindsight bias or whatever? And hopefully those tools are pretty clear, the knowledge tracking tool in particular I think it’s quite powerful.
Brett McKay: Yeah, I like that. I actually am gonna start doing that now that when I’m making a decision, like, “Here’s the information I’m using to make that decision.” Like actually write that down explicitly.
Annie Duke: Yeah, so that actually brings up a really good point. If we’re looking at something in retrospect where we don’t have any record of what we thought at the time, or what the information that we had was, or what the process that we went through, who we talked to, who we asked for advice, what they thought, any of that stuff, that then becomes really hard as we go back and try to do these reconstructions, we try to do this knowledge tracking and try to figure out what did we know at the time, or we try to reconstruct sort of these simple decision trees of like, “Here’s the decision I’m thinking about, what are the different outcomes that could occur?” It’s just hard to do in retrospect.
So the big lesson of that first section of the book is do it beforehand, that a really great decision process is gonna have you explicitly creating some sort of evidentiary record of what your beliefs are, what your rationale or thesis is for the reason that you’re thinking about doing something, what you think the different outcomes might be, how likely you think those are. It’s gonna have you interacting with other people to get their viewpoint on it, in order to improve the quality of the knowledge that’s going into the decision that you make. And that’s all gonna… You’re gonna have a record of that, so that when you do get the results you can look back and you can say, “What was my rationale? What did I believe at the time? What was the information that I had? What did I go find out?”
Right? And then you can actually ask yourself these questions in a much clearer way that’s gonna allow you to close these feedback loops in a more objective way. It won’t be a completely objective, but it’s gonna be a lot better than it would be otherwise. And this now allows you to create really good learning loops, that’s actually gonna improve your decision-making going forward in a much, much faster way. It’s gonna do a lot more heavy lifting.
Brett McKay: We’ve talked about analyzing past decisions so we can make better decisions, but then you also lay out how do you make a good decision that you haven’t made yet. And you basically lay out this process where you look at preferences, pay offs, and probabilities. You make a decision tree, so what you do is you say, “Here’s my decision”, and then you are gonna list out reasonably all the potential possible outcomes. So you can’t… Obviously you’re not gonna be able to do every single possible outcome, but the reasonable outcomes. So like the example you gave of moving from the South to Boston for a job. Possible outcomes… And you had to look at the pay offs, the up sides and the down sides, like the pluses and the negatives. So if you move to the South to Boston for this job, it could be you love the job, you love the weather; you hate the job, you love the weather; you love the weather, hate the job; you hate the job… That’s what we were kind of doing.
Annie Duke: Right.
Brett McKay: So you’re gonna do that, but I thought the really interesting part of this process is figuring out probabilities. And humans are really bad at this, for the most part, or describing probabilities. So okay, how do you do that? So how do you figure out with the decision of moving from the South to Boston, like the probability of whether you’re going to love the job and, you’ll love the weather, or love the job… How do you assign something for something you don’t even know?
Annie Duke: Yeah, so this actually brings up something really deep. The answer is, “You guess.” And it takes a particular type of mindset to be willing to do that, and the mindset is to say, “Look, no guess is ever super random, pretty much about anything.” It’s about figuring out, when we think about that… We have that distinction between a guess and an educated guess, it’s about, “How much educated can I get into it?” Because if I can get a little bit more educated into the guess than I would have if I hadn’t tried, that actually is gonna really improve my decision making.
So any decision is really just just a prediction about the future, right? If I’m thinking about moving to Boston I’m making some predictions about what’s gonna make me happiest in the future. Now, the future is always gonna be cloudy to us as mere mortals, but if you’re willing to make some guesses then you can get to a point where it’s less cloudy. And even though it’s not gonna be a perfectly clear picture where you’re gonna know exactly, “less cloudy” actually does a lot of work, it gets you pretty far.
So let’s think about what do I mean by this, that all guesses are educated guesses? That there really isn’t anything where you should say, “I don’t know”, period. Right? It should be, “What do I know?”, is really what you should be asking. So I’ll give you an example. Okay, so I have a new puppy, he’s sleeping next to me. You’ve never seen this puppy, correct?
Brett McKay: Correct.
Annie Duke: Okay, how much do you think puppy weighs?
Brett McKay: 15 pounds.
Annie Duke: Okay. And what’s the lowest amount you think the puppy weighs?
Brett McKay: Five pounds.
Annie Duke: And what’s the most amount you think the puppy weighs?
Brett McKay: 25.
Annie Duke: Okay, so that’s great. So you can’t see the puppy, we’re not on a video system, I haven’t showed you a picture of the puppy, you don’t even know what breed the puppy is.
Brett McKay: Don’t know, yeah.
Annie Duke: You don’t know how old the puppy is, you just know it’s a puppy.
Brett McKay: Right.
Annie Duke: So you know a little bit of something, about how old it is, but not a lot. And you just gave me a really good guess. And by the way, your lower bound and upper bound captured exactly the weight. So you gave a lower bound of five and an upper bound of 25. He’s 10 pounds.
Brett McKay: Okay.
Annie Duke: So you actually did your job, you found the right answer, it was in that range. Now, was your point forecast of 15 exactly right? No, but it was pretty darn close when you think about the weights of all things. If you thought, “What’s the full range of things could weigh?” It’s zero to… I don’t know, how much does the Earth weigh, right? Like millions of pounds, trillions of pounds.
So what we’ve just discovered is that when I asked you to guess at the weight of this dog, even though you’ve never seen the dog, you started, I assume, to recruit, “Well, what do I know about puppies?” Right? Like, “Well, it’s a puppy so would she describe a dog that’s over six months as a puppy? Maybe, but probably not.” I just got the puppy, so that means the puppy is probably young, right? How much do puppies in general weigh, how much do dogs in general weigh, how much do they compare to other things? And even if I asked you something like… Which I assume you don’t know the answer to, maybe you do, like, “What’s the distance between the Earth and Jupiter?”
Brett McKay: I have no clue, I would say two million miles.
Annie Duke: Right, so lower bound and upper bound.
Brett McKay: Yeah, lower bound, about 2 million. Upper bound, 10 million.
Annie Duke: Okay, so…
Brett McKay: [chuckle] Do I just sound like an idiot?
Annie Duke: I would probably… So, I know a little something more than you do, right? So I know that the sun is 93 million miles away.
Brett McKay: Oh, wow.
Annie Duke: Right, but notice though, even so, you knew it had to be in the millions. So, did you get it exactly right? No, but you cleared away a lot of the possibilities, because you understood that it at least had to have a million in front of it. And that’s actually a really big improvement over not having tried it at all. So when you think about what’s the likelihood that I’m gonna enjoy the weather in Boston, the answer isn’t, “I have no clue”, because you know things about yourself, you’ve experienced some cold weather, maybe you went up there in February for a couple of days, so you tried to get some more educated into the guess and you didn’t think it was so bad.
So, when you go up there you recognize, “Okay, I think that I’m probably likely enough to like it, that I’m willing to do this.” And are you gonna get an exact answer? Of course not. But if you can get close, it matters. So I talk about this as “the archer’s mindset”, that we’re really focused on the bull’s eye and we feel like we’ve gotta get to the bull’s eye, and what that causes us to do is either claim we have the bull’s eye when we don’t, right? Or not try at all, because we recognize we won’t be able to hit it. And instead we need to have more of an archer’s mindset, to say, “In archery you have a target, and while you might be aiming at the bull’s eye, you also get points for hitting the target.” And so we wanna realize that in decision-making we get points for hitting the target, and then beyond that we get a lot of points for actually defining the size of the target, because that does a lot of work for us.
So like in the puppy example, you said your lower bound was five pounds and your upper bound was 25 pounds for the weight of this puppy, right? Well that tells me something about how uncertain you are. When I asked you for the lower bound on Jupiter and the upper bound on Jupiter, this was a much wider range. Why? Because you’re less certain, you have less knowledge about the relationship between Earth and Jupiter than you do between the lower bound of the weight of this puppy and the upper bound of the weight of this puppy, which you know a lot more about. We can take that to the extreme. If I asked you what’s your birthday, you would give me the bull’s eye, because you know it.
Brett McKay: Gotcha. Yeah, so you know more than you think you do.
Annie Duke: Right.
Brett McKay: Gotcha. Alright, so yeah, so you assign a probability, and you even recommend, don’t just settle for likely or not likely, actually put a percentage on it, ’cause that’ll make things more concrete for you.
Annie Duke: The reason why we don’t wanna use words that… So there’s all these words that describe probability that we use every day. Like “likely, always, never, real possibility”, like we say that, “Do you wanna go out to the movies this weekend?” “Yeah, I think that’s a good possibility.” So that would be a way that we throw this around, but you can think about we do this like in a hiring process, right? Like, “What do you think of that candidate?” “I think there’s a good possibility they’d be great.”
So we use these kinds of terms all the time. Well, the reason why we don’t want to is kind of twofold. One is that we wanna be able to circle back and actually close those feedback loops, and if we use these kind of mushy terms it’s hard for us to do that, because these terms have pretty broad meanings. And when you actually ask people, like if you surveyed people on something like “real possibility”, which is a term we use a lot, and you say, “Hey, when you use that word, what probability do you actually intend?” You get a range from about 15% to 90%, so that should be the first clue that there’s a problem with those words. So the first has to do with that feedback loop, it’s that when you go back and you try to close your own feedback loops and you’ve said that something’s a real possibility, you can kind of mush around in there in order to… Motivated reasoning is gonna get there.
Right? You can say, “Well I said it was a real possibility, I didn’t say it was… ” For example, if I hire someone and they turn out to be great I get to say, “Yeah, I told you it was a real possibility they would be great.” And then if I hire someone and they turn out to be poor, I get to say, “Well, I told you it was a real possibility they’d be great, but I didn’t say it was 100%”, right? So okay, what does that help you with? So that’s kind of problem number one that it doesn’t help you with.
Problem number two that it doesn’t help you with is actually a very broad problem in decision making, which is… So when we think about problems like confirmation bias, which is just, “I notice information that confirms the things that I believe to be true, and I don’t notice information that dis-confirms the beliefs that I have.” Or something like availability bias, which is that I judge things to be more frequent that I have interacted with quite a bit, or that are more vivid for me to recall. So when we take those, what you can see is those are all “me” problems, right, they’re all things that have to do with me trying to affirm my beliefs or quirks of my own memory or my own experiences, the way that I’ve interacted with the world and what my motivations are about, the way that I’m reasoning about the world.
So Kahneman would call this “the inside view”, that when we’re reasoning about the world we reason about it from the inside view, in other words, driven by our own experiences, our own knowledge, our own perspectives on the world and the mental models that we apply to the world. And that’s actually where most of the bias is living, is in the inside view. So the antidote to the inside view is the outside view, which is essentially one of two things, it’s what’s true of the world in general.
So an example of that would be a base rate, which is just how often does something happen in a situation similar to the one that I’m considering. So I’ll give you an example of a base rate. So let’s say it’s when Coronavirus doesn’t exist yet, and you’re thinking about opening a restaurant in a certain area and you think that you have a 90% chance of being successful by the end of the first year. So that would be your guess, inside view, you think pretty well of yourself, you probably have some over-confidence, you’re probably cherry-picking some data that’s kind of getting you to that conclusion, again, not on purpose, but because that’s what we naturally do. But the base rate, this would be getting to the outside view, would be to say, “Well, how often do restaurants succeed within the first year in general in my area?” And if you look that up, what you would find out is that the percentage of restaurants that are open after the first year is 40%.
So we can see how that helps to discipline the inside view. If I think it’s 90% and the world says it’s 40%, I ought to rethink my 90% number. So that’s one way to do it, but another way to do it, and this is a great way to do it, is to actually get other people’s perspectives on your situation, because other people can be looking at your situation and they can think very different things than you do about it. This is even if they have the exact same data, they may model the data differently than you. This is even if they have modeled the data exactly the same as you, they may think that you’re supposed to do different things about it, given what the model tells you. So it’s really, really good to get other people’s perspective on the situation that you’re considering, on the decision that you’re considering.
Well, in order to do that you have to actually communicate clearly to other people what it is that you think, what it is that you believe to be true of the world. And this is where terms like “real possibility” really become a problem, because if I tell you that something is a real possibility it’s so unclear that we know what that means. You might think it means 20% and I might think it means 60%, and we could think that we totally agree that it’s “a real possibility” this candidate could do well, and you may be thinking it’s a 20% chance and I may be thinking it’s a 60% chance, and we actually disagree. But we can’t find it out because we haven’t expressed what we believe with any type of precision.
So that’s where this idea of giving essentially a bull’s eye estimate, which would be like, “I think it’s a 55% chance”, and then giving a lower and upper bound now becomes really valuable, because it’s what gets people involved in the conversation, right? So if I give the bull’s eye estimate, like “I think it’s a 55% chance”, you know exactly what I mean and you know whether you agree with that. And then when I give the lower and the upper bound I tell you how certain I am about it. So I’m giving some sense to you of what my target area is.
And what’s really wonderful when you do that, when you say, “Well, I think the puppy is 15 pounds, with a lower bound of five pounds and an upper bound of 25 pounds”, is that you have said very clearly what your beliefs are in a way that you have actually invited me into the conversation. Inherent in that lower and upper bound is a question of, “Can you help me with this? I’m telling you how much certainty I have, I’ve obviously thought about it, I’m telling you something that’s quite precise, and is there a way that you could help me narrow the range?” And that creates really great decision-making, great conversations, because you actually are very clear about what the conversation is, and you’re maximizing your access to the outside view by doing so.
Brett McKay: So this is a pretty involved process, and I imagine when you first start doing it it’ll take a long time, but I imagine the more you do it it becomes a skill, it becomes intuitive. So we’ve talked about how to improve the quality of our decisions, the other issue with decision-making that people have is the amount of time, like the bandwidth people spend on making… They just agonize. Paralysis by analysis. But one of my favorite sections, you give some sort of hacks to short-circuit that analysis by paralysis. Can you share a few that you think are really powerful that people can start using today and actually see a profound change in how much time they’re spending on decisions?
Annie Duke: Yeah, absolutely. Yeah, so I wanna say, when I talk about this stuff and I say like, “Oh, you should build out these decision trees and you should do this knowledge tracking, and you should be thinking about the probability of different things happening”, the response is like, “How am I ever gonna make a decision again? This is gonna make me go so slow.” And I just wanna remind people that I was a poker player, and obviously at the poker table you’re making decisions very, very quickly, and you’re iterating from a lot.
I obviously don’t think that you need to go really slow on every single decision, so a couple of things on that, one is that if you do understand what a robust decision process looks like, this is actually gonna help you speed up your decisions, ’cause you’re gonna be able to hone in on the things that matter instead of spending your time spinning your wheels thinking about things that don’t matter, so this just is more efficient because it tells you what you should care about, number one.
Number two is kind of like with riding a bike, you have to kind of understand it in a slow way, or driving a car, like, “This is what the gas peddle does and this is what the break pedal does. And if I turn the steering wheel this way this is what happens and… “, before you can actually put that into a more automatic, quicker process. So understanding what a really robust decision process looks like will tell you what the heart of the matter is, but it will also help you to speed up, just because you understand what it would look like in its fullest form.
But as far as most of the decisions that you’re making, generally, we kind of a little bit get it backwards. With some decisions that we should be taking quite a bit of time on, we’ll often just go really fast. I think partly because we know it’s complicated and so we sort of give up, in that sense of, “I don’t wanna guess ’cause I don’t know, and so therefore I’m just gonna go with my gut.” And there’s a wide variety of the decisions where we actually go quite slowly, where we should actually be speeding up, and it’s because we’re not thinking about the type of decision that we’re facing very well.
So in order to figure out when you can go fast, what you’re essentially figuring out is, “If I go really quickly the time that I’m saving comes at a cost, and that cost is that I’m probably gonna increase my error rate.” So the decision is just gonna be less exact, it’s gonna be less accurate. And if I increase my error rate that means that I may get a bad outcome more often than I would have if I would have taken more time. And so that once we sort of understand that, that there’s this trade-off between time and accuracy, then we can figure out when we can go fast and when we should slow down. And it’s when we can tolerate higher probability of a bad outcome. So let’s think about when we can do that, ’cause that’s kind of the broad framework that we wanna think about it through.
And we can think about this through two things, one is impact and one is option. So I’ll give you an example of a type of decision that people take a very long time with, and I’m sure you’ve experienced this. So when we all used to go to restaurants, you probably know the person who would sit with the menu and they’d be like staring at the menu, asking the wait staff for their opinion, asking every single person at the table what they were gonna order, agonizing over it, and then once the wait staff came over to take the order they’d be like, “Let me go last.” Do you know that person?
Brett McKay: Of course, yes.
Annie Duke: Is it you maybe?
Brett McKay: No. It’s not me.
Annie Duke: You did philosophy, I’m guessing you go faster.
Brett McKay: Yeah, I go really fast. I just don’t care.
Annie Duke: Yeah, but you know that person. And those people are very, very common. So it turns out that actually, if you look at the statistics, that when you take together what to wear… Which I think is probably a faster decision in a pandemic, ’cause it’s like sweat pants and then something that looks decent on top, but whatever. But what to wear, what to watch on Netflix, and what to eat, that people are taking about six to seven hours of work week time per year on those decisions. That’s pretty surprising.
Brett McKay: Right, that’s a lot of time.
Annie Duke: That’s a lot of time. And I believe that the reason that they do that actually is related to what we talked about with why people sort of guess in these situations where it’s really complicated, or are unwilling to guess, they just go with their gut. And it’s because when you’re thinking about something like ordering off a menu, I think it feels very solvable, in kind of the “know thyself” sense. You should be able to figure out what own preferences are, and you know a lot about food, and if you just asked the wait staff and you looked at a few more pictures of the dishes on Yelp, that you should be able to get this decision, and I’m gonna put it in air quotes, “right”.
Because it feels like a pretty simple decision that’s about your own preferences, and you should be able to get it right, and you’re kind of in fear of that moment where you’ve tried to decide between the chicken and the fish and you get the fish and it’s yucky. And what do you say when that happens, immediately, “I made a mistake. I should have ordered the chicken.” That’s what you say immediately. But if we go back to that hindsight bias and resulting problem, that’s just hindsight bias and resulting.
Because obviously, less a time machine, there’s no way for you to know that that fish wasn’t gonna be very good. What you knew is that you like chicken and fish, they both seem pretty good to you, you looked at the preparations and whatever, so it’s weird to call that a mistake when the food comes back poorly. And it’s weird to say, “I should have ordered the other thing, or I should have known to do that”, but that’s what we do. So we’re thinking very short term. So what we wanna do is actually think about what is the long-term impact of that decision going awry. So I’ll just ask you, so you get crappy fish, we have a meal and the fish is yucky and you’re sad ’cause it was gross. And now I catch you in a year. So it’s a year later, and I say to you, “Brett, how’s your year been?” And I’ll just ask you that, like, “How’s your year been?”
Brett McKay: It’s been… All things considered. It’s been pretty… It’s been all right.
Annie Duke: Yeah, all things considered, that’s my answer too, “All things considered, it’s been okay. So that… Do you remember that fish that you had when we were in that restaurant a year ago, and it was kind of gross?”
Brett McKay: I don’t even remember that. Right? What are you talking about?
Annie Duke: Right. Does it have any effect on your happiness today?
Brett McKay: No, not at all.
Annie Duke: Outweighs the pandemic, right?
Brett McKay: Right.
Annie Duke: So, right, so what if I catch you in a month?
Brett McKay: Same thing, I would have been like, “We had… I don’t remember”, I would have forgotten about it, or I just wouldn’t even been thinking about it. Yeah.
Annie Duke: Or, how about a year? How about a week? You’ve had 21 meals since then.
Brett McKay: No, wouldn’t even have been thinking about it. No, no, if it was really expensive I still… I might be a little… Have a little…
Annie Duke: Miffed?
Brett McKay: Yeah, miffed about it, but…
Annie Duke: So this particular exercise of, “Does it affect my happiness in a year, does it affect my happiness in a month, does it affect happiness in a week?”, it’s called “The Happiness Test”. And the reason that we wanna do this, this is a way for us to go have a conversation with the future version of ourself, so that the future version of ourself can say, “Hey, by the way, that decision makes no difference to you. You can get bad fish, I don’t care. Here I am, a year later, it doesn’t matter to me.”
And happiness here is meant as a proxy for whatever the goals are that you’re trying to achieve. Happiness, assuming when we reach our goals, one assumes that we’re happier, so that’s why I use “Happiness” as kind of a proxy. So this is a really good test to apply to figure out if I can go fast or I can go slow, so when you feel yourself hung up in that decision and you’re taking a lot of time with it, just say, “Am I gonna care about this in a year, am I gonna care about this in a month, am I gonna care about this in a week?”, and the sooner that you’re not gonna care about it, whether it turns out well or poorly, because if the fish is great and I see you in a year, and it also didn’t affect your happiness at all, this is just low impact all around, the shorter the time period in which you’re not gonna care how it turns out, the faster you should go.
So this is the first thing. This has to do with impact. This decision is low impact. Good or bad, it doesn’t matter. Now, in the case of ordering off a menu, it’s particularly a low-impact ’cause it’s also what we would call a repeating option. So remember I said to you “When I see you in a week, you’ve had 21 meals since then”, assuming you eat three times a day. So that’s what I’m referring to there, is it’s a repeating option, so even if your lunch is bad you get to go have something for dinner. So you basically get to try again pretty quickly, and that’s gonna be true also of what to wear or what to watch on Netflix. Dating, right?
If a date goes poorly, so what, it’s just a date. And it’s a repeating option. You can get right back on Tinder or Bumble or whatever your app is that you like, and you can click and go on a date with somebody else. So that’s also a repeated option. Choosing classes in college, it’s a repeated option, you get to do it a lot. So when we’re repeating options we can go a little bit faster with those decisions. And when they’re low impact we can go pretty fast, so those are kind of the two things on the impact side. Now, there’s another framework that we wanna think about, which is optionality, which has to do with how easy is it for me to quit the thing that I’m doing and go and do something else.
So people may have heard Jeff Bezos talking about type 1 or type 2 decisions, or two-way door decisions versus one-way door decisions, and this is basically what he’s getting at. The easier it is for me to quit something, the faster I can go, because I’m gonna be more tolerant of getting a bad outcome because I can switch. I can just quit and go do something else. So this is actually a really, really important concept for great decision making. So we can apply that, if you’re in a hiring situation, the difference between hiring an intern versus hiring someone who’s quite senior, it’s gonna be harder on the company, on you, to unwind a relationship with someone who’s senior. So that is a less reversible decision, it also happens to be higher impact. So we’ve got both of those things working together. Whereas an intern, if the intern doesn’t work out well that’s relatively low impact, but it’s also very easy to unwind. It’s not hard to part ways with an intern in the same way that it can be very difficult to part ways with someone who’s quite senior. So that there we can sort of see how should we be spending our decision-making time, and what you’ll see, people will spend…
Like they’ll find a couple of interns who look like they might be really great for the one job, and they’ll just agonize over that decision, “Which one should I choose? I don’t know, I don’t know what to do”, but this is a decision in which you actually should be going pretty fast on, certainly compared to how much time you might spend on the more senior person. So that… You can see how this kind of allows us to allocate our time, right? Dating versus marrying is another example of that. It’s pretty easy to quit a date, people do it sometimes in the middle of the dates. People do it sometimes before they actually sit down, they look across the room, I don’t think that’s very nice but people will do that, I don’t recommend it. I think you should at least talk to the person. But marrying, obviously that is much harder to unwind, it’s more difficult to actually quit that decision.
So that’s one of the things that we wanna think about, and then you can start to take that and say, “Well now I could actually take that into my decision-making life and use that as a strategy”, this would be called Decision Stacking, which is when I think that I’m gonna be facing a very big decision, what could I do that would give me a lot of information that would help me to improve the quality, to improve the educated guesses I’m making that are going into that bigger decision, what could I do now that’s pretty low impact and pretty reversible that would help me with that bigger decision?
So as an example, if you’re thinking about moving to a new city or moving to a new neighborhood, renting first before you buy, that’s an example of decision stacking. Agile software development is an example of decision stacking. I’m not gonna do a large batch software development where I’m gonna have to roll this out to my whole customer base all at once, I’m gonna do a small test that’s pretty beta to a few people and see how they like it, that little handful of people. And then if it doesn’t work out it’s not a big deal, I can just roll the code back. I didn’t somehow upend my whole customer base, so you’re lowering impact of making it much more reversible.
Another really good example of this type of decision stacking strategy would be pop-up stores. If I’m thinking about releasing a new consumer-packaged good, for example, and I don’t know whether people are gonna respond to it and I’d like to test it maybe in a new city or something like that, I don’t wanna sign a year-long lease or try to put this across all Whole Foods or something like that, I can just do a pop-up store where the impact of that not working out is not so great. If I do find something good about it, then that’s awesome. I can get some information out of that, and it’s very easy to reverse and shut down, because I haven’t made any long-term commitments and I’m doing this in a very small way. So this becomes actually really powerful that it doesn’t just help you to figure out how to speed your decisions up, but it also gives you a decision strategy for improving the quality of those long-term commitments that you’re gonna have to make by trying to figure out how to do some lower-impact, more reversible decisions beforehand, in order to get you the information that you want for that longer term decision, which is basically what dating is. It’s decision stacking.
Brett McKay: So another tool you use, I started using right away after I read about it, is if you’re agonized between two or three decisions, you give the examples like, “Well should go to Paris or Rome, which one should I do?” And you just make this super simple way to frame, it’s like, “Well, if you could only go to Rome, would you be happy?” Yeah, I would be happy if that’s the one place I could go this year. “Well, if you could go to Paris would you be happy if that’s the only place you could go this year?” Yeah, that would… I’d be fine. I was like, “Well, then either one would be a good decision.” So just flip a coin basically.
Annie Duke: Yeah, so people actually will ask me like, “When is it okay to go with your gut?” and this is the situation where I say it’s fine. And the reason why it’s fine is ’cause it doesn’t matter. So I’m a big fan of using your gut to make decisions when it makes no difference, so use your gut to order off a menu is totally fine, but this is a good example. So obviously if you’re thinking about a European vacation, you’re trying to decide between Paris or Rome, this doesn’t pass the happiness test. If you have a crappy Rome vacation and I see you in a year, it probably did affect your happiness over the year. It’s not a repeating option for most people, they can’t just go on another vacation in the next month, right? This clearly has a big downside, it’s very expensive. You can’t really reverse it. “Oh, I don’t like Rome, let me go somewhere else immediately, I’ll just abandon and be somewhere else”, it doesn’t sort of satisfy all that stuff that would tell you that you can go fast. This is clearly something that’s very high impact.
But what happens to us is that when we sort get into these decisions and we get a couple of options, and the category of the decision is high impact, “My big vacation for the year”, this is a very high impact, now we sort of get down into, “We’ve got two options that we’re trying to decide between in that category of thing that’s gonna really matter, and now… ” You know what everybody does, it’s like you’re on Trip Advisor, you’re looking at every single review, you’re asking anybody you know who’s been to either Paris or Rome, and it’s just total anxiety.
But what you just pointed out with the only option test is that what’s hanging you up about this decision, what makes this decision feel so hard, is that the options are, from your vantage point, identical. And what I mean by “from your vantage point” is your vantage point where you aren’t omniscient and you don’t have a time machine. This is very, very important. You cannot see how those vacations would go, in some kind of perfect sense where the image is crystal clear. It’s gonna be cloudy, the same as the Boston problem. You don’t have all the information you need about either place, but from the standpoint of the information that you do have and what you know about your preferences and what you can afford and the time that you have to go, Paris and Rome are kind of identical to each other, given the acuity that you have on this issue. Right?
So they’re the same. And the way that you can find out there the same, it’s through the only option test. If Paris were the only option that I had, would I be really, really happy with it? Of course, the answer is yes. If Rome were the only option that I had, would I be really, really happy with it? Of course, the answer is yes. So this brings up a really important decision process, which is when a decision is hard, it means it’s easy. Meaning when a decision is hard in this particular way, that you have two options that you can’t decide between, what that means is it’s really easy, because what that is telling you is that the options are identical, that there is really no difference between the two. So whichever one you choose, it’s probably a pretty good option, which is what you get to through the only option test.
So we can go back to that intern problem. You have two interns that you’re thinking about for one job, they both seem really great, and now you’re agonizing about which one you should hire. But you should just step back and say, “If intern A was the only person I could hire, would I be ecstatic to have this person as the person to fill the job? Yes. If intern B were the only person that I had available to hire, would I be ecstatic to have that person in the job? Yes.” Okay, well then you’re done, and you can flip a coin, you can go with your gut, you could ask someone else to decide, I don’t care. But don’t take any more time on the decision.
And then we can sort of take this and go back to this, we can take a step back and say, “What is this telling us?” And what it’s telling us is that decisions are generally thresholding problems, that there’s a sorting process which is, “Of all the options that I have available to me, which of them satisfies the requirements that I have for thinking that this is something that I would want to choose?” So that’s the sorting process, which means I need to get it above a certain threshold where this is gonna be reasonable for me to choose. In the case of a European vacation it’s, “This is the amount of money that I have to spend. I’d like there to be great architecture, I’d like the food to be amazing, I want there to be history, I’d like it to be a place where I can walk everywhere”, as an example.
Okay, so now you figured out “Here are my requirements, I wanna look at the options that I have of places that I can go and figure out what meets that threshold.” Once something is above that threshold you’re done with the sorting and now you’re in the picking part, picking between options that all have met the threshold, that have all met that sorting process and satisfied that sorting process. And once you’re in the picking part of the decision, flip a coin.
Brett McKay: I love it. It’s really, really helpful. Well Annie, this has been a great conversation, where an people go to learn more about the book?
Annie Duke: Okay, so annieduke.com is a great place to go find out about me, because that has kind of all things Annie Duke on it. You can find my books there, you can find video of me talking, you can get links to podcasts that I’ve done, there’s also a contact form. I actually love hearing from people who’ve heard me speak or have read my work. In fact, How To Decide was born of conversations with people who contacted me, because what I found after I put out Thinking In Bets, was that people were asking me, “Okay, I see what you were saying in Thinking in Bets about uncertainty and the way it really frustrates our decisions, how would I actually make great decisions? What are the tools that I could use? What would the process look like? How can I think about these things in a clearer way?”
And I just realized they were asking me to write something that was more how-to. And so I did, so I find it very helpful when people actually reach out to me, so please don’t be afraid to do that. So that’s one place you can find me, I’m on Twitter @annieduke. And then the last thing is I would love it if people would check out the Alliance for Decision Education, kind of rolling back to the beginning of the conversation.
We really think it’s an emergency right now to get decision education into K-12. And when we think about the information ecosystem that we live in and how much information is coming in, and there’s disinformation and a real ability to fall into serious echo chambers, become extremized, this necessity to be able to navigate the world and figure out what’s true and then figure out what to do about it, in both the information ecosystem that we’re living in but also right now, when we think about what’s happening in terms of career trajectories and technology and what jobs are gonna look like in five years or 10 years, there’s so much around that and it’s changing so rapidly that equipping our youth to be able to sort of navigate that changing landscape we think is just incredibly important. And as you pointed out, this is not something that’s taught in school, and so at the alliance we’re really trying to change that. So I would love it if people would check the Alliance for Decision Education out.
Brett McKay: Fantastic. Well, Annie Duke, thanks for your time, it’s been a pleasure.
Annie Duke: Well, thank you so much for having me back.
Brett McKay: My guest today was Annie Duke, she’s the author of the book, How to Decide. It’s available on Amazon.com and book stores everywhere. You can find out more information about her work at her website, annieduke.com. Also check out our show notes at aom.is/howtodecide, where you can find links to resources where you can delve deeper into this topic.
Well, that wraps up another edition of The AOM Podcast, check out our website at artofmanliness.com, where you find our podcast archives as well as thousands of articles written over the years about pretty much anything you can think of. And if you’d like to enjoy ad-free episodes of the AOM podcast, you can do so at Stitcher premium. Head over to stitcherpremium.com, sign up, use code MANLINESS at checkout for a free month trial. Once you’re signed up, download the Stitcher app on Android or iOS and you can start enjoying ad-free episodes of the AOM podcast. And if you haven’t done so already, I’d appreciate if you take one minute to give us a review in Apple podcast or Stitcher. It helps out a lot. And you’ve done that already, thank you. Please consider sharing the show with a friend or family member who you think would get something out of it. As always, thank for the continued support. Until next time, this is Brett McKay, reminding all of you listening to the AOM podcast, put what you’ve heard into action.
