Bayes and What He Hath Wrought

Thomas Bayes was an 18th century British statistician, philosopher and Presbyterian minister. He’s known today because he formulated Bayes’ Theorem, which has since given rise to Bayseian probability, Bayseian inference, Bayseian epistemology, Bayesian efficiency and Bayseian networks, among other things.

The reason I bring this up is that philosophers, especially the ones who concentrate on logic and the theory of knowledge, often mention something Bayseian, usually in glowing terms. It’s been a source of consternation for me. I’ve tried to understand what the big deal is, but pretty much failed. All I’ve really gotten out of these efforts is the idea that if you’re trying to figure out a probability, it helps to pay attention to new evidence. Duh.

Today, however, the (Roughly) Daily blog linked to an article by geneticist Johnjoe McFadden called “Why Simplicity Works”. In it, he offers a simple explanation of Bayes’ Theorem, which for some reason I found especially helpful. Here goes:

Just why do simpler laws work so well? The statistical approach known as Bayesian inference, after the English statistician Thomas Bayes (1702-61), can help explain simplicity’s power.

Bayesian inference allows us to update our degree of belief in an explanation, theory or model based on its ability to predict data. To grasp this, imagine you have a friend who has two dice. The first is a simple six-sided cube, and the second is more complex, with 60 sides that can throw 60 different numbers. [All things being equal, the odds that she’ll throw either one of the dice at this point are 50/50].

Suppose your friend throws one of the dice in secret and calls out a number, say 5. She asks you to guess which dice was thrown. Like astronomical data that either the geocentric or heliocentric system could account for, the number 5 could have been thrown by either dice. Are they equally likely?

Bayesian inference says no, because it weights alternative models – the six- vs the 60-sided dice – according to the likelihood that they would have generated the data. There is a one-in-six chance of a six-sided dice throwing a 5, whereas only a one-in-60 chance of the 60-sided dice throwing a 5. Comparing likelihoods, then, the six-sided dice is 10 times more likely to be the source of the data than the 60-sided dice.

Simple scientific laws are preferred, then, because, if they fit or fully explain the data, they’re more likely to be the source of it.

Hence, in this case, before your friend rolls one of the dice, there is the same probability that she’ll roll either one. With the new evidence — that she rolled a 5 — the probability changes. To Professor McFadden’s point, the simplest explanation for why she rolled a 5 is that she used the dice with only 6 sides (she didn’t roll 1, 2,3, 4 or 6), not the dice with 60 sides (she didn’t roll 1, 2, 3, 4, 6, 7, 8, 9, 10, . . . 58, 59 or 60).

Now it’s easier to understand explanations like this one from the Stanford Encyclopedia of Philosophy:

Bayes’ Theorem is a simple mathematical formula used for calculating conditional probabilities. It figures prominently in subjectivist or Bayesian approaches to epistemology, statistics, and inductive logic. Subjectivists, who maintain that rational belief is governed by the laws of probability, lean heavily on conditional probabilities in their theories of evidence and their models of empirical learning. Bayes’ Theorem is central to these enterprises both because it simplifies the calculation of conditional probabilities and because it clarifies significant features of subjectivist positions. Indeed, the Theorem’s central insight — that a hypothesis is confirmed by any body of data that its truth renders probable — is the cornerstone of all subjectivist methodology. . . .

To illustrate, suppose J. Doe is a randomly chosen American who was alive on January 1, 2000. According to the United States Center for Disease Control, roughly 2.4 million of the 275 million Americans alive on that date died during the 2000 calendar year. Among the approximately 16.6 million senior citizens (age 75 or greater) about 1.36 million died. The unconditional probability of the hypothesis that our J. Doe died during 2000, H, is just the population-wide mortality rate P(H) = 2.4M/275M = 0.00873. To find the probability of J. Doe’s death conditional on the information, E, that he or she was a senior citizen, we divide the probability that he or she was a senior who died, P(H & E) = 1.36M/275M = 0.00495, by the probability that he or she was a senior citizen, P(E) = 16.6M/275M = 0.06036. Thus, the probability of J. Doe’s death given that he or she was a senior is PE(H) = P(H & E)/P(E) = 0.00495/0.06036 = 0.082. Notice how the size of the total population factors out of this equation, so that PE(H) is just the proportion of seniors who died. One should contrast this quantity, which gives the mortality rate among senior citizens, with the “inverse” probability of E conditional on H, PH(E) = P(H & E)/P(H) = 0.00495/0.00873 = 0.57, which is the proportion of deaths in the total population that occurred among seniors.

Exactly.

What Should Sleeping Beauty Say, Logically Speaking? (Part 2)

In yesterday’s post, I described the so-called “Sleeping Beauty Problem”. Mathematicians and philosophers have been debating this problem for the past 15 years (most recently in response to an article at the Quanta Magazine site). What should Sleeping Beauty say when she wakes up on Monday or Tuesday and is asked the following question about a coin flip that happened on Sunday: “What are the odds that the coin we flipped on Sunday came up heads?”.

To review: Sleeping Beauty doesn’t know what day it is when she wakes up or whether she’s already woken up during the experiment. But she is told that if the coin flip on Sunday came up heads, she’s only being awakened on Monday. If, however, the coin came up tails, she’s being awakened on both Monday and Tuesday. “Halfers” think the odds that the coin came up heads is 1/2. “Thirders” think the answer is 1/3.

Here’s the answer Pradeep Mutalik, the author of the Quanta article, gave in response to a whole lot of comments:

This is the crux of the idea: that there are two propositions in the Sleeping Beauty problem:

1) the probability of heads when the fair coin is tossed, which is obviously 1/2. This is the proposition modeled by halfers.
2) the probability of heads encountered later, which thanks to the experimental conditions that are asymmetric between heads and tails, is 1/3. This is the proposition modeled by thirders.

… the same problem statement can mean tossed if understood one way and encountered if interpreted another way.

Once you accept that there are two valid interpretations, the solution of the Sleeping Beauty problem can be expressed in one sentence:

A fair coin is tossed: The probability of it landing heads when tossed is ½; the probability of Sleeping Beauty encountering heads later, thanks to the asymmetric experimental conditions, is 1/3.

And that, in a nutshell, is all there is to it.

Mr. Mutalik also added this note in response to my own comment on his article, in which I came out in support of the 1/3 answer:

As in my reply to Rich, your answer is correct for the heads “encountered” probability. The heads “tossed” probability is, and remains ½ until SB learns the actual result of the coin toss.

The point of all this is to show that there is no incompatibility in the thirder and halfer arguments. The same person can simultaneously be both a halfer and a thirder.

As a new supporter of the thirder position, I’m not convinced by Mr. Mutalik’s response. As I wrote in the previous post, it’s clear that the odds of getting heads or tails when a coin is flipped are always 1/2 heads and 1/2 tails. That’s how coin flips work (putting aside the extremely rare occasions when they land on their edge and don’t fall either way). 

But the question presented to Sleeping Beauty is: “What are the odds that the coin flip came up heads?” Since she knows how the experiment is being conducted when she’s asked this question, shouldn’t she conclude that the odds are 1/3, not 1/2? After all, she knows that a coin that came up tails means this could be either Monday or Tuesday. If it came up heads, it can only be Monday. So the odds that the coin came up tails is twice as likely as the odds that it came up heads (i.e., 2/3 vs. 1/3).

It seems to me that asking Sleeping Beauty to ignore the information she’s been given about the experiment is asking her to give an answer she knows is incorrect. What a question means depends on its context. If the context in this case were simply: “Please explain how coin flips work. In particular, what are the odds that a fair, properly-executed coin flip will come up heads instead of tails, such as the coin flip we executed on Sunday?” In that context, the correct answer is obviously 1/2. Everybody should be a halfer when asked how coin flips work.

But that isn’t the context in which Sleeping Beauty is being asked to give an answer. She’s not being asked to explain how coin flips work in general. She understands that she’s involved in a bizarre, possibly unethical (the memory-destroying drug!) experiment and that the question she’s being asked is part of that experiment. She’s being asked what the odds are given her current situation. That’s why I think it’s more reasonable for Sleeping Beauty to be a thirder than a halfer. She should answer “1/3”. It’s twice as likely that the coin came up tails!

Whether she should be allowed to go back to sleep after giving the correct answer, so that she can later be awakened by a handsome young prince, or whether she should receive appropriate medical care and reacquire a normal sleeping pattern, is an ethical question, not a logical one, and beyond the scope of this post.

What Should Sleeping Beauty Say, Logically Speaking?

Imagine that Sleeping Beauty agrees to take part in an experiment. First, she’s going to be given a special memory-loss drug that will make her forget what day it is. Then she’ll go to sleep. While she’s asleep, the people running the experiment will flip a coin. If the coin comes up heads, they’ll wake her up on Monday and ask her a question. If it comes up tails, they’ll wake her up on Monday, ask her the same question, give her the drug, let her fall asleep again, and then wake her up on Tuesday, asking the same question as before.

Because of the drug, however, she won’t know what day it is when she wakes up. Furthermore, she won’t know whether this is the first or second time she’s woken up. Each day will be a brand new experience for her. But each time Sleeping Beauty wakes up during the experiment, she’s told about the coin flip and the rule that says heads will mean she will wake up on Monday, and tails will mean she will wake up Monday and Tuesday. 

Now here’s the problem: Each time Sleeping Beauty wakes up and is told the rules of the experiment, she’s asked this question: “What are the odds that the coin flip came up heads?” That is, what are the odds that the coin came up heads and this is Monday (as opposed to tails and this is either Monday or Tuesday)? What should Sleeping Beauty say?

I came across this problem at the Quanta Magazine site yesterday. You may be surprised to learn that experts have been arguing about its solution for years:

The famous Sleeping Beauty problem has polarized communities of mathematicians — probability theorists, decision theorists and philosophers — for over 15 years…. This simple mathematical problem has generated an unusually heated debate. The entrenched arguments between those who answer “one-half” (the camp called “halfers”) and those who say “one-third” (the “thirders”) put political debates to shame… Halfers and thirders tend to remain firmly rooted in their view of the Sleeping Beauty problem. Both camps can certainly do the math, so what makes them butt heads in vain? Is the problem underspecified? Is it ambiguous?

So, should Sleeping Beauty say there is a 1/2 chance that the coin came up heads, since there’s always a 1/2 chance that coin flips come up heads and a 1/2 chance that they come up tails? Should Sleeping Beauty be a halfer?

Or should she be a thirder and say there is only a 1/3 chance it was heads? After all, if it’s Monday, she was awakened because the coin came up heads or tails. If it’s Tuesday, she’s awake because the coin came up tails. That means it’s twice as likely she’s awake because the coin came up tails. There’s a 2/3 chance the coin came up tails and a 1/3 chance it came up heads. It sounds like Sleeping Beauty should be a thirder.

But not so fast! There was only that one coin flip on Sunday and we all know that coins have a 1/2 chance of coming up heads! Maybe she should be a halfer?

The Quanta article is fairly long and delves into why halfers and thirders give the answers they do, as well as why they often resist changing their minds. The comments that follow the article are even longer and include mathematical formulas. I didn’t read all the comments, and I’ve never studied probability, but I left my own comment anyway:

I wasn’t a halfer or a thirder before reading the article. Now I’m a committed thirder.

If you were to simply ask Sleeping Beauty whether a fair coin toss came up heads, she should say the odds were 1/2. Without any other information, that’s the rational answer. But you’re asking what odds Sleeping Beauty should assign, given the additional information she’s been given about the experiment. Since today could be either Monday or Tuesday (as far as she knows), it’s more likely that the coin came up tails. The fully-informed, rational answer she should give is 1/3.

So I think Sleeping Beauty should say this each morning: “Given how coin flips work, the odds are 1/2 that heads came up. But given how coin flips work and given what you’ve told me about this peculiar experiment, the odds are only 1/3 that heads came up. I can easily flip back and forth between the halfer and thirder positions, but why should I ignore the additional information you’ve given me? Taking into account what I  know about the experiment, I must conclude that the odds are 1/3 that heads came up. If that makes me a thirder, so be it. Now where’s my prince?”

But she shouldn’t take my word for it. She should make up her own mind.

The author of the article, Pradeep Mutalik, responded to my comment and everyone else’s last night. I’ll post his response tomorrow in case you haven’t already rushed to the Quanta site to read it.