Bayes Rule

Okay. For now, this is how this blog will work. With my imperfect knowledge of materials, honestly, writing an organized teaching material will be difficult; otherwise I’ll be writing a book 😛 Instead, I’ll be posting small bit by bit of what comes up in my mind, and when these small bits become dense enough to cover a solid topic, then probably I will turn that into a proper page with in an organized format. Until then, here it goes.

Let’s talk about Bayes rule. In a mathematical expression, or formula, it simply states:

    \[ p(A|B) = \frac{p(B|A)\cdot p(A)}{p(B)} \]

Here, p(a|b) represents a probability density function of an event a happening given an event b; we’ll talk about probability distributions in future posts. Or if it helps, you can take it in terms of probabilities as well:

    \[ \mathrm{Pr}(A|B) = \frac{\mathrm{Pr}(B|A)\cdot \mathrm{Pr}(A)}{\mathrm{Pr}(B)} \]

where \mathrm{Pr}(a|b) represents the probability of event a happening given an event b

Bayes rule (or Bayes theorem) is widely used in probabilistic robotics literature and algorithms. Why is that? Because it lets you know more about p(A|B) without directly measuring this, or more often you don’t have a way to measure this. Bayes rule lets you compute this p(A|B) as long as you know p(A) and p(B|A). Imagine you have a variable A in which state you cannot have a direct measurement, but you know how B behaves when event A has occurred. In such case, however, Then, Bayes rule states that with only a priori knowledge of A and the conditional probability of B given A, you can gain knowledge about the posterior information of A.

I know these are a lot to digest, and it’s not easy to feel how this helps. In my posts, I’ll try to use many examples as I can so that it’s easier to see its applications.

example) An apple in a paper bag

Imagine a paper bag on a table with an apple in it. The apple can be a red Gala (R) or a green Fuji (G); yes, you only have two choices here. You want to know which kind of the two apples is in the bag.

Because the paper bag is not transparent, you cannot see the type of the apple. Then, what are the chance of that apple being a red Gala? Since you haven’t touched it and can’t see anything, let’s say we’re 50/50.

You start thinking of how to tell which apple it is. You cannot open the bag, but you’re free to lift the bag. And, you happen to know that Fuji apples are generally heavier than Gala apples! And once you lift the bag, you will feel its weight! So can we use that information to know which apple is more likely in the bag? 

Yes, you can. You probably won’t know it for sure, but you can be somewhat confident with your answer.

So you go to the table and lift the bag up. You then realize that the paper bag does feel a bit heavier than you would for a bag with a Gala apple. Hmm. Okay – it feels like a Fuji apple, but you’re not too sure how precise your feeling is; maybe you were holding it for too long and your arm got tired. Let’s say you’re 60% sure that it’s heavier than a Gala. 

Good. We have all the information to use Bayes rule now. Let’s start with setting up some variables. What we want to know is the probability of the type of apple (X) being G given the heaviness you feel. Let’s denote the your measurement (your feeling) of heaviness as Y.

Again, there are two types of apples: X\in\{R, G\} and your measurements are Y\in\{\mathrm{heavy},\mathrm{light}\}. And we want to know \mathrm{Pr}(X=G|Y=\text{heavy}). You could compute \mathrm{Pr}(X=R|Y=\text{heavy}) instead, and it will be equal to 1 - \mathrm{Pr}(X=G|Y=\text{heavy}).

Applying Bayes rule, the above equation becomes:

    \[ \mathrm{Pr}(X=G|Y=\text{heavy}) = \frac{\mathrm{Pr}(Y=\text{heavy}|X=G) \cdot \mathrm{Pr}(X=G)}{\mathrm{Pr}(Y=\text{heavy})} \]

Let’s see what these terms mean: \mathrm{Pr}(Y=\text{heavy}|X=G) means the probability of you feeling heavy when the apple is G; this is something you do have an idea. Previously we mentioned that you’re about 60% sure; so the value is 0.6. The other term, \mathrm{Pr}(X=G) is your a priori information about the apple being G. Since we’re 50/50 sure, the value becomes 0.5.

What about \mathrm{Pr}(Y=\text{heavy})? This term is asking what the probability of you feeling heavy in general. This is an odd thing. Without knowing (or given) the type of apple in the bag, how would you know how likely you’ll feel heavy? Given our limited choices, however, we can actually compute this. We’ll use law of total probability which states:

    \[ \mathrm{Pr}(A) = \sum_{B} \mathrm{Pr}(A, B) = \sum_{B} \mathrm{Pr}(A|B) \cdot \mathrm{Pr}(B) \]

Note that \mathrm{Pr}(A) = \sum_{B} \mathrm{Pr}(A, B) is for computing marginal distribution. If you have two (random) variables A and B, you can compute a marginal distribution of one by integrating out the other variable, which is what that summation (or integration in continuous space) in above equation is. 

Applying this to our apple problem,

    \begin{equation*} \begin{align*} \mathrm{Pr}(Y=\text{heavy}) &=\sum_{X} \mathrm{Pr}(Y=\text{heavy},X) \\ &= \sum_{X} \mathrm{Pr}(Y=\text{heavy}|X) \cdot \mathrm{Pr}(X) \\ &= \mathrm{Pr}(Y=\text{heavy}|X=G) \cdot \mathrm{Pr}(X=G) \\ & \quad\quad +\mathrm{Pr}(Y=\text{heavy}|X=R) \cdot \mathrm{Pr}(X=R) \end{align*} \end{equation*}

The term \mathrm{Pr}(Y=\text{light}|X=G) = 1 - \mathrm{Pr}(Y=\text{light}|X=G)=0.4 because \sum_{Y\in\{\text{heavy,light}\}} \mathrm{Pr}(Y|X=G) = 1. For the other term, \mathrm{Pr}(Y=\text{heavy}|X=R), we’ll need to set a value for this term. You can assign the same value as \mathrm{Pr}(Y=\text{light}|X=G) , but they fundamentally have a different meaning. This term asks what’s the likelihood of you feeling heavy when the apple is R. Maybe your sense of heaviness completely changes lifting a Gala (R) apple, or you’ve never seen one and have no clue of their weights. Let’s say you’ve never seen a Gala apple, thus have no clue. This will make you feeling heavy given a Gala apple, you’re equally unsure whether it’s heavy or light — thus \mathrm{Pr}(Y=\text{heavy}|X=R)=0.5

Now we have all the pieces and can carry out the calculation.

    \begin{equation*} \begin{align*} \mathrm{Pr}(X=G|Y=\text{heavy}) &= \frac{\mathrm{Pr}(Y=\text{heavy}|X=G) \cdot \mathrm{Pr}(X=G)}{\mathrm{Pr}(Y=\text{heavy})} \\ &= \frac{0.6 \cdot 0.5}{0.4 \cdot 0.5 + 0.5 \cdot 0.5} \\ &= \frac{0.3}{0.2 + 0.25} = \frac{0.3}{0.45} = 0.6667 \end{align*} \end{equation*}

So you’re 66.67% sure that the apple is a green Fuji (G) given your feeling of heaviness!

Leave a Reply

Your email address will not be published. Required fields are marked *