📐 Bayes' Theorem: A Comprehensive Guide

Overview of Bayes' theorem (also known as Bayes' law or Bayes' rule)
Mathematical statement, derivations, and formulations
Practical examples including medical diagnosis, drug testing, and bent coins
Philosophical interpretations: Bayesian vs. Frequentist
Advanced forms including random variables and odds form
Generalizations such as the 3-event theorem and inference rules

💡 Core Concepts & Introduction

Bayes' theorem is a foundational mathematical rule in probability theory, named after Thomas Bayes and developed independently by Pierre-Simon Laplace in the 18th century. It provides a way to invert conditional probabilities, allowing the probability of a cause to be found given its effect.

“Bayes' theorem gives a mathematical rule for inverting conditional probabilities, allowing the probability of a cause to be found given its effect.”

Key Applications

Bayesian Inference: An approach to statistical inference where it is used to invert the probability of observations given a model configuration (the likelihood function) to obtain the probability of the model configuration given the observations (the posterior probability).
Diagnostic Testing: Calculating the likelihood of a condition given a positive or negative test result.

🧮 Statement of the Theorem

Standard Mathematical Equation

For events $A$ and $B$ where $P(B) \neq 0$ :

$P(A\vert B)={\frac {P(B\vert A)P(A)}{P(B)}}$

Component Definitions

$P(A\vert B)$ (Posterior Probability): The probability of event $A$ occurring given that $B$ is true.
$P(B\vert A)$ (Likelihood): The probability of event $B$ occurring given that $A$ is true (also interpreted as the likelihood function evaluated at $A$ given a fixed $B$ ).
$P(A)$ (Prior Probability): The probability of observing $A$ without any given conditions (prior to new evidence).
$P(B)$ (Marginal Probability): The unconditioned probability of observing $B$ .

Derivation from Joint Probabilities

The joint probability of both $A$ $A$ and $B$ $B$ happening ( $P(A \cap B)$ $P (A \cap B)$ ) can be expressed in two ways:
- $P(A \cap B) = P(A \vert B)P(B)$
- $P(A \cap B) = P(B \vert A)P(A)$
Equating the two products:
- $P(A \vert B)P(B) = P(B \vert A)P(A)$
Dividing both sides by $P(B)$ yields Bayes' theorem.

Law of Total Probability Expansion

If events $A_1, A_2, \dots$ are mutually exclusive and exhaustive (one is certain to occur, but no two can occur together), then by the law of total probability:

$P(B) = \sum_{i} P(B|A_i)P(A_i)$

Substituting this into the denominator gives the expanded formula:

$P(A_i|B) = \frac{P(A_i)P(B|A_i)}{\sum_{j} P(A_j)P(B|A_j)}$

For a binary pair of events ( $A$ and $\neg A$ ), the denominator simplifies to two terms:

$P(A|B) = \frac{P(B|A)P(A)}{P(B|A)P(A) + P(B|\neg A)P(\neg A)}$

📊 Practical Examples

1. Medical Diagnosis

Suppose a doctor tests a patient for a disease. Let:

$E$ : Event that the patient has the disease ( $P(E)$ is prevalence rate).
$F$ : Event that the patient tests positive ( $P(F|E)$ is true positive rate / sensitivity).

P({\text{Cancer}}|{\text{Symptoms}}) &= \frac{P({\text{Symptoms}}|{\text{Cancer}})P({\text{Cancer}})}{P({\text{Symptoms}}|{\text{Cancer}})P({\text{Cancer}})+P({\text{Symptoms}}|{\text{Non-Cancer}})P({\text{Non-Cancer}})} \\ &= \frac{1\times 0.00001}{1\times 0.00001+(10/99999)\times 0.99999} = \frac{1}{11}\approx 9.1\% \end{aligned}$$ ### 2. Drug Testing - **Sensitivity (TPR):** $0.99$ (99% true positive rate) - **Specificity (TNR):** $0.99$ (1% false positive rate / FPR = $0.01$) - **Drug Prevalence:** $0.003$ ($0.3\%$ of people use the drug) $$\begin{aligned} P({\text{User}}\vert {\text{Positive}}) &= \frac{P({\text{Positive}}\vert {\text{User}})P({\text{User}})}{P({\text{Positive}}\vert {\text{User}})P({\text{User}})+P({\text{Positive}}\vert {\text{Non-user}})P({\text{Non-user}})} \\ &= \frac{0.99\times 0.003}{0.99\times 0.003+0.01\times 0.997}\approx 23\% \end{aligned}$$ > *Result:* Even though the drug test is **99% accurate**, most of its positive results will be false due to low base rates. ### 3. Bent Coins An urn contains: - **Type A (Fair):** $P(H|A) = 0.5$, Count = 2 ($P(A) = 2/5$) - **Type B (Biased):** $P(H|B) = 0.6$, Count = 2 ($P(B) = 2/5$) - **Type C (Biased):** $P(H|C) = 0.9$, Count = 1 ($P(C) = 1/5$) Using the total probability law, $P(H) = 0.62$, yielding: $$P(A|H) = \frac{0.2}{0.62} \approx 32\%$$ --- ## 🧠 Philosophical Interpretations | Interpretation Aspect | Bayesian (Epistemological) | Frequentist | |:----------------------|:---------------------------|:------------| | **Meaning of Probability** | Measures a rational **degree of belief**. | Measures a **proportion of outcomes** over repeated experiments. | | **$P(A)$ Definition** | Initial degree of belief in proposition $A$ (Prior). | Proportion of outcomes with property $A$. | | **$P(B|A)$ Definition** | Degree of belief in evidence $B$ given $A$ is true. | Proportion of outcomes with property $B$ out of those with property $A$. | | **Core Utility** | Links belief before and after accounting for evidence. | Maps relative frequencies of occurrences. | --- ## 📈 Alternative Forms & Generalizations ### 1. Prior and Likelihood Proportionality When evidence $B$ is fixed in Bayesian inference, the denominator $P(B)$ remains constant. The relation can be expressed as direct proportionality: $$P(A|B) \propto P(A) \cdot P(B|A)$$ *In other words, the posterior is proportional to the prior times the likelihood.* ### 2. Random Variables For two continuous random variables $X$ and $Y$ using probability density functions: $$f_{X\vert Y=y}(x)={\frac {f_{Y\vert X=x}(y)f_{X}(x)}{f_{Y}(y)}}$$ - Formulated using conditional distributions and the **Radon–Nikodym theorem** (formalized by Andrey Kolmogorov in 1933). - Widely utilized in modern **Markov chain Monte Carlo (MCMC)** methods. ### 3. Odds Form (Bayes Factor) Bayes' theorem can be framed using odds: $$O(A|B) = O(A) \frac{P(B|A)}{P(B|\neg A)}$$ - **$O(A|B)$:** Posterior odds - **$O(A)$:** Prior odds - **$\frac{P(B|A)}{P(B|\neg A)}$:** Bayes factor or **likelihood ratio** ### 4. Bayes' Theorem for 3 Events Adding a third conditioning event $C$ (with $P(C) > 0$): $$P(A\vert B\cap C)={\frac {P(B\vert A\cap C)\,P(A\vert C)}{P(B\vert C)}}$$ ### 5. Inference Rules in Subjective Probability In subjective frameworks, Bayes' theorem serves as the fundamental consistency rule for how an agent should rationally **update or modify their beliefs** upon receiving new information $B$.

Bayes' Theorem: Statistics Study Notes