Probability of choosing exactly one point on the number line or a real number, say 2.5,
from the interval \([2,3]\) is almost = 0, because there are infinitely many points between 2 and 3.

Also, we can NOT say that choosing exactly 2.5 is impossible, because it exists there on the number line.
But, for all practical purposes, \(P(2.5) = 0\).

Therefore, we say that \(P=0\) means “Highly Unlikely” and NOT “Impossible”.

Extending this line of reasoning, we can say that probability of NOT choosing 2.5, \(P(!2.5) = 1\).
Theoretically yes, because there are infinitely many points between 2 and 3.
But, we cannot say for sure that we cannot choose 2.5 exactly.
There is some probability of choosing 2.5, but it is very small.

Therefore, we say that \(P=1\) means “Almost Sure” and NOT “Certain”.

Here, in this case we can say that \(P(7)=0\) and that means Impossible.

Similarly, we can say that \(P(get ~any ~number ~between ~1 ~and ~6)=1\) and \(P=1 => \) Certain.

Conditional Probability:
It is the probability of an event occurring, given that another event has already occurred.
Allows us to update probability when additional information is revealed.

Bayes’ Theorem:
It is a formula that uses conditional probability.
It allows us to update our belief about an event’s probability based on new evidence.
We know from conditional probability and chain rule that:

Combining all the above equations gives us the Bayes’ Theorem:

Law of Total Probability:
Overall probability of an event B, considering all the different, mutually exclusive ways it can occur.
If A₁, A₂, …, Aₙ are a set of events that partition the sample space, such that they are -

• Mututally exclusive : \(A_i \cap A_j = \emptyset\) for all \(i, j\)
• Exhaustive: \(A₁ \cup A₂ ... \cup Aₙ = \Omega\) for all \(i \neq j\) $$ P(B) = \sum_{i=1}^{n} P(A_i)*P(B \mid A_i) $$ where \(n\) is the number of mutually exclusive partitions of the sample space \(\Omega\) .

Independence of Events:
Two events are independent if the occurrence of one event does not affect the probability of the other event.
There are 3 types of independence of events:

• Mutual Independence
• Pair-Wise Independence
• Conditional Independence

Mutual Independence:
Joint probability of two events is equal to the product of the individual probabilities of the two events.
\(P(A \cap B) = P(A)*P(B)\)

Joint probability: The probability of two or more events occurring simultaneously.
\(P(A \cap B)\) or \(P(A, B)\)

Binomial Distribution:
It extends the Bernoulli distribution by modeling the number of successes that occur over a fixed number of independent trials.
n = Number of trials
k = Number of successes
p = Probability of success
Mean = np
Variance = np(1-p)

\(PMF, P(x=k) = \binom{n}{k}p^k(1-p)^{n-k}\), where \(k \in \{0,1,2,3,...,n\}\)
\(\binom{n}{k} = \frac{n!}{k!(n-k)!}\) i.e number of ways to achieve ‘k’ successes in ‘n’ independent trials.

Note: Bernoulli is a special case of Binomial distribution where n = 1.
Also read about Multinomial distribution i.e where number of possible outcomes is > 2.

Total number of outcomes in 3 coin tosses = 2^3 = 8
Desired outcomes i.e 2 heads in 3 coin tosses = \(\{HHT, HTH, THH\}\) = 3
Probability of getting exactly 2 heads in 3 coin tosses = \(\frac{3}{8}\) = 0.375
Now lets solve the question using the binomial distribution formula.

Number of successes, k = 1
Number of trials, n = 10
Probability of success, p = 1/3
Probability of winning lottery, P(k=1) =

Poisson Distribution:
It expresses the probability of an event happening a certain number of times ‘k’ within a fixed interval of time.
Given that:

1. Events occur with a known constant average rate.
   
2. Occurrence of an event is independent of the time since the last event.
   
   Parameters:
   \(\lambda\): Expected number of events per interval
   \(k\) = Number of events in the same interval
   Mean = \(\lambda\)
   Variance = \(\lambda\)
   

PMF = Probability of occurrence of ‘k’ events in the same interval

Note: Useful to count data where total population size is large but the probability of an individual event is small.

Expected (average) number of emails per hour, \(\lambda\) = 5
Probability of receiving exactly k=3 emails in the next hour =

Probability Density Function(PDF):
This is a function used for continuous random variables to describe the likelihood of the variable taking on a value within a specific range or interval.
Since, at any given point the probability of a continuous random variable is zero, we find the probability within a given range.
Note: Called ‘density’ because probability is spread continuously over a range of values rather than being concentrated at a single point as in PMF.
e.g: Uniform, Gaussian, Exponential, etc.

Note: PDF is a continuous function \(f(x)\).
It is also the derivative of Cumulative Distribution Function (CDF) \(F_X(x)\)

\(PDF = f(x) = F'(X) = \frac{dF_X(x)}{dx} \)

Uniform Distribution:
All the outcomes within the given range are equally likely to occur.
Also known as ‘fair’ distribution.
Note: This is a natural starting point to understand randomness in general.

Mean = Median = \( \frac{a+b}{2} ~if~ x \in [a,b] \)
Variance = \( \frac{(b-a)^2}{12} \)

Standard uniform distribution: \( X \sim U(0,1) \)

PDF in terms of mean(\(\mu\)) and standard deviation(\(\sigma\)) -

Gaussian(Normal) Distribution:
It is a continuous probability distribution characterized by a symmetrical, bell-shaped curve with most data clustered around the central average, with frequency of values decreasing as they move away from the center.

• Most outcomes are average; extremely low or extremely high values are rare.
  
• Characterised by mean and standard deviation/variance.
  
• Peak at mean = median, symmetric around mean.
  Note: Most important and widely used distribution.
  
  \[ X \sim N(\mu, \sigma^2) \] $$ \begin{aligned} PDF = f(x) = \dfrac{1}{\sqrt{2\pi}\sigma}e^{-\dfrac{(x-\mu)^2}{2\sigma^2}} \\ \end{aligned} $$
  Mean = \(\mu\)
  Variance = \(\sigma^2\)
  

Standard normal distribution:

Any normal distribution can be standardized using Z-score transformation:

Exponential Distribution:
It is used to model the amount of time until a specific event occurs.
Given that:

1. Events occur with a known constant average rate.
   
2. Occurrence of an event is independent of the time since the last event.
   
   Parameters:
   Rate parameter: \(\lambda\): Average number of events per unit time
   Scale parameter: \(\mu ~or~ \beta \): Mean time between events
   Mean = \(\frac{1}{\lambda}\)
   Variance = \(\frac{1}{\lambda^2}\)
   \[ \lambda = \dfrac{1}{\beta} = \dfrac{1}{\mu}\]
   

Mean time to serve 1 customer = \(\mu\) = 4 minutes
So, \(\lambda\) = average number of customers served per unit time = \(1/\mu\) = 1/4 = 0.25 minutes
Probability to serve a customer in less than 3 minutes can be found using CDF -

So, probability of a customer being served in less than 3 minutes is 53%(approx).

In this case x = 2 minutes, and \(\lambda\) = 0.25 so,

So, probability of a customer being served in greater than 2 minutes is 60%(approx).

Task: We want to find the probability of the electronic item lasting for \(x > t+s \) days,
given that it has already lasted for \(x>t\) days.
This is a ‘conditional probability’.
Since,

We want to find: \( P(X > t+s \mid X > t) = ~? \)

Hence, probability that the electronic item will survive for an additional time ’s’ days
is independent of the time ’t’ days it has already survived.

Lets see the proof for the above statement.

Poisson Case:
The probability of observing exactly ‘k’ events in a time interval of length ’t’
with an average rate of \( \lambda \) events per unit time is given by -

The event that waiting time until next event > t, is same as observing ‘0’ events in that interval.
=> We can use the PMF of Poisson distribution with k=0.

Exponential Case:
Now, lets consider exponential distribution that models the waiting time ‘T’ until next event.
The event that waiting time ‘T’ > t, is same as observing ‘0’ events in that interval.

Observation:
Exponential distribution is a direct consequence of Poisson distribution probability of ‘0’ events.

Using Poisson:
Average failure rate, \(\lambda\) = 1/20 = 0.05
Time interval, t = 10 hours
Number of events, k = 0
Average number of events in interval, (\(\lambda t\)) = (1/20) * 10 = 0.5
Probability of having NO failures in the next 10 hours = ?

Using Exponential:
What is the probability that wait time until next failure > 10 hours?
Waiting time, T > t = 10 hours.
Average number of failures per hour, \(\lambda\) = 1/20 per hour

Therefore, we have seen that this problem can be solved using both Poisson and Exponential distribution.

Expectation:
Long run average of the outcomes of a random experiment.
When we talk about ‘Expectation’ we mean - on an average.

• Value we expect to get when we repeated an experiment multiple times and took an average of the results.
• Measures the central tendency of a data distribution.
• Similar to ‘Center of Mass’ in Physics.

Note: If the probability of each outcome is different, then we take a weighted average.

Discrete Case:
\(E[X] = \sum_{i=1}^{n} x_i.P(X = x_i) \)

Continuous Case:
\(E[X] = \int_{-\infty}^{\infty} x.f(x) dx \)

Possible outcomes are: \(x_1 = 100 ,~ x_2 = -50 \)
Probability of each outcome is \(P(X = 100) = 0.5,~ P(X = -50) = 0.5 \)

Therefore, the expected value of the amount that you will win per toss is Rs. 25 in the long run.

Variance:
It is the average of the squared differences from the mean.

• Measures the spread or variability in the data distribution.
  
• If the variance is low, then the data is clustered around the mean i.e less variability;
  and if the variance is high, then the data is widely spread out i.e high variability.
  

Variance in terms of expected value, where \(E[X] = \mu\), is given by:

Note: This is the computational formula for variance, as it is easier to calculate than the average of square distances from mean.

PDF of continuous uniform random variable =

We know \(E[X]\) already, now we will calculate \(E[X^2]\):

Now, we know both \(E[X]\) and \(E[X^2]\), so we can calculate \(Var[X]\):

Co-Variance:
It is the measure of how 2 variables X & Y vary together.
It gives the direction of the relationship between the variables.

Note: For both direction as well as magnitude, we use Correlation.
Let’s use expectation to compute the co-variance of two random variables X & Y:

Note:

• In a multivariate setting, relationship between all the pairs of random variables are summarized in a square symmetrix matrix called ‘Co-Variance Matrix’ \(\Sigma\).
  
• Covariance of a random variable with self gives the variance, hence the diagonals of covariance matrix are variances.
  

Moments:
They are statistical measures that describe the characteristics of a probability distribution, such as its central tendency, spread/variability, and asymmetry.
Note: We will discuss ‘raw’ and ‘central’ moments.

Important moments:

1. Mean : Central tendency
2. Variance : Spread/variability
3. Skewness : Asymmetry
4. Kurtosis : Tailedness

Moment Generating Function:
It is a function that simplifies computation of moments, such as, the mean, variance, skewness, and kurtosis, by providing a compact way to derive any moment of a random variable through differentiation.

• Provides an alternative to PDFs and CDFs of random variables.
• A powerful property of the MGF is that its \(n^{th}\) derivative, when evaluated at \((t=0)\) = the \(n^{th}\) moment of the random variable.

Not all probability distributions have MGFs; sometimes, integrals may not converge and moment may not exist.

Since, \(MGF = M_X(t) = E[e^{tX}] \), we can write the MGF as:

Say, \( E[X^n] = m_n \), where \(m_n\) is the \(n^{th}\) moment of the random variable, then:

Important: Differentiating \(M_X(t)\) with respect to \(t\), \(i\) times, and setting \((t =0)\) gives us the \(i^{th}\) moment of the random variable.

Set \(t=0\), to get the first moment:

Similarly, second moment = \(M''_X(t) = m_2 = E[X^2]\)
And, \(n^{th}\) derivative = \(M^{(n)}_X(t) = m_n = E[X^n]\)

e.g: \( Var[X] = M''_X(0) - (M'_X(0))^2 \)

Note: If we know the MGF of a random variable, then we do NOT need to do integration or summation to find the moments.
Continuous:

Discrete:

Read more about Integration & Differentiation

Joint Probability Distribution:
It describes the probability of 2 or more random variables occurring simultaneously.

• The random variables can be from different distributions, such as, discrete and continuous.
  

Joint CDF:

Discrete Case:

Continuous Case:

Joint PMF:

Key Properties:

1. \(P(X = x, Y = y) \ge 0 ~ \forall (x,y) \)
2. \( \sum_{i} \sum_{j} P(X = x_i, Y = y_j) = 1 \)

Joint PDF:

Key Properties:

1. \(f_{X,Y}(x,y) \ge 0 ~ \forall (x,y) \)
2. \( \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f_{X,Y}(x,y) dy dx = 1 \)
   

Let A & B be discrete random variables associated with the outcome of the ball drawn from first and second bags respectively.

Since, the draws are independent, joint probability = P(A) * P(B)
Each of the 4 cells in above table shows the probability of combination of results from 2 draws or joint probability.

Marginal Probability Distribution:
It describes the probability distribution of an individual random variable in a joint distribution, without considering the outcomes of other random variables.

• If we have the joint distribution, then we can get the marginal distribution of each random variable from it.
  
• Marginal probability equals summing the joint probability across other random variables.
  

Marginal CDF:
We know that Joint CDF =

Marginal CDF =

Discrete Case:

Continuous Case:

Law of Total Probability
We know that Joint Probability Distribution =

The events \((Y=y)\) partition the sample space, such that:

1. \( (Y=y_1) \cap (Y=y_2) \cap ... \cap (Y=y_n) = \Phi \)
2. \( (Y=y_1) \cup (Y=y_2) \cup ... \cup (Y=y_n) = \Omega \)
   

From Law of Total Probability, we get:

Marginal PMF:

Marginal PDF:

Joint PMF = \( P_{X,Y}(x,y) = P(X=x, Y=y) = 1/6*1/2 = 1/12\)
Marginal PMF of X = \( P_X(x) =\sum_{y \in \mathbb{\{H,T\}}} P_{X,Y}(x,y) = = 1/12+1/12 = 1/6\)
=> Marginally, X is uniform over 1-6 i.e a fair die.

Marginal PMF of Y = \( P_Y(y) = \sum_{1}^6 P_{X,Y}(x,y) = 6*(1/12) = 1/2 \)
=> Marginally, Y is uniform over H,T i.e a fair coin.

Marginal PDF = \(f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x,y) dy \)
Joint PDF =

Marginal PDF =

Let A & B be discrete random variables associated with the outcome of the ball drawn from first and second bags respectively.

We can see from the table above - P(A=Red) is the sum of joint distribution over all possible values of B i.e Red & Blue.

Conditional Probability:
It measures the probability of an event occurring given that another event has already happened.

• It provides a way to update our belief about the likelihood based on new information.
  

P(A, B) = Joint Probability of A and B
P(B) = Marginal Probability of B

=> Conditional Probability = Joint Probability / Marginal Probability

Conditional CDF:

Discrete Case:

Continuous Case:

Conditional PMF:

Conditional PDF:

Let A & B be discrete random variables associated with the outcome of the ball drawn from first and second bags respectively.
A = Red ball in first draw
B = Blue ball in second draw.

Therefore, probability of drawing a red ball in the first draw, given that a blue ball is drawn in second draw = 2/5.

Conditional Expectation:
This gives us the conditional expectation of a random variable X, given another random variable Y=y.

Discrete Case:

Continuous Case:

Conditional Variance:
This gives us the variance of a random variable calculated after taking into account the value(s) of another related variable.

For example:

• Variance of car’s mileage for city driving might be small, but the variance will be large for mix of city and highway driving.

Note: Models that take into account the change in variance or heteroscedasticity tend to be more accurate.

Independence of Random Variables
It means that the knowing the outcome of one random variable does not impact the probability of the other random variable.
Two random variables X & Y are independent if:

• We know that if 2 random variables X,Y are independent, then their covariance is zero, since there is no linear relationship between them.
  
• But the converse may NOT be true, i.e, if the covariance is zero, then we can say for sure that the random variables are independent.
  

Read more about Covariance

Independent & Identically Distributed(I.I.D)
I.I.D assumption for samples(data points) in a dataset means that the samples are:

• Independent, i.e, each sample is independent of the other.
• Identically distributed, i.e, each sample is drawn from the same probability distribution.

Convergence in Probability:
A sequence of random variables \(X_1, X_1, \dots, X_n\) is said to converge in probability to a known random variable \(X\),
if for every number \(\epsilon >0 \), the following is true:

where,
\(X_n\): is the estimator or sample based random variable.
\(X\): is the known or limiting or target random variable.
\(\epsilon\): is the tolerance level or margin of error.

Read more about Limits

Almost Sure Convergence:
A sequence of random variables \(X_1, X_2, \dots, X_n\) is said to almost surely converge to a known random variable \(X\),
for \(n \ge 1\), if the following is true:

where,
\(X_n\): is the estimator or sample based random variable.
\(X\): is the known or limiting or target random variable.

If, \(X_n \xrightarrow{Almost ~ Sure} X \), => \(X_n \xrightarrow{Probability} X \)
But, converse is NOT true.

*Note: Almost Sure convergence is hardest to satisfy amongst all convergence, such as, convergence in probability, convergence in distribution, etc.

Read more about Limits

Weak Law of Large Numbers (WLLN):
This law states that given a sequence of independent and identically distributed (IID) samples \(X_1, X_1, \dots, X_n\) from a random variable with finite mean, the sample mean (\(\bar{X_n}\)) converges in probability to the expected value \(E[X]\) or population mean (\( \mu \)).

Note: Does NOT guarantee that sample mean will be close to population mean,
but instead says that - the probability of sample mean being far away from the population mean is low.

Read more about Limits

Strong Law of Large Numbers (SLLN):
This law states that given a sequence of independent and identically distributed (IID) samples \(X_1, X_1, \dots, X_n\) from a random variable with finite mean, the sample mean (\(\bar{X_n}\)) converges almost surely to the expected value \(E[X]\) or population mean (\( \mu \)).

Note:

• It guarantees that the sequence of sample averages itself converges to population mean, with exception of set of outcomes that has probability = 0.
• Almost certain guarantee; Much stronger statement than Weak Law of Large Numbers.
  
  

Read more about Limits

Markov’s Inequality:
It gives an upper bound for the probability that a non-negative random variable will NOT exceed, based on its expected value.

Note: It gives a very loose upper bound.

Read more about Expectation

Hence, a 25% chance of serving more than 200 customers.

Hence, there is a 78% chance that a randomly selected student gets a score of 90 marks or more.

Chebyshev’s Inequality:
It states that the probability of a random variable deviating from its mean is small if its variance is small.

• It is a more powerful version of Markov’s Inequality.
  
• It uses variance of the distribution in addition to the expected value or mean.
  
• Also, it does NOT assume the random variable to be non-negative.
  
• It uses more information about the data i.e mean and variance.
  

Note: It gives a tighter upper bound than Markov’s Inequality.

Given, the standard deviation of the test score \(\sigma\) = 10 marks.
=> Variance = \(\sigma^2\) = 100

Hence, Chebyshev’s Inequality gives a far tighter upper bound of 25% than Markov’s Inequality of 78%(approx).

Chernoff Bound:
It is an upper bound on the probability that a random variable deviates from its expected value.
It’s an exponentially decreasing bound on the “tail” of a random variable’s distribution, which can be calculated using its moment generating function.

• It is used for sum or average of independent random variables (not necessarily identically distributed).
• It provides exponentially tighter bounds, better than Chebyshev’s Inequality’s quadratic decay.
• It uses all moments to capture the full shape of the distribution, using the moment generating function(MGF).

Proof:

For the sum of ’n’ independent random variables,

Note: Used to compute how far the sum of independent random variables deviate from their expected value.

Read more about Moment Generating Function

Surprise Factor:
It is a measure of the amount of information gained when a specific, individual event occurs, and is defined based on the probability of that event.
It is defined as the negative logarithm of the probability of the event.

• It is also called ‘Surprisal’.

Note: Logarithm(for base > 1) is a monotonically increasing function, so as x increases, log(x) also increases.

• So, if probability P(x) increases, then surprise factor S(x) decreases.
• Common events have a high probability of occurrence, hence a low surprise factor.
• Rare events have a low probability of occurrence, hence a high surprise factor.

Units:

• The unit of surprise factor with log base 2 is bits.
  
• with base ’e’ or natural log its nats (natural units of information).
  

Entropy:
It conveys how much ‘information’ we expect to gain from a random event.

• Entropy is the average or expected value of surprise factor of a random variable.
• More uniform the distribution ⇒ greater average surprise factor, since all possible outcomes are equally likely.

Read more about Expectation

• Case 1:
  Toss a fair coin; \(P(H) = P(T) = 0.5 = 2^{-1}\)
  Surprise factor (Heads or Tails) = \(-log(P(x)) = -log_2(0.5) = -log_2(2^{-1}) = 1~bit\)
  Entropy = \( \sum_{x \in X} P(x)log(-P(x)) = 0.5*1 + 0.5*1 = 1 ~bit \)

• Case 2:
  Toss a biased coin; \(P(H) = 0.9 P(T) = 0.1 \)
  Surprise factor (Heads) = \(-log(P(x)) = -log_2(0.9) \approx 0.15 ~bits \)
  Surprise factor (Tails) = \(-log(P(x)) = -log_2(0.1) \approx 3.32 ~bits\)
  Entropy = \( \sum_{x \in X} P(x)log(-P(x)) = 0.9*0.15 + 0.1*3.32 \approx 0.47 ~bits\)
  

Therefore, a biased coin is less surprising on an average than a fair coin, hence lower entropy.

Cross Entropy:
It is a measure of the average ‘information gain’ or ‘surprise’ when using an imperfect model \(Q\) to encode events from a true model \(P\).

• It measures how surprised we are on an average, if the true distribution is \(P\), but we predict using another distribution \(Q\).
  

A model is trained to classify images as ‘cat’ or ‘dog’. Say, for an input image the true label is ‘cat’, so the true distribution:
\(P = [1.0 ~(cat), 0.0 ~(dog)]\).
Let’s calculate the cross-entropy for the outputs of 2 models A & B.

• Model A:
  \(Q_A = [0.8 ~(cat), 0.2 ~(dog)]\)
  Cross Entropy = \(H(P, Q_A) = -\sum_{i=1}^n P(x_i)log(Q_A(x_i)) \)
  \(= -[1*log_2(0.8) + 0*log_2(0.2)] \approx 0.32 ~bits\)
  Note: This is very low cross-entropy, since the predicted value is very close to actual, i.e low surprise.
  

• Model B:
  \(Q_B = [0.2 ~(cat), 0.8 ~(dog)]\)
  Cross Entropy = \(H(P, Q_B) = -\sum_{i=1}^n P(x_i)log(Q_B(x_i)) \)
  \(= -[1*log_2(0.2) + 0*log_2(0.8)] \approx 2.32 ~bits\)
  Note: Here the cross-entropy is very high, since the predicted value is quite far from the actual truth, i.e high surprise.
  

Kullback Leibler (KL) Divergence:
It measures the information lost when one probability distribution \(Q\) is used to approximate another distribution \(P\).
It quantifies the ‘extra cost’ in bits needed to encode data using the approximate distribution \(Q\) instead of the true distribution \(P\).

KL Divergence = Cross Entropy(P,Q) - Entropy(P)

Note:

• If \(P = Q\) ,i.e, P and Q are the same distributions, then KL Divergence = 0.
• KL divergence is NOT symmetrical ,i.e, \(D_{KL}(P \parallel Q) ⍯ (D_{KL}(Q \parallel P)\).

Using the same cat, dog classification problem example as mentioned above.
A model is trained to classify images as ‘cat’ or ‘dog’. Say, for an input image the true label is ‘cat’, so the true distribution:
\(P = [1.0 ~(cat), 0.0 ~(dog)]\).
Let’s calculate the KL divergence for the outputs of the 2 models A & B.
Let’s calculate entropy first, and we can reuse the cross-entropy values calculated above already.
Entropy: \(H(P) = -\sum_{x \in X} P(x)log(P(x)) = -[1*log_2(1) + 0*log_2(0)] = 0 ~bits\)
Note: \(0*log_2(0) = 0\), is an approximation hack, since \(log(0)\) is undefined.

• Model A:
  \( D_{KL}(P \parallel Q) = H(P, Q) - H(P) = 0.32 - 0 = 0.32 ~bits \)
  Model A incurs an additional 0.32 bits of surprise due to its imperfect prediction.
  

• Model B:
  \( D_{KL}(P \parallel Q) = H(P, Q) - H(P) = 2.32 - 0 = 2.32 ~bits \)
  Model B has much more ‘information loss’ or incurs higher ‘penalty’ of 2.32 bits as its prediction was from from the truth.
  

Jensen-Shannon Divergence:
It is a smoothed and symmetric version of the Kullback-Leibler (KL) divergence and is calculated by averaging the
KL divergences between each of the two distributions and their combined average distribution.

• Symmetrical and smoothed version of KL divergence.
• Always finite; KL divergence can be infinite if \( P ⍯ 0 ~and~ Q = 0 \).
  • Makes JS divergence more stable for ML models where some predicted probabilities may be exactly 0.

Let’s continue the cat and dog image classification example discussed above.
A model is trained to classify images as ‘cat’ or ‘dog’. Say, for an input image the true label is ‘cat’, so the true distribution:
\(P = [1.0 ~(cat), 0.0 ~(dog)]\).

Step 1: Calculate the average distribution M.

Step 2: Calculate \(D_{KL}(P \parallel M)\).

Step 3: Calculate \(D_{KL}(Q \parallel M)\).

Step 4: Finally, lets put all together to calculate \(D_{JS}(P \parallel Q)\).

Therefore, lower JS divergence value => P and Q are more similar.

Parametric Model Estimation:
It is the process of finding the best-fitting finite set of parameters \(\Theta\) for a model that assumes a specific probability distribution for the data.
It involves using the dataset to estimate the parameters (like the mean and standard deviation of a normal distribution) that define the model.

\(P(X \mid \Theta) \) : Probability of seeing data \(X: (X_1, X_2, \dots, X_n) \), given the parameters \(\Theta\) of the underlying probability distribution from which the data is assumed to be generated.

Goal of estimation:
We observed data, \( D = \{X_1, X_2, \dots, X_n\} \), and we want to infer the the unknow parameters \(\Theta\) of the underlying probability distribution, assuming that the data is generated I.I.D.
Read more for I.I.D

Note: Most of the time, from experience, we know the underlying probability distribution of data, such as, Bernoulli, Gaussian, etc.

Maximum Likelihood Estimation:
It is the most popular frequentist approach to estimate the parameters of a model.
This method helps us find the parameters \(\Theta\) that make the data most probable.

Likelihood Function:
Say, we have data, \(D = X_1, X_2, \dots, X_n\) are I.I.D discrete random variable with PMF = \(P_{\Theta}(.)\)
Then, the likelihood function is the probability of observing the data, \(D = \{X_1, X_2, \dots, X_n\}\), given the parameters \(\Theta\).

Task:
Find the value of parameter \(\Theta_{ML}\) that maximizes the likelihood function.

In order to find the parameter \(\Theta_{ML}\) that maximises the likelihood function,
we need to take the first derivative of the likelihood function with respect to \(\Theta\) and equate it to zero.
But, taking derivative of a product is challenging, so we will take the logarithm on both sides.
Note: Log is a monotonically increasing function, i.e, as x increases, log(x) increases too.

Let us denote the log-likelihood function as \(\bar{L}\).

Therefore, Maximum Likelihood Estimation is the parameter \(\Theta_{ML}\) that maximises the log-likelihood function.

Given \(X_1, X_2, \dots, X_n\) are I.I.D. Bernoulli random variable with PMF as below:

Estimate the parameter \(\theta\) using Maximum Likelihood Estimation.

Let, \(n_1 = \) number of 1’s in the dataset.

Likelihood Function:

Log-Likelihood Function:

Maximum Likelihood Estimation:

In order to find the parameter \(\theta_{ML}\), we need to take the first derivative of the log-likelihood function with respect to \(\theta\) and equate it to zero.
Read more about Derivative

Say, e.g., we have 10 observations for the Bernoulli random variable as: 1,0,1,1,0,1,1,0,1,0.
Then, the parameter \(\Theta_{ML} = \frac{6}{10} = 0.6\) i.e proportion of 1’s.

Likelihood Function:

Log-Likelihood Function:

Note: Here, the first term \( -\frac{n}{2} log (2\pi) \) is a constant wrt both \(\mu\) and \(\sigma\), so we can ignore the term.

Maximum Likelihood Estimation:

Instead of finding \(\mu\) and \(\sigma\) that maximises the log-likelihood function,
we can find \(\mu\) and \(\sigma\) that minimises the negative of the log-likelihood function.

Now, lets differentiate the log likelihood function wrt \(\mu\) and \(\sigma\) separately to get \(\mu_{ML}, \sigma_{ML}\).

Lets, calculate \(\mu_{ML}\) first by taking the derivative of the log-likelihood function wrt \(\mu\) and equating it to 0.
Read more about Derivative

Similarly, we can calculate \(\sigma_{ML}\) by taking the derivative of the log-likelihood function wrt \(\sigma\) and equating it to 0.

Note: In general MLE is biased, i.e does NOT give an unbiased estimate => divides by \(n\) instead of \((n-1)\).

Bayesian Statistics:
Bayesian statistics model parameters by updating initial beliefs (prior probabilities) with observed data to form a final belief (posterior probability) using Bayes’ Theorem.
Instead of a single point estimate, it provides a probability distribution over possible parameter values, which allows to quantify uncertainty and yields more robust models, especially with limited data.

Bayes’ Theorem:

Read more about Bayes’ Theorem

\(P(\Theta)\): Prior: Initial distribution of \(\Theta\) before seeing the data.
\(P(X \mid \Theta)\): Likelihood: Conditional distribution of data \(X\), given the parameter \(\Theta\).
\(P(\Theta \mid X)\): Posterior: Conditional distribution of parameter \(\Theta\), given the data \(X\).
\(P(X)\): Evidence: Probability of seeing the data \(X\).

Let, \(n_1 = \) number of 1’s in the dataset.
Prior: \(P(\Theta)\) : \(\Theta \sim U(0, 1) \), i.e, parameter \(\Theta\) comes from a continuos uniform distribution in the range [0,1].
\(f_{\Theta}(\theta) = 1, \theta \in [0, 1] \)

Likelihood: \(P_{X \mid \Theta}(x \mid \theta) = \theta^{n_1} (1 - \theta)^{n - n_1} \).

Posterior:

Note: The most difficult part is to calculate the denominator \(f_{X}(x)\).
So, either we try NOT to compute it all together, or we try to map it to some known functions to make calculations easier.

We know that we can get the marginal probability by integrating the joint probability over another variable.

Also from conditional probability, we know:

From equations 1 and 2, we have:

Now let’s replace the value of \(f_{X}(x) \) in the posterior from equation 3:
Posterior:

Beta Function:
It is a special mathematical function denoted by B(a, b) or β(a, b) that is defined by the integral formula:

Note: We can see that the denominator of the posterior is of the form of Beta function.
Posterior:

Suppose, in the above example, we are told that the parameter \(\Theta\) is closer to 1 than 0.
How will we incorporate this useful information (apriori knowledge) into our parameter estimation?

Since, we know apriori that \(\Theta\) is closer to 1 than 0, we should take this initial belief into account to do our parameter estimation.

Prior:

Posterior:

Note: If we do NOT have enough data, then we should NOT ignore our intial belief.
However, if we have enough data, then the data will override our initial belief and the posterior will be dominated by data.

We can use Bayesian Point Estimators, after getting the posterior distribution, to summarize it with a single value for practical use, such as,

• Maximum A Posteriori (MAP) Estimator
• Minimum Mean Square Error (MMSE) Estimator

Maximum A Posteriori (MAP) Estimator:
It finds the mode(peak) of the posterior distribution.

• MAP has the minimum probability of error, since it picks single most probable value.
  

Given that :
\(\Theta\) is discrete, with probability 0.5 for both 0 and 1.
=> The Gaussian distribution is equally likely to be centered at 0 or 1.
Variance: \(\sigma^2 = 1\)

Prior:

Likelihood:

Posterior:

We need to find:

Taking log on both sides:

Let’s calculate the log-likelihood function first -

Here computing \(f_{X}(x)\) is difficult.
So, instead of differentiating above equation wrt \(\Theta\), and equating to 0,
we will calculate the above expression for \(\theta=0\) and \(\theta=1\) and compare the values.
This way we can get rid of the common value \(f_{X}(x)\) in both expressions that is not dependent on \(\Theta\).

When \(\theta=1\):

Similarly, when \(\theta=0\):

So, we can say that \(\theta = 1\) only if :
the value of the above expression for \(\theta = 1\) > the value of the above expression for \(\theta = 0\) From equation 3 and 4:

Note: If the prior is uniform then \(\Theta_{MAP} = \Theta_{MLE}\), because uniform prior does NOT give any information about the initial bias, and all possibilities are equally likely.

What if, in the above example, we know that the initial belief is not uniform but biased towards 0?
Prior:

Now, let’s compare the log-posterior for both the cases i.e \(\theta=0\) and \(\theta=1\) as we did earlier.
But, note that this time the probabilities for \(\theta=0\) and \(\theta=1\) are different.

So, we can say that \(\theta = 1\) only if :
the value of the above expression for \(\theta = 1\) > the value of the above expression for \(\theta = 0\) From equation 3 and 4 above:

Therefore, we can see that \(\Theta_{MAP}\) is extra biased towards 0.

Note: For a non-uniform prior \(\Theta_{MAP}\) estimate will be pulled towards the prior’s mode.

Minimum Mean Square Error (MMSE) Estimation:
Minimizes the expected value of squared error.
Mean of the posterior distribution is the conditional expectation of parameter \(\Theta\), given the data.

• Posterior mean minimizes the the mean squared error.

\(\hat\Theta(X)\): Predicted value.
\(\Theta\): Actual value.

Let’s revisit the above examples that we used for MLE.

Case 1: Uniform Continuous Prior for parameter \(\Theta\) of Bernoulli distribution
Prior:

Posterior:

Let’s calculate the \(\Theta_{MMSE}\):

Similarly, for the second case where the prior is biased towards 1.
Case 2: Prior is biased towards 1 for parameter \(\Theta\) of Bernoulli distribution
Prior:

Posterior:

So, here in this case our \(\Theta_{MMSE}\) is: