It is the probability of an event occurring, given that another event has already occurred.
Allows us to update probability when additional information is revealed.

It is a formula that uses conditional probability.
It allows us to update our belief about an event’s probability based on new evidence.
We know from conditional probability and chain rule that:

Combining all the above equations gives us the Bayes’ Theorem:

Overall probability of an event B, considering all the different, mutually exclusive ways it can occur.
If A₁, A₂, …, Aₙ are a set of events that partition the sample space, such that they are -

• Mututally exclusive : \(A_i \cap A_j = \emptyset\) for all \(i, j\)
• Exhaustive: \(A₁ \cup A₂ ... \cup Aₙ = \Omega\) for all \(i \neq j\) \[ P(B) = \sum_{i=1}^{n} P(A_i)*P(B \mid A_i) \] where \(n\) is the number of mutually exclusive partitions of the sample space \(\Omega\) .

Two events are independent if the occurrence of one event does not affect the probability of the other event.
There are 3 types of independence of events:

• Mutual Independence
• Pair-Wise Independence
• Conditional Independence

Joint probability of two events is equal to the product of the individual probabilities of the two events.
\(P(A \cap B) = P(A)*P(B)\)

Joint probability: The probability of two or more events occurring simultaneously.
\(P(A \cap B)\) or \(P(A, B)\)

It extends the Bernoulli distribution by modeling the number of successes that occur over a fixed number of independent trials.
n = Number of trials
k = Number of successes
p = Probability of success
Mean = np
Variance = np(1-p)

\(PMF, P(x=k) = \binom{n}{k}p^k(1-p)^{n-k}\), where \(k \in \{0,1,2,3,...,n\}\)
\(\binom{n}{k} = \frac{n!}{k!(n-k)!}\) i.e number of ways to achieve ‘k’ successes in ‘n’ independent trials.

Note: Bernoulli is a special case of Binomial distribution where n = 1.
Also read about Multinomial distribution i.e where number of possible outcomes is > 2.

It expresses the probability of an event happening a certain number of times ‘k’ within a fixed interval of time.
Given that:

1. Events occur with a known constant average rate.
   
2. Occurrence of an event is independent of the time since the last event.
   
   Parameters:
   \(\lambda\): Expected number of events per interval
   \(k\) = Number of events in the same interval
   Mean = \(\lambda\)
   Variance = \(\lambda\)
   

PMF = Probability of occurrence of ‘k’ events in the same interval

Note: Useful to count data where total population size is large but the probability of an individual event is small.

This is a function used for continuous random variables to describe the likelihood of the variable taking on a value within a specific range or interval.
Since, at any given point the probability of a continuous random variable is zero, we find the probability within a given range.
Note: Called ‘density’ because probability is spread continuously over a range of values rather than being concentrated at a single point as in PMF.
e.g: Uniform, Gaussian, Exponential, etc.

Note: PDF is a continuous function \(f(x)\).
It is also the derivative of Cumulative Distribution Function (CDF) \(F_X(x)\)

\(PDF = f(x) = F'(X) = \frac{dF_X(x)}{dx} \)

All the outcomes within the given range are equally likely to occur.
Also known as ‘fair’ distribution.
Note: This is a natural starting point to understand randomness in general.

Mean = Median = \( \frac{a+b}{2} ~if~ x \in [a,b] \)
Variance = \( \frac{(b-a)^2}{12} \)

Standard uniform distribution: \( X \sim U(0,1) \)

PDF in terms of mean(\(\mu\)) and standard deviation(\(\sigma\)) -

It is a continuous probability distribution characterized by a symmetrical, bell-shaped curve with most data clustered around the central average, with frequency of values decreasing as they move away from the center.

• Most outcomes are average; extremely low or extremely high values are rare.
  
• Characterised by mean and standard deviation/variance.
  
• Peak at mean = median, symmetric around mean.
  Note: Most important and widely used distribution.
  
  \[ X \sim N(\mu, \sigma^2) \] $$ \begin{aligned} PDF = f(x) = \dfrac{1}{\sqrt{2\pi}\sigma}e^{-\dfrac{(x-\mu)^2}{2\sigma^2}} \\ \end{aligned} $$
  Mean = \(\mu\)
  Variance = \(\sigma^2\)
  

Standard normal distribution:

Any normal distribution can be standardized using Z-score transformation:

It is used to model the amount of time until a specific event occurs.
Given that:

1. Events occur with a known constant average rate.
   
2. Occurrence of an event is independent of the time since the last event.
   
   Parameters:
   Rate parameter: \(\lambda\): Average number of events per unit time
   Scale parameter: \(\mu ~or~ \beta \): Mean time between events
   Mean = \(\frac{1}{\lambda}\)
   Variance = \(\frac{1}{\lambda^2}\)
   \[ \lambda = \dfrac{1}{\beta} = \dfrac{1}{\mu}\]
   

Long run average of the outcomes of a random experiment.
When we talk about ‘Expectation’ we mean - on an average.

• Value we expect to get when we repeated an experiment multiple times and took an average of the results.
• Measures the central tendency of a data distribution.
• Similar to ‘Center of Mass’ in Physics.

Note: If the probability of each outcome is different, then we take a weighted average.

Discrete Case:
\(E[X] = \sum_{i=1}^{n} x_i.P(X = x_i) \)

Continuous Case:
\(E[X] = \int_{-\infty}^{\infty} x.f(x) dx \)

It is the average of the squared differences from the mean.

• Measures the spread or variability in the data distribution.
  
• If the variance is low, then the data is clustered around the mean i.e less variability;
  and if the variance is high, then the data is widely spread out i.e high variability.
  

Variance in terms of expected value, where \(E[X] = \mu\), is given by:

Note: This is the computational formula for variance, as it is easier to calculate than the average of square distances from mean.

It is the measure of how 2 variables X & Y vary together.
It gives the direction of the relationship between the variables.

Note: For both direction as well as magnitude, we use Correlation.
Let’s use expectation to compute the co-variance of two random variables X & Y:

Note:

• In a multivariate setting, relationship between all the pairs of random variables are summarized in a square symmetrix matrix called ‘Co-Variance Matrix’ \(\Sigma\).
  
• Covariance of a random variable with self gives the variance, hence the diagonals of covariance matrix are variances.
  

They are statistical measures that describe the characteristics of a probability distribution, such as its central tendency, spread/variability, and asymmetry.
Note: We will discuss ‘raw’ and ‘central’ moments.

Important moments:

1. Mean : Central tendency
2. Variance : Spread/variability
3. Skewness : Asymmetry
4. Kurtosis : Tailedness

It is a function that simplifies computation of moments, such as, the mean, variance, skewness, and kurtosis, by providing a compact way to derive any moment of a random variable through differentiation.

• Provides an alternative to PDFs and CDFs of random variables.
• A powerful property of the MGF is that its \(n^{th}\) derivative, when evaluated at \((t=0)\) = the \(n^{th}\) moment of the random variable.

Not all probability distributions have MGFs; sometimes, integrals may not converge and moment may not exist.

Since, \(MGF = M_X(t) = E[e^{tX}] \), we can write the MGF as:

Say, \( E[X^n] = m_n \), where \(m_n\) is the \(n^{th}\) moment of the random variable, then:

Important: Differentiating \(M_X(t)\) with respect to \(t\), \(i\) times, and setting \((t =0)\) gives us the \(i^{th}\) moment of the random variable.

Set \(t=0\), to get the first moment:

Similarly, second moment = \(M''_X(t) = m_2 = E[X^2]\)
And, \(n^{th}\) derivative = \(M^{(n)}_X(t) = m_n = E[X^n]\)

e.g: \( Var[X] = M''_X(0) - (M'_X(0))^2 \)

Note: If we know the MGF of a random variable, then we do NOT need to do integration or summation to find the moments.
Continuous:

Discrete:

Read more about Integration & Differentiation

It describes the probability of 2 or more random variables occurring simultaneously.

• The random variables can be from different distributions, such as, discrete and continuous.
  

Joint CDF:

Discrete Case:

Continuous Case:

Joint PMF:

Key Properties:

1. \(P(X = x, Y = y) \ge 0 ~ \forall (x,y) \)
2. \( \sum_{i} \sum_{j} P(X = x_i, Y = y_j) = 1 \)

Joint PDF:

Key Properties:

1. \(f_{X,Y}(x,y) \ge 0 ~ \forall (x,y) \)
2. \( \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f_{X,Y}(x,y) dy dx = 1 \)
   

It describes the probability distribution of an individual random variable in a joint distribution, without considering the outcomes of other random variables.

• If we have the joint distribution, then we can get the marginal distribution of each random variable from it.
  
• Marginal probability equals summing the joint probability across other random variables.
  

Marginal CDF:
We know that Joint CDF =

Marginal CDF =

Discrete Case:

Continuous Case:

Law of Total Probability
We know that Joint Probability Distribution =

The events \((Y=y)\) partition the sample space, such that:

1. \( (Y=y_1) \cap (Y=y_2) \cap ... \cap (Y=y_n) = \Phi \)
2. \( (Y=y_1) \cup (Y=y_2) \cup ... \cup (Y=y_n) = \Omega \)
   

From Law of Total Probability, we get:

Marginal PMF:

Marginal PDF:

It measures the probability of an event occurring given that another event has already happened.

• It provides a way to update our belief about the likelihood based on new information.
  

P(A, B) = Joint Probability of A and B
P(B) = Marginal Probability of B

=> Conditional Probability = Joint Probability / Marginal Probability

Conditional CDF:

Discrete Case:

Continuous Case:

Conditional PMF:

Conditional PDF:

This gives us the conditional expectation of a random variable X, given another random variable Y=y.

Discrete Case:

Continuous Case:

This gives us the variance of a random variable calculated after taking into account the value(s) of another related variable.

For example:

• Variance of car’s mileage for city driving might be small, but the variance will be large for mix of city and highway driving.

Note: Models that take into account the change in variance or heteroscedasticity tend to be more accurate.

It means that the knowing the outcome of one random variable does not impact the probability of the other random variable.
Two random variables X & Y are independent if:

• We know that if 2 random variables X,Y are independent, then their covariance is zero, since there is no linear relationship between them.
  
• But the converse may NOT be true, i.e, if the covariance is zero, then we can say for sure that the random variables are independent.
  

Read more about Covariance

I.I.D assumption for samples(data points) in a dataset means that the samples are:

• Independent, i.e, each sample is independent of the other.
• Identically distributed, i.e, each sample is drawn from the same probability distribution.

A sequence of random variables \(X_1, X_1, \dots, X_n\) is said to converge in probability to a known random variable \(X\),
if for every number \(\epsilon >0 \), the following is true:

where,
\(X_n\): is the estimator or sample based random variable.
\(X\): is the known or limiting or target random variable.
\(\epsilon\): is the tolerance level or margin of error.

Read more about Limits

A sequence of random variables \(X_1, X_2, \dots, X_n\) is said to almost surely converge to a known random variable \(X\),
for \(n \ge 1\), if the following is true:

where,
\(X_n\): is the estimator or sample based random variable.
\(X\): is the known or limiting or target random variable.

If, \(X_n \xrightarrow{Almost ~ Sure} X \), \( \implies X_n \xrightarrow{Probability} X \)
But, converse is NOT true.

*Note: Almost Sure convergence is hardest to satisfy amongst all convergence, such as, convergence in probability, convergence in distribution, etc.

Read more about Limits

This law states that given a sequence of independent and identically distributed (IID) samples \(X_1, X_1, \dots, X_n\) from a random variable with finite mean, the sample mean (\(\bar{X_n}\)) converges in probability to the expected value \(E[X]\) or population mean (\( \mu \)).

Note: Does NOT guarantee that sample mean will be close to population mean,
but instead says that - the probability of sample mean being far away from the population mean is low.

Read more about Limits

This law states that given a sequence of independent and identically distributed (IID) samples \(X_1, X_1, \dots, X_n\) from a random variable with finite mean, the sample mean (\(\bar{X_n}\)) converges almost surely to the expected value \(E[X]\) or population mean (\( \mu \)).

Note:

• It guarantees that the sequence of sample averages itself converges to population mean, with exception of set of outcomes that has probability = 0.
• Almost certain guarantee; Much stronger statement than Weak Law of Large Numbers.
  
  

Read more about Limits

It gives an upper bound for the probability that a non-negative random variable will NOT exceed, based on its expected value.

Note: It gives a very loose upper bound.

Read more about Expectation

It states that the probability of a random variable deviating from its mean is small if its variance is small.

• It is a more powerful version of Markov’s Inequality.
  
• It uses variance of the distribution in addition to the expected value or mean.
  
• Also, it does NOT assume the random variable to be non-negative.
  
• It uses more information about the data i.e mean and variance.
  

Note: It gives a tighter upper bound than Markov’s Inequality.

It is an upper bound on the probability that a random variable deviates from its expected value.
It’s an exponentially decreasing bound on the “tail” of a random variable’s distribution, which can be calculated using its moment generating function.

• It is used for sum or average of independent random variables (not necessarily identically distributed).
• It provides exponentially tighter bounds, better than Chebyshev’s Inequality’s quadratic decay.
• It uses all moments to capture the full shape of the distribution, using the moment generating function(MGF).

Proof:

For the sum of ’n’ independent random variables,

Note: Used to compute how far the sum of independent random variables deviate from their expected value.

Read more about Moment Generating Function

It is a measure of the amount of information gained when a specific, individual event occurs, and is defined based on the probability of that event.
It is defined as the negative logarithm of the probability of the event.

• It is also called ‘Surprisal’.

Note: Logarithm(for base > 1) is a monotonically increasing function, so as x increases, log(x) also increases.

• So, if probability P(x) increases, then surprise factor S(x) decreases.
• Common events have a high probability of occurrence, hence a low surprise factor.
• Rare events have a low probability of occurrence, hence a high surprise factor.

Units:

• The unit of surprise factor with log base 2 is bits.
  
• with base ’e’ or natural log its nats (natural units of information).
  

It conveys how much ‘information’ we expect to gain from a random event.

• Entropy is the average or expected value of surprise factor of a random variable.
• More uniform the distribution ⇒ greater average surprise factor, since all possible outcomes are equally likely.

Read more about Expectation

It is a measure of the average ‘information gain’ or ‘surprise’ when using an imperfect model \(Q\) to encode events from a true model \(P\).

• It measures how surprised we are on an average, if the true distribution is \(P\), but we predict using another distribution \(Q\).
  

It measures the information lost when one probability distribution \(Q\) is used to approximate another distribution \(P\).
It quantifies the ‘extra cost’ in bits needed to encode data using the approximate distribution \(Q\) instead of the true distribution \(P\).

KL Divergence = Cross Entropy(P,Q) - Entropy(P)

Note:

• If \(P = Q\) ,i.e, P and Q are the same distributions, then KL Divergence = 0.
• KL divergence is NOT symmetrical ,i.e, \(D_{KL}(P \parallel Q) ⍯ (D_{KL}(Q \parallel P)\).

It is a smoothed and symmetric version of the Kullback-Leibler (KL) divergence and is calculated by averaging the
KL divergences between each of the two distributions and their combined average distribution.

• Symmetrical and smoothed version of KL divergence.
• Always finite; KL divergence can be infinite if \( P ⍯ 0 ~and~ Q = 0 \).
  • Makes JS divergence more stable for ML models where some predicted probabilities may be exactly 0.

It is the process of finding the best-fitting finite set of parameters \(\Theta\) for a model that assumes a specific probability distribution for the data.
It involves using the dataset to estimate the parameters (like the mean and standard deviation of a normal distribution) that define the model.

\(P(X \mid \Theta) \) : Probability of seeing data \(X: (X_1, X_2, \dots, X_n) \), given the parameters \(\Theta\) of the underlying probability distribution from which the data is assumed to be generated.

Goal of estimation:
We observed data, \( D = \{X_1, X_2, \dots, X_n\} \), and we want to infer the the unknow parameters \(\Theta\) of the underlying probability distribution, assuming that the data is generated I.I.D.
Read more for I.I.D

Note: Most of the time, from experience, we know the underlying probability distribution of data, such as, Bernoulli, Gaussian, etc.

It is the most popular frequentist approach to estimate the parameters of a model.
This method helps us find the parameters \(\Theta\) that make the data most probable.

Likelihood Function:
Say, we have data, \(D = X_1, X_2, \dots, X_n\) are I.I.D discrete random variable with PMF = \(P_{\Theta}(.)\)
Then, the likelihood function is the probability of observing the data, \(D = \{X_1, X_2, \dots, X_n\}\), given the parameters \(\Theta\).

Task:
Find the value of parameter \(\Theta_{ML}\) that maximizes the likelihood function.

In order to find the parameter \(\Theta_{ML}\) that maximises the likelihood function,
we need to take the first derivative of the likelihood function with respect to \(\Theta\) and equate it to zero.
But, taking derivative of a product is challenging, so we will take the logarithm on both sides.
Note: Log is a monotonically increasing function, i.e, as x increases, log(x) increases too.

Let us denote the log-likelihood function as \(\bar{L}\).

Therefore, Maximum Likelihood Estimation is the parameter \(\Theta_{ML}\) that maximises the log-likelihood function.

Bayesian statistics model parameters by updating initial beliefs (prior probabilities) with observed data to form a final belief (posterior probability) using Bayes’ Theorem.
Instead of a single point estimate, it provides a probability distribution over possible parameter values, which allows to quantify uncertainty and yields more robust models, especially with limited data.

Bayes’ Theorem:

Read more about Bayes’ Theorem

\(P(\Theta)\): Prior: Initial distribution of \(\Theta\) before seeing the data.
\(P(X \mid \Theta)\): Likelihood: Conditional distribution of data \(X\), given the parameter \(\Theta\).
\(P(\Theta \mid X)\): Posterior: Conditional distribution of parameter \(\Theta\), given the data \(X\).
\(P(X)\): Evidence: Probability of seeing the data \(X\).

It finds the mode(peak) of the posterior distribution.

• MAP has the minimum probability of error, since it picks single most probable value.
  

Minimizes the expected value of squared error.
Mean of the posterior distribution is the conditional expectation of parameter \(\Theta\), given the data.

• Posterior mean minimizes the the mean squared error.

\(\hat\Theta(X)\): Predicted value.
\(\Theta\): Actual value.

A single number that describes the central, typical, or representative value of a dataset, e.g, mean, median, and mode.
The mean is the average, the median is the middle value in a sorted list, and the mode is the most frequently occurring value.

• A single representative value can be used to compare different groups or distributions.

The artihmetic average of a set of numbers i.e sum all values and divide by the number of values.
\(mean = \frac{1}{n}\sum_{i=1}^{n}x_i\)

• Most common measure of central tendency.
• Represents the ‘balancing point’ of data.
• Sample mean is denoted by \(\bar{x}\), and population mean by \(\mu\).

Pros:

• Uses all datapoints in its calculation, providing a comprehensive measure.

Cons:

• Highly sensitive to outliers i.e exteme values.

The middle value of a sorted list of numbers. It divides the dataset into 2 equal halves.
Calculation:

• Arrange the data points in ascending order.
• If the number of data points is even, the median is the average of the two middle values.
• If the number of data points is odd, the median is the middle value i.e \((\frac{n+1}{2})^{th}\) element.

Pros:

• Not impacted by outliers, making it a more robust/reliable measure, especially for skewed distributions.

Cons:

• Does NOT use all the datapoints in its calculation.

The most frequently occurring value in a dataset.

• Dataset can have 1 mode i.e unimodal, 2 modes i.e bimodal, and more than 2 modes i.e multimodal.
  • If NO value repeats, then NO mode.

Pros:

• Only measure of central tendency that can be used for categorical/nominal data, such as, gender, blood group, level of education, etc.
• It can reveal important peaks in data distribution.

Cons:

• A dataset can have multiple modes, or no mode at all, which can make mode less informative.

The difference between the largest and smallest values in a dataset. Simplest measure of dispersion
\(range = max - min\)

Pros:

• Easy to calculate and understand.

Cons:

• Only considers the the 2 extreme values of dataset and ignores the distribution of data in between.
• Highly sensitive to outliers.

The average of the squared distance of each value from the mean.
Measures the spread of data points.

\(sample ~ variance = s^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \bar{x})^2\)

\(population ~ variance = \sigma^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \mu)^2\)

Cons:

• Highly sensitive to outliers, as squaring amplifies the weight of extreme data points.
• Less intuitive to understand, as the units are square of original units.

The square root of the variance, measures average distance of data points from the mean.

• Low standard deviation indicates that the data points are clustered around the mean, whereas
  high standard deviation means that the data points are spread out over a wide range.

\(s = sample ~ standard ~ deviation \)
\(\sigma = population ~ standard ~ deviation \)

It is the average of absolute deviation or distance of all data points from mean.

\( mad = \frac{1}{n}\sum_{i=1}^{n}|x_i - \bar{x}| \)

Pros:

• Less sensitive to outliers as compared to standard deviation..
• More intuitive and simpler to understand.

It measures the asymmetry of a data distribution.
Tells us whether the data is concentrated on one side of mean and is there a long tail stretching on the other side.

Positive Skew:

• Tail is longer on the right side of the mean.
• Bulk of data is on the left side of the mean, but there are a few very high values pulling the mean towards the right.
• Mean > Median > Mode.

Negative Skew:

• Tail is longer on the left side of the mean.
• Bulk of data is on the right side of the mean, but there are a few very high values pulling the mean towards the left.
• Mean < Median < Mode.

Zero Skew:

• Perfectly symmetrical like a normal distribution.
• Mean = Median = Mode.

It measures the “tailedness” of a data distribution.
It describes how much the data is concentrated in tails (fat or thin) versus the center.

• It can tell us about the frequency of outliers in the data.
  • Thick tails => More outliers.

Excess Kurtosis:
Excess kurtosis is calculated by subtracting 3 from standard kurtosis in order to compare with normal distribution.
Normal distribution has kurtosis = 3.

Mesokurtic:

• Excess kurtosis = 0 i.e normal kurtosis.
• Tails are neither too thick nor too thin.

Leptokurtic:

• High kurtosis, i.e, excess kurtosis > 0 (+ve).
• Heavy or thick tails => High probability of outliers.
• Sharp peak => High concentration of data around mean.
• E.g: Student’s t-distribution, Laplace distribution, etc.
• High risk stock portfolios.

Platykurtic:

• Low kurtosis, i.e, excess kurtosis < 0 (-ve).
• Thin tails => Low probability of outliers.
• Low peak => more uniform distribution of values.
• E.g: Uniform distribution, Bernoulli(P=0.5) distribution, etc.
• Investment in fixed deposits.

It indicates the percentage of scores in a dataset that are equal to or below a specific value.
Here, the complete dataset is divided into 100 equal parts.

• \(k^{th}\) percentile => at least \(k\) percent of the data points are equal to or below the value.
• It is a relative comparison, i.e, compares a score with the entire group’s performance.
• Quartiles are basis for box plots.

They are special percentiles that divide the complete dataset into 4 equal parts.

Q1 => 25th percentile, value below which 25% of the data falls.
Q2 => 50th percentile, value below which 50% of the data falls; median.
Q3 => 75th percentile, value below which 75% of the data falls.

It is the single number that measures the spread of middle 50% of the data, i.e Q1-Q3.

• More robust measure of spread than range as is NOT impacted by outliers.

IQR = Q3 - Q1

It measures the direction of linear relationship between two variables \(X\) and \(Y\).

\(N\) = size of population
\(\mu_{x}\) = population mean of \(X\)
\(\mu_{y}\) = population mean of \(Y\)

\(n\) = size of sample
\(\bar{x}\) = sample mean of \(X\)
\(\bar{y}\) = sample mean of \(Y\)

Note: We have a term (n-1) instead of n in the denominator to make it an unbiased estimate, called Bessel’s Correction.

If both \((x_i - \bar{x})\) and \((y_i - \bar{y})\) have the same sign, then the product is positive(+ve).
If both \((x_i - \bar{x})\) and \((y_i - \bar{y})\) have opposite signs, then the product is negative(-ve).
The final value of covariance depends on the sum of the above individual products.

\( \begin{aligned} \text{Cov}(X, Y) &> 0 &&\Rightarrow \text{ } X \text{ and } Y \text{ increase or decrease together} \\ \text{Cov}(X, Y) &= 0 &&\Rightarrow \text{ } \text{No linear relationship} \\ \text{Cov}(X, Y) &< 0 &&\Rightarrow \text{ } \text{If } X \text{ increases, } Y \text{ decreases (and vice versa)} \end{aligned} \)

Limitation:
Covariance is scale-dependent, i.e, units of X and Y impact its magnitude.
This makes it hard to make comparisons of covariance across different datasets.
E.g: Covariance between age and height will NOT be same as the covariance between years of experience and salary.

Note:It only measures the direction of the relationship, but does NOT give any information about the strength of the relationship.

It measures both the strength and direction of the linear relationship between two variables \(X\) and \(Y\).
It is a standardized version of covariance that gives a dimensionless measure of linear relationship.

There are 2 popular ways to calculate correlation coefficient:

1. Pearson Correlation Coefficient (r)
   
2. Spearman Rank Correlation Coefficient (\(\rho\))
   

It is a standardized version of covariance and most widely used measure of correlation.
Assumption: Data is normally distributed.

\(\sigma_{x}\) and \(\sigma_{y}\) are the standard deviations of \(X\) and \(Y\).

Range of \(r\) is between -1 and 1.
\(r = 1\) => perfect +ve linear relationship between X and Y
\(r = -1\) => perfect -ve linear relationship between X and Y
\(r = 0\) => NO linear relationship between X and Y.

Note: A correlation coefficient of 0.9 means that there is a strong linear relationship between X and Y, irrespective of their units.

It is a measure of the strength and direction of the monotonic relationship between two ranked variables \(X\) and \(Y\).
It captures monotonic relationship, meaning the variables move in the same or opposite direction,
but not necessarily a linear relationship.

• It is used when Pearson’s correlation is not suitable, such as, ordinal data, or when the continuous data does not meet the assumptions of linear methods, such as, Pearson’s correlation.
• Non-parametric measure of correlation that uses ranks instead of raw data.
• Quantifies how well the ranks of one variable predict the ranks of the other variable.
• Range of \(\rho\) is between -1 and 1.
  

It is the true average of the entire group.
It describe the central tendency of the entire population.

\( \mu = \frac{1}{N}\sum_{i=1}^{N}x_i \)
N: Number of data points

It is the average of a smaller representative subset (a sample) of the entire population.
It provides an estimate of the population mean.

\( \bar{x} = \frac{1}{n}\sum_{i=1}^{n}x_i \)
n: size of sample

This law states that for a sequence of of I.I.D random variables \( X_1, X_2, \dots, X_n \),
with finite mean and variance, the distribution of the sample mean \( \bar{X} \) approaches a normal distribution as \( n \rightarrow \infty \), regardless of its original population distribution.
The distribution of the sample mean is : \( \bar{X} \sim N(\mu, \sigma^2/n)\)

Let, \( X_1, X_2, \dots, X_n \) are I.I.D random variables.

• Population mean = \(E[X_i] = \mu < \infty\)
• Population Variance = \(Var[X_i] = \sigma^2 < \infty \)
• Sample mean = \( \bar{X_n} = \frac{1}{n}\sum_{i=1}^{n}X_i = \frac{1}{n}(X_1 + X_2+ \dots +X_n) \)
• Variance of sample means = \( Var[\bar{X_n}] = Var[\frac{1}{n}(X_1+ X_2+ \dots+ X_n)]\)

Now, let’s calculate the variance of sample means.
We know that:

1. \(Var[X+Y] = Var[X] + Var[Y] \), for independent random variables X and Y.
2. \(Var[cX] = c^2Var[X] \), for constant ‘c’.

Let’s apply above 2 rules on the variance of sample means equation above:

Since, standard deviation = \(\sigma = \sqrt{Variance}\)
Therefore, Standard Deviation\([\bar{X_n}] = \sqrt{\frac{\sigma^2}{n}} = \frac{\sigma}{\sqrt{n}}\)
The standard deviation of the sample means is also known as “Standard Error”.

Note: We can also standardize the sample mean, i.e, mean centering and variance scaling.
Standardisation helps us to use the Z-tables of normal distribution.

We know that, a standardized random variable \(Y_i = \frac{X_i - \mu}{\sigma}\)
Similarly, standardized sample mean:

Note: For practical purposes, \(n \ge 30\) is considered as a sufficient sample size for the CLT to hold.

It is a range of values that is likely to contain the true population mean, based on a sample.
Instead of giving a point estimate, it gives a range of values with confidence level.

For normal distribution, confidence interval :

\(\bar{X}\): Sample mean
\(Z\): Z-score corresponding to confidence level
\(n\): Sample size
\( \sigma \): Population Standard Deviation

Applications:

• A/B testing, i.e., compare 2 or more versions of a product.
• ML model performance evaluation, i.e, instead of giving a single performance score of say 85%,
  it is better to provide a 95% confidence interval, such as, [82.5%, 87.8%].
  

It is a statistical inference framework used to make decisions about a population parameter, such as, the mean, variance, distribution, correlation, etc., based on a sample of data. It provides a formal method to evaluate competing claims.

Null Hypothesis (\(H_0\)):
Status quo or no-effect or no difference statement; almost always contains a statement of equality.

Alternative Hypothesis (\(H_1 ~or~ H_a\)):
The statement representing an effect, a difference, or a relationship.
It must be true if the null hypothesis is rejected.

It is the probability of wrongly rejecting a true null hypothesis, known as a Type I error or false +ve rate.

• Tolerance level of wrongly accepting alternate hypothesis.
• If the p-value < \(\alpha\), we reject the null hypothesis and conclude that the finding is statistically NOT so significant..

It is a specific point on the test-statistic distribution that defines the boundaries of the null hypothesis acceptance/rejection region.

• It tells us that at what value (\(t_{\alpha}\)) of test statistic will the area under curve be equal to the significance level \(\alpha\).

• For a right tailed/sided test:

  • if \(t_{obs} > t_{\alpha} => p_{value} < \alpha\); therefore, reject null hypothesis.
  • if \(t_{obs} < t_{\alpha} => p_{value} \ge \alpha\); therefore, failed to reject null hypothesis.

  

It is the probability that a hypothesis test will correctly reject a false null hypothesis (\(H_{0}\)) when the alternative hypothesis (\(H_{a}\)) is true.

• Power of test = \(1 - \beta\)

• Probability of correctly accepting alternate hypothesis (\(H_{a}\))

• \(\alpha\): Probability of wrongly accepting alternate hypothesis \(H_{a}\)

• \(\beta\): Probability of wrongly rejecting alternate hypothesis \(H_{a}\)

  

It is a standardized objective measure that complements p-value by clarifying whether a statistically significant finding has any real world relevance.
It quantifies the magnitude of relationship between two variables.

• Larger effect size => more impactful effect.
• Standardized (mean centered + variance scaled) measure allows us to compare the imporance of effect across various studies or groups, even with different sample sizes.

Effect size is measured using Cohen’s d formula:

\(\bar{X}\): Sample mean
\(s_p\): Pooled Standard deviation
\(n\): Sample size
\(s\): Standard deviation

Note: Theoretically, Cohen’s d value can range from negative infinity to positive infinity.
but for practical purposes, we use the following value:
small effect (\(d=0.2\)), medium effect (\(d=0.5\)), and large effect (\(d\ge 0.8\)).

• More overlap => less effect i.e low Cohen’s d value.

  

It is a statistical test that is used to determine whether the sample mean is equal to a hypothesized value or
is there a significant difference between the sample means of 2 groups.

• It is a parametric test, since it assumes data to be approximately normally distributed.
• Appropriate when:
  • sample size n < 30.
  • population standard deviation \(\sigma\) is unknown.
• It is based on Student’s t-distribution.

It is a continuous probability distribution that is a symmetrical, bell-shaped curve similar to the normal distribution but with heavier tails.

• Shape of the curve or mass in tail is controlled by degrees of freedom.

  

There are 3 types of T-Test:

1. 1-Sample T-Test: Test if sample mean differs from hypothesized value.
2. 2-Sample T-Test: Test whether there is a significant difference between the means of two independent groups.
3. Paired T-Test: Test whether 2 related samples differ, e.g., before and after.

It is used to test whether the sample mean is equal to a known/hypothesized value.
Test statistic (t):

where,
\(\bar{x}\): sample mean
\(\mu\): hypothesized value
\(s\): sample standard deviation
\(n\): sample size
\(\nu = n-1 \): degrees of freedom

It is used to determine whether there is a significant difference between the means of two independent groups.
There are 2 types of 2-sample t-test:

1. Unequal Variance
2. Equal Variance

Unequal Variance:
In this case, the variance of 2 independent groups is not equal.
Also called, Welch’s t-test.
Test statistic (t):

Equal Variance:
In this case, both samples come from equal or approximately equal variance.
Test statistic (t):

Here, degrees of freedom (for equal variance) \(\nu\) = \(n_1 + n_2 - 2\).

\(\bar{x}\): sample mean
\(s\): sample standard deviation
\(n\): sample size
\(\nu\): degrees of freedom

It is a statistical test used to determine whether there is a significant difference between mean of 2 groups or sample and population mean.

• It is a parametric test, since it assumes data to be normally distributed.
• Appropriate when:
  • sample size \(n \ge 30\).
  • population standard deviation \(\sigma \) is known.
• It is based on Gaussian/normal distribution.
• It compares the difference between means relative to standard error, i.e, standard deviation of sampling distribution of sample mean.

There are 2 types of Z-Test:

• 1-Sample Z-Test: Used to compare the mean of a sample mean with a population mean.
• 2-Sample Z-Test: Used to compare the sample means of 2 independent samples.

It is a standardized score that measures how many standard deviations a particular data point is away from the population mean \(\mu\).

• Transform a normal distribution \(\mathcal{N}(\mu, \sigma^2)\) to a standard normal distribution \(Z \sim \mathcal{N}(0, 1)\).
• Standardized score helps us compare values from different normal distributions.

Z-score is calculated as:

x: data point
\(\mu\): population mean
\(\sigma\): population standard deviation

e.g:

1. Z-score of 1.5 => data point is 1.5 standard deviations above the mean.
2. Z-score of -2.0 => data point is 2.0 standard deviations below the mean.

Z-score helps to define probability areas:

• 68% of the data points fall within \(\pm 1 \sigma\).
• 95% of the data points fall within \(\pm 2 \sigma\).
• 99.7% of the data points fall within \(\pm 3 \sigma\).

Note:

• Z-Test applies the concept of Z-score to sample mean rather than a single data point.

It is used to test whether the sample mean \(\bar{x}\) is significantly different from a known population mean \(\mu\).

Test Statistic:

\(\bar{x}\): sample mean
\(\mu\): hypothesized population mean
\(\sigma\): population standard deviation
\(n\): sample size
\(\sigma / \sqrt{n}\): standard error of mean

Read more about Standard Error

Note: Test statistic Z follows a standard normal distribution \(Z \sim \mathcal{N}(0, 1)\).

It is used to test whether the sample means \(\bar{x_1}\) and \(\bar{x_2}\) of 2 independent samples are significantly different from each other.
Test Statistic:

\(\bar{x_1}\): sample mean of first sample
\(\bar{x_2}\): sample mean of second sample
\(\sigma_1\): population standard deviation of first sample
\(\sigma_2\): population standard deviation of second sample
\(n_1\): sample size of first sample
\(n_2\): sample size of second sample

Note: Test statistic Z follows a standard normal distribution \(Z \sim \mathcal{N}(0, 1)\).

It is a statistical hypothesis test used to determine if there is a significant difference between the proportion of a characteristic in two independent samples or to compare a sample proportion to a known population value

• It is used when dealing with categorical data, such as, success/failure, male/female, yes/no etc.

It is of 2 types:

1. 1-Sample Z-Test of Proportion: Used to test whether the observed proportion in a sample differs from hypothesized proportion.
2. 2-Sample Z-Test of Proportion: Used to compare whether the 2 independent samples differ in their proportions.

The categorical data,i.e success/failure, is discrete that can be modeled as Bernoulli distribution.
Let’s understand how this Bernoulli random variable can be approximated as a Gaussian distribution for a very large sample size, using Central Limit Theorem.
Read more about Central Limit Theorem

It is used to test whether the observed proportion in a sample differs from hypothesized proportion.
\(\hat{p} = \frac{X}{n}\): Proportion of success observed in a sample
\(p_0\): Specific propotion value under the null hypothesis
\(SE_0\): Standard error of sample proportion under the null hypothesis
Z-Statistic: Measures how many standard errors is the observed sample proportion \(\hat{p}\) away from \(p_0\)
Test Statistic:

It is used to compare whether the 2 independent samples differ in their proportions.

• Standard test used in A/B testing.

A random variable Q is said to follow a chi-square distribution with ’n’ degrees of freedom,i.e \(\chi^2(n)\),
if it is the sum of squares of ’n’ independent random variables that follow a standard normal distribution, i.e, \(N(0,1)\).

Key Properties:

1. Non-negative, since sum of squares.
2. Asymmetric, right skewed.
3. Shape depends on the degrees of freedom; as \(\nu\) increases, the distribution becomes more symmetric and approaches a normal distribution.

It is formed by squaring the approximately standard normal counts above, and summing them up.
For \(k\) categories, the test statistic is:

Note: For very large ’n’, the Pearson’s chi-square (\(\chi^2\)) test statistic follows a chi-square (\(\chi^2\)) distribution.

It is used to compare the observed frequency distribution of a single categorical variable to a hypothesized or expected probability distribution.
It can be used to determine whether a sample taken from a population follows a particular distribution, e.g., uniform, normal, etc.

Test Statistic:

\(O_{i}\): Observed count for \(i^{th}\) category
\(E_{i}\): Expected count for \(i^{th}\) category, under null hypothesis \(H_0\)
\(k\): Number of categories
\(\nu\): Degrees of freedom = k - 1- m
\(m\): Number of parameters estimated from sample data to determine the expected probability
Note: Typical m=0, since, NO parameters are estimated.

It is used to determine whether an association exists between two categorical variables, using a contingency(dependency) table.
It is a non-parametric test, i.e, does NOT make any assumption about the underlying distribution of the data.

Test Statistic:

\(O_{ij}\): Observed count for \(cell_{i,j}\)
\(E_{ij}\): Expected count for \(cell_{i,j}\), under null hypothesis \(H_0\)
\(R\): Number of rows
\(C\): Number of columns
\(\nu\): Degrees of freedom = (R-1)*(C-1)

Let’s understand the above test statistic in more detail.
We know that, if 2 random variables A & B are independent, then,
\(P(A \cap B) = P(A, B) = P(A)*P(B)\)
i.e Joint Probability = Product of marginal probabilities.

Null Hypothesis \(H_0\): \(A\) and \(B\) are independent.
Alternative Hypothesis \(H_a\): \(A\) and \(B\) are dependent or associated.
N = Sample size
\(P(A_i) \approx \frac{Row ~~ Total_i}{N}\)

\(P(B_j) \approx \frac{Col ~~ Total_j}{N}\)

\(E_{ij}\) : Expected count for \(cell_{i,j}\) = \( N*P(A_i)*P(B_j)\)

=> \(E_{ij}\) = \(N*\frac{Row ~~ Total_i}{N}*\frac{Col ~~ Total_j}{N}\)

=> \(E_{ij}\) = \(\frac{Row ~~ Total_i * Col ~~ Total_j}{N}\)

\(O_{ij}\): Observed count for \(cell_{i,j}\)

It is a table that summarizes model’s predictions against the actual class labels, detailing where the model succeeded and where it failed.
It is used for binary or multi-class classification problems.

Type-1 Error:
It is the number of false positives.
e.g.: Model predicted that a patient has diabetes, but the patient actually does NOT have diabetes; “false alarm”.

Type-2 Error:
It is the number of false negatives. e.g.: Model predicted that a patient does NOT have diabetes, but the patient actually has diabetes; “a miss”.

It is a graphical plot that shows the discriminating ability of a binary classifier system, as its discrimination threshold is varied.
Y-axis: True Positive Rate (TPR), Recall, Sensitivity
\(TPR = \frac{TP}{TP + FN}\)

X-axis: False Positive Rate (FPR); (1 - Specificity)
\(FPR = \frac{FP}{FP + TN}\)

Note: A binary classifier model outputs a probability score between 0 and 1.
and a threshold (default=0.5) is applied to the probability score to get the final class label.
\(p \ge 0.5\) => Positive Class
\(p < 0.5\) => Negative Class

Algorithm:

1. Sort the data by the probability score in descending order.
   
2. Set each probability score as the threshold for classification and calculate the TPR and FPR for each threshold.
   
3. Plot each pair of (TPR, FPR) for all ’n’ data points to get the final ROC curve.
   

e.g.:

Set the threshold \(\tau_1\) = 0.95, calculate \({TPR}_1, {FPR}_1\)
Set the threshold \(\tau_2\) = 0.85, calculate \({TPR}_2, {FPR}_2\)
Set the threshold \(\tau_3\) = 0.72, calculate \({TPR}_3, {FPR}_3\)
…
…
Set the threshold \(\tau_n\) = 0.05, calculate \({TPR}_n, {FPR}_n\)

Now, we have ’n’ pairs of (TPR, FPR) for all ’n’ data points.
Plot the points on a graph to get the final ROC curve.

AU ROC = AUC = Area under the ROC curve = Area under the curve

Note:

1. If AUC < 0.5, then invert the labels of the classes.
2. ROC does NOT perform well on imbalanced data.
   • Either balance the data or
   • Use Precision-Recall curve.

It is used to evaluate the performance of a binary classifier model across various thresholds.
It is similar to the ROC curve, but it uses Precision instead of TPR on the Y-axis.
Plots Precision (Y-axis) against Recall (X-axis) for different classification thresholds.
Note: It is useful when the data is imbalanced.

AU PRC = PR AUC = Area under Precision-Recall curve

It is a collection of scalars(numbers) that has both magnitude and direction.
Geometrically, it is a line segment in space characterized by its length(magnitude) and direction.
By convention, we represent vectors as column vectors.
e.g.: \(\vec{x} = \mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}_{\text{n×1}}\) i.e ’n’ rows and 1 column.

Important: In machine learning, we will use bold notation \(\mathbf{v}\) to represent vectors, instead of arrow notation \(\mathbf{v}\).

Swap the rows and columns, i.e, a column vector becomes a row vector after transpose.
e.g: \(\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_d \end{bmatrix}_{\text{d×1}}\)

\(\mathbf{v}^\mathrm{T} = \begin{bmatrix} v_1 & v_2 & \cdots & v_d \end{bmatrix}_{\text{1×d}}\)

The length (or magnitude or norm) of a vector \(\mathbf{v}\) is the distance from the origin to the point represented
by \(\mathbf{v}\) in n-dimensional space.

\(\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_d \end{bmatrix}_{\text{d×1}}\)

Length of vector:

Note: The length of a zero vector is 0.

The direction of a vector tells us where the vector points in space, independent of its length.

Direction of vector:

It is a collection of vectors that can be added together and scaled by numbers (scalars), such that, the results are still in the same space.
Vector space or linear space is a non-empty set of vectors equipped with 2 operations:

1. Vector addition: for any 2 vectors \(a, b\), \(a + b\) is also in the same vector space.
   
2. Scalar multiplication: for a vector \(\mathbf{v}\), \(\alpha\mathbf{v}\) is also in the same vector space; where \(\alpha\) is a scalar.
   Note: These operations must satisfy certain rules (called axioms), such as, associativity, commutativity, distributivity, existence of a zero vector, and additive inverses.
   

e.g.: Set of all points in 2D is a vector space.

A set of vectors are linearly independent if NO vector in the set can be expressed as a linear combination of the other vectors in the set.

The only solution for :
\(\alpha_1 \mathbf{u}_1 + \alpha_2 \mathbf{u}_2 + \cdots + \alpha_k \mathbf{u}_k\) = 0
is \(\alpha_1 = \alpha_2, \cdots, = \alpha_k = 0\).

e.g.:

1. The below 3 vectors are linearly independent.
   \(\mathbf{u} = \begin{bmatrix} 1 \\ 1 \\ 1 \\ \end{bmatrix}_{\text{3×1}}\), \(\mathbf{v} = \begin{bmatrix} 1 \\ 2 \\ 3 \\ \end{bmatrix}_{\text{3×1}}\), \(\mathbf{w} = \begin{bmatrix} 1 \\ 3 \\ 6 \\ \end{bmatrix}_{\text{3×1}}\)

2. The below 3 vectors are linearly dependent.
   \(\mathbf{u} = \begin{bmatrix} 1 \\ 1 \\ 1 \\ \end{bmatrix}_{\text{3×1}}\), \(\mathbf{v} = \begin{bmatrix} 1 \\ 2 \\ 3 \\ \end{bmatrix}_{\text{3×1}}\), \(\mathbf{w} = \begin{bmatrix} 2 \\ 4 \\ 6 \\ \end{bmatrix}_{\text{3×1}}\)
   
   because, \(\mathbf{w} = 2\mathbf{v}\), and we have a non-zero solution for the below equation:
   \(\alpha_1 \mathbf{u} + \alpha_2 \mathbf{v} + \alpha_3 \mathbf{w} = 0\);
   \(\alpha_1 = 0, \alpha_2 = -2, \alpha_3 = 1\) is a valid non-zero solution.

Note:

1. A common method to check linear independence is to arrange the column vectors in a matrix form and calculate its determinant, if determinant ≠ 0, then the vectors are linearly independent.
2. If number of vectors > number of dimensions, then the vectors are linearly dependent.
   Since, the \((n+1)^{th}\) vector can be expressed as a linear combination of the other ’n’ vectors in n-dimensional space.
3. In machine learning, if a feature can be expressed in terms of other features, then it is linearly dependent,
   and the feature is NOT bringing any new information.
   e.g.: In 2 dimensions, 3 vectors \(\mathbf{x_1}, \mathbf{x_2}, \mathbf{x_3} \) are linearly dependent.

Span of a set of vectors is the geometric shape by all possible linear combinations of those vectors, such as, line, plane, or higher dimensional volume.
e.g.:

1. Span of a single vector (1,0) is the entire X-axis.
2. Span of 2 vectors (1,0) and (0,1) is the entire X-Y (2D) plane, as any vector in 2D plane can be expressed as a linear combination of the 2 vectors - (1,0) and (0,1).

It is the minimal set of linearly independent vectors that spans or defines the entire vector space, providing a unique co-ordinate system for every vector within the space.
Every vector in the vector space can be represented as a unique linear combination of the basis vectors.
e.g.:

1. X-axis(1,0) and Y-axis(0,1) are the basis vectors of 2D space or form the co-ordinate system.
   
2. \(\mathbf{u} = (3,1)\) and \(\mathbf{v} = (-1, 2) \) are the basis of skewed or parallelogram co-ordinate system.
   

Note: Basis = Dimensions

Two vectors are orthogonal if their dot product is 0.
A set of vectors \(\mathbf{x_1}, \mathbf{x_2}, \cdots ,\mathbf{x_n} \) are said to be orthogonal if:
\(\mathbf{x_i} \cdot \mathbf{x_j} = 0 \forall i⍯j\), for every pair, i.e, every pair must be orthogonal.
e.g.:

1. \(\mathbf{u} = (1,0)\) and \(\mathbf{v} = (0,1) \) are orthogonal vectors.
   
2. \(\mathbf{u} = (1,0)\) and \(\mathbf{v} = (1,1) \) are NOT orthogonal vectors.
   

Note:

1. Orthogonal vectors are linearly independent, but the inverse may NOT be true,
   i.e, linear independence does NOT imply that vectors are orthogonal. e.g.:
   Vectors \(\mathbf{u} = (2,0)\) and \(\mathbf{v} = (1,3) \) are linearly independent but NOT orthogonal.
   Since, \(\mathbf{u} \cdot \mathbf{v} = 2*1 + 3*0 = 2 ⍯ 0\).
   

2. Orthogonality is the most extreme case of linear independence, i.e \(90^{\degree}\) apart or perpendicular.
   

Orthonormal vectors are vectors that are orthogonal and have unit length.
A set of vectors \(\mathbf{x_1}, \mathbf{x_2}, \cdots ,\mathbf{x_n} \) are said to be orthonormal if:
\(\mathbf{x_i} \cdot \mathbf{x_j} = 0 \forall i⍯j\) and \(\|\mathbf{x_i}\| = 1\), i.e, unit vector.

e.g.:

1. \(\mathbf{u} = (1,0)\) and \(\mathbf{v} = (0,1) \) are orthonormal vectors.
   
2. \(\mathbf{u} = (1,0)\) and \(\mathbf{v} = (0,2) \) are NOT orthonormal vectors.
   

It is a set of vectors that functions as a basis for a space while also being orthonormal,
meaning each vector is a unit vector (has a length of 1) and all vectors are mutually perpendicular (orthogonal) to each other.
A set of vectors \(\mathbf{x_1}, \mathbf{x_2}, \cdots ,\mathbf{x_n} \) are said to be orthonormal basis of a vector space \(\mathbb{R}^{n \times 1}\), if every vector:

e.g.:

1. \(\mathbf{u} = (1,0)\) and \(\mathbf{v} = (0,1) \) form an orthonormal basis for 2-D space.
   
2. \(\mathbf{u} = (\frac{1}{\sqrt{2}},\frac{1}{\sqrt{2}})\) and \(\mathbf{v} = (-\frac{1}{\sqrt{2}},-\frac{1}{\sqrt{2}}) \) also form an orthonormal basis for 2-D space.
   

Note: In a n-dimensional space, there are only \(n\) possible orthonormal bases.

It is a square matrix that whose rows and columns are orthonormal vectors, i.e, they are perpendicular to each other and have a unit length.

\( \mathbf{A} \mathbf{A}^\mathrm{T} = \mathbf{A}^\mathrm{T} \mathbf{A} = \mathbf{A} \mathbf{A}^{-1} = \mathbf{I} \)
=> \( \mathbf{A}^\mathrm{T} = \mathbf{A}^{-1} \)

Note:

1. Orthogonal matrix preserves the length and angles of vectors, acting as rotation or reflection in geometry.
2. The determinant of an orthogonal matrix is always +1 or -1, because \( \mathbf{A} \mathbf{A}^\mathrm{T} = \mathbf{I} \)

Let’s check out why the rows and columns of an orthogonal matrix are orthonormal.
\( \mathbf{A} = \begin{bmatrix} a_{11} & a_{12} \\ \\ a_{21} & a_{22} \end{bmatrix} _{\text{2 x 2}} \), => \( \mathbf{A}^\mathrm{T} = \begin{bmatrix} a_{11} & a_{21} \\ \\ a_{12} & a_{22} \end{bmatrix} _{\text{2 x 2}} \)

Since, \( \mathbf{A} \mathbf{A}^\mathrm{T} = \mathbf{I} \)
\( \begin{bmatrix} a_{11} & a_{12} \\ \\ a_{21} & a_{22} \end{bmatrix} \begin{bmatrix} a_{11} & a_{21} \\ \\ a_{12} & a_{22} \end{bmatrix} = \begin{bmatrix} a_{11}^2 + a_{12}^2 & a_{11}a_{21} + a_{12}a_{22} \\ \\ a_{21}a_{11} + a_{21}a_{12} & a_{21}^2 + a_{22}^2 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ \\ 0 & 1 \end{bmatrix} \)

Equating the terms, we get:
\( a_{11}^2 + a_{12}^2 = 1 \) => row 1 is a unit vector
\( a_{21}^2 + a_{22}^2 = 1 \) => row 2 is a unit vector
\( a_{11}a_{21} + a_{12}a_{22} = 0 \) => row 1 and row 2 are orthogonal to each other, since dot product = 0
\( a_{21}a_{11} + a_{21}a_{12} = 0 \) => row 1 and row 2 are orthogonal to each other, since dot product = 0

Therefore, the rows and columns of an orthogonal matrix are orthonormal.

It decomposes any matrix into a rotation, a scaling (based on singular values), and another rotation.
It is a generalization of the eigen value decomposition(for square matrix) to rectangular matrices.
Any rectangular matrix can be decomposed into a product of three matrices using SVD, as follows:

\(\mathbf{U}\): Set of orthonormal eigen vectors of \(\mathbf{AA^T}_{m \times m} \)
\(\mathbf{V}\): Set of orthonormal eigen vectors of \(\mathbf{A^TA}_{n \times n} \)
\(\mathbf{\Sigma}\): Rectangular diagonal matrix, whose diagonal values are called singular values; square root of non-zero eigen values of \(\mathbf{AA^T}\).

Note: The number of non-zero diagonal entries in \(\mathbf{\Sigma}\) = rank of matrix \(\mathbf{A}\).
Rank (r): Number of linearly independent rows or columns of a matrix.

\( \Sigma = \begin{bmatrix} \sigma_{11} & 0 & \cdots & 0 \\ \\ 0 & \sigma_{22} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ \cdots & \cdots & \sigma_{rr} & \vdots \\ 0 & 0 & \cdots & 0 \end{bmatrix} \), such that, \(\sigma_{11} \geq \sigma_{22} \geq \cdots \geq \sigma_{rr}\), r = rank of \(\mathbf{A}\).

Singular value decomposition, thus refers to the set of scale factors \(\mathbf{\Sigma}\) that are fundamentally linked to the matrix’s singularity and rank.

The process of approximating any matrix by a matrix of a lower rank, using singular value decomposition.
It is used for data compression.

Any matrix A of rank ‘r’ can be written as sum of rank=1 outer products:

\( \mathbf{u}_i : i^{th}\) column vector of \(\mathbf{U}\)
\( \mathbf{v}_i^T : i^{th}\) column vector of \(\mathbf{V}^T\)
\( \sigma_i : i^{th}\) singular value, i.e, diagonal entry of \(\mathbf{\Sigma}\)

Since, the singular values are arranged from largest to smallest, i.e, \(\sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_r\),
The top ranks capture the vast majority of the information or variance in the matrix.

So, in order to get the best rank-k approximation of the matrix, we simply truncate the summation after the k’th term.

The approximation \(\mathbf{A_k}\) is called the low rank approximation of \(\mathbf{A}\), which is achieved by keeping only the largest singular values and corresponding vectors.

Applications:

1. Image compression
2. Data compression, such as, LoRA (Low Rank Adaptation)

It is a dimensionality reduction technique that finds the direction of maximum variance in the data.
Note: Some loss of information will always be there in dimensionality reduction, because there will be some variability in data along the direction that is dropped, and that will be lost.

Goal:
Fundamental goal of PCA is to find the new set of orthogonal axes, called the principal components, onto which the data can be projected, such that, the variance of the projected data is maximum.

Note: Modern libraries (like sklearn.decomposition.PCA) default to Singular Value Decomposition (SVD) for PCA; but we will use Eigen Value Decomposition to understand the algorithm.

It is a measure of the size of a vector or distance from the origin.
Vector norm is a function that maps a vector to a real number, i.e, \({\| \cdot \|} : \mathbb{R}^n \rightarrow \mathbb{R}\).
Vector Norm should satisfy following 3 properties:

1. Non-Negativity:
   Norm is always greater than or equal to zero,
   \( {\| x \|} \ge 0\), and \( {\| x \|} = 0\), if and only if \(\vec{x} = \vec{0}\).
   
2. Homogeneity (or Scaling):
   \( {\| \alpha x \|} = |\alpha| {\| x \|} \).
   
3. Triangle Inequality:
   \( {\| x + y \|} \le {\| x \|} + {\| y \|} \).
   

P-Norm:
It is a generalised form of most common family of vector norms, also called Minkowski norm.
It is defined as:

We can change the value of \(p\) to get different norms.

L1 Norm:
It is the sum of absolute values of all the elements of a vector; also known as Manhattan distance.
p=1:

L2 Norm:
It is the square root of the sum of squares of all the elements of a vector; also known as Euclidean distance.
p=2:

L-\(\infty\) Norm:
It is the maximum of absolute values of all the elements of a vector; also known as Chebyshev distance.
p=\(\infty\):

It is a function that assigns non-negative size or magnitude to a matrix.
Matrix Norm is a function that maps a matrix to a non-negative real number, i.e, \({\| \cdot \|} : \mathbb{R}^{m \times n} \rightarrow \mathbb{R}\)
It should satisfy following 3 properties:

1. Non-Negativity:
   Norm is always greater than or equal to zero,
   \( {\| x \|} \ge 0\), and \( {\| x \|} = 0\), if and only if \(\vec{x} = \vec{0}\).
   
2. Homogeneity (or Scaling):
   \( {\| \alpha x \|} = |\alpha| {\| x \|} \).
   
3. Triangle Inequality:
   \( {\| x + y \|} \le {\| x \|} + {\| y \|} \).
   

There are 2 types of matrix norms:

1. Element wise norms, e.g,, Frobenius norm
2. Vector induced norms Frobenius Norm:
   
   
   

Frobenius Norm:
It is equivalent to the Euclidean norm of the matrix if it were flattened into a single vector.
If A is a matrix of size \(m \times n\), then, Frobenius norm is defined as:

\(\sigma_i\) is the \(i\)th singular value of matrix A.

Vector Induced Norm:
It measures the maximum stretching a matrix can apply when multiplied with a vector,
where the vector has a unit length under the chosen vector norm.

Matrix Induced by Vector P-Norm:
P-Norm:

P=1 Norm:

P=\(\infty\) Norm:

P=2 Norm:
Also called Spectral norm, i.e, maximum factor by which the matrix can stretch a unit vector in Euclidean norm.

Gradient Descent:
It is a first order iterative optimization algorithm that is used to find the local minimum of a differentiable function.
It iteratively adjusts the parameters of the model in the direction opposite to the gradient of cost function, since moving opposite to the direction of gradient leads towards the minima.

Algorithm:

1. Initialize the weights/parameters with random values.

2. Calculate the gradient of the cost function at current parameter values.

3. Update the parameters using the gradient.

   \[ w_{new} = w_{old} - \eta \cdot \frac{\partial f}{\partial w_{old}} \\[10pt] \eta: \text{ learning rate or step size to take for each parameter update} \]

4. Repeat steps 2 and 3 iteratively until convergence (to minima).

Newton’s Method:
It is a second-order iterative gradient based optimization technique known for its extremely fast convergence.
When close to optimum, it achieves quadratic convergence, better than gradient descent’s linear convergence.

Algorithm:

1. Start at a random point \(x_k\).
2. Compute the slope at \(x_k, ~i.e,~ f'(x_k)\).
3. Compute the curvature at \(x_k, ~i.e,~ f''(x_k)\).
4. Draw a parabola at \(x_k\) that locally approximates the function.
5. Jump directly to the minimum of that parabola; that’s the next step. Note: So, instead of walking down the slope step by step (gradient descent), we are jumping straight to the point where the curve bends downwards towards the bottom.