❌ Raw text is messy and inconsistent.

e.g., “WOw!!! The new iphone 18 pro is SOOO good! I luvv it… best phone ever? Check it out at someCoolSite.com #tech #apple”

So, first we need to do some pre-processing of this messy text, before it can be used for model training.
There are 2 main steps:

• Cleaning: Removing punctuation, lowercasing, stop word removal and stripping special characters.
• Stemming/Lemmatization: Reducing words to their root form.

Process of converting raw text into numerical vectors that machines can understand.

We will discuss the following ways to represent(embeddings) text as vectors:

• One Hot Encoding (OHE) (discrete)
• Bag of Words (BoW), TF-IDF (statistical)
• Word2Vec, GloVe, FastText (distributed)

Requirements of Good Text Representation(Embeddings)

• Capture meaning/similarity (semantics)
• Capture context
• Compact

Limitation of Word2Vec
Word2Vec only uses local context and is excellent at analogies and capturing semantics but ignores the vast amount of statistical information available in the global co-occurrence counts.
As, we saw earlier in the case of TF-IDF that uses the statistics of the entire corpus.

GloVe leverages global co-occurrence matrix, allowing it to better understand how words relate across the entire dataset, resulting in superior semantic analogies and better representation of relationships between distant words.

Research Paper: GloVe: Global Vectors for Word Representations, Pennington et al., Stanford University, 2014, https://nlp.stanford.edu/pubs/glove.pdf

Glove model leverages statistical information by training only on the nonzero elements in a word-word co-occurrence matrix, and uses matrix factorization to generate low-dimensional word representations.

\(X_{ij}\) : Number of times word \(c_j\) appears in the context of word \(w_i\)

Word-word matrix or word-context matrix \(X\) is factorized into 2 matrices, such that:

Note: We do an exponentiation because dot product can be negative but the word count is always positive.

We have to find the 2 matrices such that the difference between the dot product and the actual word co-occurrence count is minimum.
This can be formulated as an optimization problem, because we have to find some kind of optimum, in this case minimum value.

Optimization Problem

where, F is Frobenius Norm

Read more about Optimization
Read more about Frobenius Norm

Estimate word and context vectors by solving the optimization problem:

Note: In the paper they empirically found that the most suitable value of \(\alpha = 3/4\).

Let’s understand all the parameters.

• \(f(X_{ij})\): Weighting function.
  • If \(X_{ij} = 0\): \(f(x) = 0\); Ignore pairs that never co-occur.
  • If \(X_{ij}\) is small: \(f(x)\) is also small, but increases as count increases.
  • If \(X_{ij}\) is very large (for stop words): \(f(x)\) saturates (usually at 1); prevents common stop-words from over-influencing the vectors.
• Bias terms
  • \(b_i\): Captures the “inherent frequency” of the target word \(w_i\).
    • If word \(w_i\) is very common (like “the”), \(b_i\) will be large.
  • \(\tilde{b}_j\): Captures the “inherent frequency” of the context word \(c_j\).

At each step, the RNN maintains a “hidden state” that captures information about the words seen so far.

Recurrent Relation

\(h_{t-1}\): Hidden/latent vector that captures all the information seen till time instance ‘\(t-1\)’.

Note: Hidden state can theoretically capture information from the entire history of the sequence.

RNN Architecture

RNN Parameters

Note: Number of parameters is independent of input sequence length, because the same neural network is processing different words at different time stamps.

Sequence Information
See below how different words are input at different time stamps and the hidden state stores the sequence information seen till that time stamp.

Note: The input \(x_1, x_2, \dots, x_T \) are the vector representations of the words or word embeddings, earlier OHE was used, but after 2013, Word2Vec or GloVe etc. are mostly used.

Teacher Forcing
Say we are training the RNN model for text generation or language modeling, i.e, we predict the next word based on all the words seen in the past.

e.g. Let’s assume that we are using the following sentence to train the model:
“This is a cat.”
Now at time ’t’ the model sees the current word (\(w_t\)) “a” and tries to predict the “t+1” or next word.
It generates a probability distribution (softmax output) \(p_{t+1}\) of all the words in the vocabulary.
But the model has just started training, and it has no idea of sentences now, so it can predict any garbage value, which may be totally meaningless.
So, in order to train the model and learn the patterns from the training data instead of feeding the predicted word \(p_{t+1}\) as the next input i.e \(w_{t+1}\), we feed the actual word i.e “cat” as \(w_{t+1}\).

This is called Teacher Forcing.

Note: This stops the error (caused by predicting some garbage value during training) from being propagated through the network.

RNN Algorithm

• For every sentence ‘i’ in the corpus:
  • For every token ‘t’ in the sentence:
    • Compute probability distribution \(p_t\) on the vocabulary, given the previous ‘t-1’ words in the sentence.
    • Compute the loss (cross entropy) between \(p_t\) and \(y_t\) (desired OHE). \[ E_t(\theta) = - \sum_{j=1}^{|V|} y_{t,j} ~ \text{log}(p_{t,j}) \]
    • Compute the average overall training loss for the sentence ‘i’. \[ E_i(\theta) = \frac{1}{T} \sum_{t=1}^{T} E_t(\theta) \]
• Compute gradient information using back propagation through time and update model parameters.

LSTM solves vanishing gradient problem by introducing:

• Cell State (\(C_t\)): which acts as a “long-term” memory conveyor belt that runs parallel to the
• Hidden State (\(h_t\)): the “short-term” working memory (similar to RNN)

LSTM Memory Cell

LSTM Architecture

• Each gate in an LSTM functions as a single-layer feedforward neural network that lives inside the LSTM cell.
• Each gate has its own unique, learnable weight matrices (\(W\)) and bias vectors (\(b\)).

GRU is simplified LSTM with fewer gates, fewer parameters and simpler architecture, while retaining the ability to capture long-term dependencies, often leading to faster training times and reduced computational costs.

• Simpler Architecture:
  • GRU has two gates (reset, update) instead of three.
  • This reduces the number of weight matrices by roughly 25-33%, leading to faster training and lower memory consumption.
• Structural Simplicity:
  • By merging the “Cell State” (\(C_t\)) and “Hidden State” (\(h_t\)) into a single vector, it eliminates the redundant storage of information.
• Performance on Small Data:
  • Due to fewer parameters, GRUs are often less prone to overfitting than LSTMs when training on smaller datasets.

GRU Memory Cell

GRU Architecture

• Each gate in an GRU functions as a single-layer feedforward neural network that lives inside the GRU cell.
• Each gate has its own unique, learnable weight matrices (\(W\)) and bias vectors (\(b\)).

Decoder decides parts of the source sentence to pay attention to.
By letting the decoder have an attention mechanism, we relieve the encoder from the burden of having to encode all information in the source sentence into a fixed-length vector.
While decoding an output pay attention to only relevant inputs.

Decoder Context

Note: Now the decoder context is not fixed.
While predicting the next word, the decoder (instead of relying only on the final encoder hidden state) dynamically pays attention to only relevant context from encoder at that time step.

Why it matters?
This allows the decoder to “look back” at the entire input sequence, preventing the information loss that occurs in basic encoder-decoder models when handling long sentences.

Goal
To determine the context vector (relevant) at each time step of decoder.

Before that we need to understand the meaning of few terms, viz., “Query”, “Key” & “Value”.

Research Paper: Neural Machine Translation by Jointly Learning to Align and Translate ; Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, 2014, https://arxiv.org/pdf/1409.0473

Dot Product Attention

Note: Faster to compute dot product than additive attention.

Research Paper: Effective Approaches to Attention-based Neural Machine Translation, Luong et al., 2015, https://arxiv.org/pdf/1508.04025

Scaled Dot Product Attention

Research Paper: Attention Is All You Need, Vaswani et al. , 2017, https://arxiv.org/pdf/1706.03762

Read more about Dot Product

In 2017, Transformer architecture, proposed that “Self-Attention” can be used completely for machine translation instead of RNNs.

Using “Self-Attention” made the model highly parallelizable because the matrix calculations can be done on GPUs in parallel.

Transformer Architecture

Research Paper: Attention Is All You Need, Vaswani et al. , 2017, https://arxiv.org/pdf/1706.03762

Transformer replaced recurrence with attention, enabling faster training and superior performance in NLP and AI tasks.

Source: Attention is all you need, Vaswani et al., 2017; https://arxiv.org/pdf/1706.03762

Transformer Architecture
It was designed for machine translation, so it has 2 parts, viz., encoder and decoder.

Encoder
The encoder is composed of a stack of N = 6 identical layers.

• Each encoder layer has 2 sub-layers:
  • Multi-Head Self Attention
  • Fully connected Neural Network (Feed Forward)
• Residual connection around each of the sub-layers.
• Each sub-layer has layer normalization.

Decoder
The decoder is also composed of a stack of N = 6 identical layers.

• Each decoder layer has 3 sub-layers:
  • Masked Multi-Head Attention
  • Multi-Head Encoder Decoder (Cross) Attention
  • Fully connected Neural Network (Feed Forward)
• Residual connection around each of the sub-layers.
• Each sub-layer has layer normalization.

Unlike Batch Normalization, which normalizes across the mini-batch for each individual feature, Layer Normalization normalizes across all the features (dimensions) for a single training instance.

• By centering mean (\(\mu\)) around zero and scaling them to variance (\(\sigma^2\)) of one, it prevents exploding or vanishing gradients, enables faster convergence.
• The mean (\(\mu\)) and variance (\(\sigma^2\)) are calculated over all features(dimensions) for that specific token.

Reason: Transformers primarily use LayerNorm because it treats each token independently.
This is critical for natural language processing (NLP) where sentences in a batch often have different lengths.

Read more about Batch Normalization

RoPE positional encoding is used in Llama and Mistral LLMs.

Does not add a vector to the embedding, instead, it rotates the Query (\(Q\)) and Key (\(K\)) vectors in 2D planes.
After rotation, when we take the dot product, we automatically get the relative position of tokens.

where ‘\(m\)’ is the position

• Generative Pre-Trained Transformer (GPT); Decoder only
• Bidirectional Encoder Representations from Transformers (BERT); Encoder only; Sentiment analysis, Question answering, Search query etc.
• Bidirectional and Auto-Regressive Transformers (BART); Encoder-Decoder; Text summarization, Document denoising (reconstruct corrupted text) etc.
• Text-To-Text Transfer Transformer (T5); Encoder-Decoder; Translation, Text summarization, etc; uses task-specific prefix

GPT Architecture (Decoder Only)

GPT (2018) was the first Large Language Model (LLM) based on Transformer (decoder only).
Because, of the highly parallelizable architecture of Transformer, it was possible to scale and train the model on internet scale data, which gave us the magical powers of LLMs.

Train the model to recognize when to pause text generation and output a specific JSON format to invoke an external tool/function.
Say, we ask some information, like weather of a city, the LLM will not have that information and need to search the web or call some external API.

Note: Tool calling in LLMs is core enabler that transformed LLMs from passive chatbots into Agentic AI.

• Function Calling (specific API):
  • User Prompt: “What is the weather in Pune?”
  • Model Output:
    • { “tool_call”: { “name”: “get_weather”, “arguments”: {“location”: “Pune”, “unit”: “celsius”} } }
• Tool Calling:

  • Model decides to use a Code Interpreter or a Web Search tool that is not just a simple API function.

    • Tool 1 (Calculator): {“tool”: “calculator”, “input”: “sqrt(15625)"}
    • Tool 2(Web Search): {“tool”: “web_search”, “query”: “recent AI news”}
      • {“name”: “web_search”, “arguments”: {“query”: “top AI news 2026/site:hackernews.com OR site:arxiv.org OR site:ieee.org OR site:wired.com”}

Research Paper: Toolformer: Language Models Can Teach Themselves to Use Tools, Schick et al. , 2023, https://arxiv.org/pdf/2302.04761

“Polishing” phase that aligns an LLM’s raw intelligence with human values, safety, and helpfulness.

Performed using Direct Preference Optimization (DPO).

Key Use Cases of RLHF

• Instruction Following: Ensuring the model actually follows complex constraints (e.g., “Write this in 50 words only.”).
• Safety Guardrails: Teaching the model to decline requests for hate speech or dangerous instructions.
• Subjective Nuance: Aligning the “tone” of the model (e.g., making it sound professional vs. casual).
• Factuality: Encouraging the model to say “I don’t know” when it is uncertain, rather than hallucinating a confident but wrong answer.
• Coding Style: Ranking code that is more “pythonic” or efficient even if multiple versions are functionally correct.

Research Paper: InstructGPT: Training language models to follow instructions with human feedback, Ouyang et al., 2022, https://arxiv.org/pdf/2203.02155

In SFT or RLHF we are not teaching the model language from scratch but only teaching a pre-trained model to follow instructions.

Training LLMs is very costly, both wrt to time and resources.

Challenge
When we fine-tune an LLM, we do not just store the model’s weights.
We also have to store optimizer states, gradients, and activations.

7B parameters LLM

Solution
Instead of retraining the entire \(d_{model} \times d_{model}\) weight matrices, PEFT methods only update a tiny fraction (often \(<1\%\)) of the parameters while keeping the original “knowledge” of the pre-trained Transformer frozen.

Key PEFT methods include LoRA (most popular), QLoRA, Prefix Tuning, and Adapter layers.

Low-Rank Adaptation (LoRA)
It freezes the original model weights and injects two small, trainable low-rank matrices into the attention layers.
Reduces trainable parameters by up to 99% while maintaining near-identical performance to full fine-tuning.
Prevents “catastrophic forgetting”.

Low Rank Matrices
When we train a model, we are looking for a change in weights, which we call \(\Delta W\).

Note: ‘\(r\)’ is the rank, a small integer (e.g., 4, 8, or 16).

PEFT (LoRA) Working

• Initialization:
  • Matrix ‘\(B\)’ is initialized to 0.
  • Matrix ‘\(A\)’ is initialized with random Gaussian noise.
  • At the very start of training,\(BA = 0\), meaning the model starts with the exact same behavior as the original pre-trained model.
• Forward Pass:
  • Input ‘\(x\)’ is passed through both the frozen weights and the trainable “adapters” in parallel.
  • \(h = Wx + BAx\)
• Back Propagation:
  • Gradients are only calculated and updated for matrices ‘\(A\)’ and ‘\(B\)’.
  • Because ‘\(r\)’ is so small, we are often training less than 1% of the total parameters.
• Inference:
  • After training, we can mathematically “merge” the weights:\(W_{updated} = W + BA\).
  • This results in a single matrix of the original size, meaning the model runs just as fast as the original with zero overhead.

Example
\(W_q = d \times d\) and \(B = d \times r\), \(A = r \times d\)
Let, d = 1000, and r = 4.

• Full Matrix (\(W\)): \(1000 \times 1000 = 1,000,000\) parameters
• Matrix B: \(1000 \times 4 = 4,000\) parameters
• Matrix A: \(4 \times 1000 = 4,000\) parameters
• \(\frac{\text{LoRA Parameters}}{\text{Full Parameters}} = \frac{8,000}{1,000,000} = 0.008\)

Therefore, we train only 0.8% of the parameters for that layer.
Across the whole model, this usually results in training ~0.1% to 1%.

Limitations of Word Vectors
Before 2018, models like Word2Vec and GloVe provided “context-free” embeddings.
The word “bank” had the same vector whether it was a “river bank” or a “bank account”.

Google AI Language team developed, Transformer (encoder only) based language model designed to understand the meaning of words in a text by using the context from both directions.

BERT Architecture (Encoder Only)

BERT Input Representation

Note: BERT uses WordPiece embeddings (Wu et al., 2016) with a 30,000 token vocabulary.

Read more about WordPiece

Research Paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin et al., 2018; https://arxiv.org/pdf/1810.04805

Modification of the pre-trained BERT network that use Siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity.

✅ This reduces the effort for finding the most similar pair from 65 hours with BERT / RoBERTa to about 5 seconds with SBERT, while maintaining the accuracy from BERT.

SBERT Architecture

SBERT Pooling
SBERT adds a pooling operation to the output of BERT / RoBERTa to derive a fixed sized (768) sentence embedding.

3 pooling strategies:

• Computing the mean of all output vectors (MEAN-strategy); Default
• Computing a max-over-time of the output vectors (MAX-strategy)
• Using the output of the CLS-token

Research Paper: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Reimers & Gurevych, 2019; https://arxiv.org/pdf/1908.10084