NLP Study Notes (2): Neural Networks and Word Representations - Bojin Li

These are my notes from taking COMP90042 NLP at the University of Melbourne; this part covers neural networks and word representations: from FFNNs, RNNs and LSTMs to Transformers, then count-based word vectors, Word2Vec and evaluation methods.

1. Feedforward Neural Network (FFNN)

1.1 Fundamentals of Deep Learning

Deep learning is a branch of machine learning that mainly uses multilayer neural networks to learn data representations.
Why is it called deep? Because many layers are stacked together (multiple nonlinear transformations).
Inspired by how the human brain works, neural networks consist of many computational units called neurons.

Feed-forward NN

The fully connected feedforward network used in this section is also called a multilayer perceptron (MLP).
Information flows only forward: input layer → hidden layer → output layer, with no recurrent connections; sequences usually need to be arranged into fixed-length inputs in advance.
Each arrow carries a weight, indicating its importance.
Some layers have nonlinear activation functions (otherwise stacking multiple layers is equivalent to a single linear transformation).

Figure 1-1　A three-layer feedforward network: Input layer → Hidden layer → Output layer; each arrow is a weight.

A single neuron: a function $h = f(x)$

Given input $x$ , compute a scalar $h$
Use the weight $w$ to scale the input and add the bias $b$
Apply a nonlinear function (logistic sigmoid, tanh / hyperbolic tangent, ReLU)
$w$ and $b$ are the model's parameters

h = \tanh\!\left(\sum_j w_j x_j + b\right)

Activation function	Formula	Characteristics
Sigmoid	$\sigma(z)=\dfrac{1}{1+e^{-z}}$	Output in (0,1), can be treated as a probability; saturation at both ends causes vanishing gradients
Tanh	$\tanh(z)=\dfrac{e^z-e^{-z}}{e^z+e^{-z}}$	Output in (-1,1), zero-centered, easier to train than sigmoid
ReLU	$\max(0, z)$	Fast to compute, mitigates vanishing gradients; the negative half-axis “dies”

1.2 Output Layer

Binary classification

The output layer only needs one node;
Use sigmoid as the activation function, outputting $P(y=1\mid x)\in(0,1)$ .

Multi-class classification

For however many classes there are, the output layer has that many nodes;
Use softmax to convert node outputs into a probability distribution that sums to 1:

\text{softmax}(z_i) = \frac{\exp(z_i)}{\sum_j \exp(z_j)}

1.3 Training Methods

(a) Loss function

Objective: maximize the total probability of the training data $L$ :

L = \prod_{i=1}^{m} P(y_i \mid x_i)

Equivalently: minimize $-\log L$ (negative log-likelihood / cross-entropy) to optimize the parameters:

-\log L = -\sum_{i} \log P(y_i \mid x_i)

Reason for taking the log: it turns a product into a sum and reduces numerical underflow; after adding the negative sign, training can be done by minimizing the loss. This transformation does not make the neural network objective convex.

(b) Gradient descent

Purpose: find the weights that minimize the loss function.
Update rule: $\theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}$
TensorFlow / PyTorch use autodiff (automatic differentiation) to compute gradients, with no need to derive them by hand.
Variants: SGD → mini-batch → Momentum / RMSProp / Adam.

Because NNs have so many parameters, they are very prone to overfitting (low bias, high variance).
So regularisation is very important in NN training:

Method	Formula/approach	Effect
L1-norm	$\lambda\sum\vert \theta\vert$	Sparse weights (feature selection)
L2-norm	$\lambda\sum\theta^2$	Constrains weight magnitude; under ordinary SGD it corresponds to weight decay
Dropout	Randomly set some neuron outputs to 0 during training	Reduces neuron co-adaptation; can be understood as approximately combining many subnetworks that share parameters

(d) Dropout

If dropout rate = 0.1, then each output to which dropout is applied has a 10% probability of becoming 0;
It can be used on any layer, but in practice is mainly used on hidden layers;
Dropout is disabled at inference. The commonly used inverted dropout divides retained outputs by the retention probability during training, so no extra scaling is needed at inference.

1.4 Application 1: Topic Classification

Task: given a document, classify it into a predefined set of topics (economy / politics / sport…). Input: bag-of-words.

Training

Randomly initialize $W$ and $b$
Forward computation:

\vec{x} = [0,2,3,0] \;\rightarrow\; \vec{h} = \tanh(W_2\vec{x} + \vec{b}_1) \;\rightarrow\; \vec{y} = \text{softmax}(W_3\vec{h})

Output probability distribution $\vec{y} = [0.1, 0.6, 0.3]$ , corresponding to $C_1, C_2, C_3$
If the true label is $C_1$ : $L = -\log(0.1) \approx 2.303$

Prediction

Test input $\vec{x} = [1,3,5,0]$
Forward pass gives $\vec{y} = [0.2, 0.1, 0.7]$
Predicted class $= C_3$ (take argmax)

Directions for improvement

Add bag of bigrams as input (to capture local word order)
Preprocessing: lemmatise + remove stop words
Instead of raw counts, use TF-IDF or a 0/1 indicator (only record whether it appears)

1.5 Application 2: Word Embedding

Definition: map discrete word symbols to a low-dimensional continuous vector space.
Purpose: enable the model to capture similarity between words (dog vs. cat, walking vs. running).
In an FFNN, an embedding is essentially a learnable lookup table; each column of the matrix $W_1$ is the vector of one word.

1.6 Application 3: FFNN Language Model

Predict $P(w_i = \text{grass} \mid w_{i-3}=\text{a},\, w_{i-2}=\text{cow},\, w_{i-1}=\text{eats})$

Procedure:

For a, cow, eats look up word embeddings ( $W_1$ )
Take the three vectors and concatenate them together.
Feed into the network: $\vec{h} = \tanh(W_2\vec{x} + \vec{b}_1)$ , $\vec{y} = \text{softmax}(W_3\vec{h})$
$\vec{y}$ gives a probability distribution over the entire vocabulary

Parameter distribution:

Most parameters are in word embeddings $W_1$ (size $= d_1 \times \vert V\vert$ )
and in output embeddings $W_3$ (size $= \vert V\vert \times d_h$ , where $d_h$ is the hidden-layer dimension)

1.7 FFNN Language Model vs Count-based N-gram

Dimension	Count-based N-gram	FFNN N-gram
Training cost	Extremely low (only requires counting occurrences)	Relatively high (requires gradient descent)
Sparsity	Severe (unseen n-grams have probability 0)	Mitigated (similar words share parameters, allowing generalization to unseen combinations)
Long context	As context grows longer, parameter/sparsity problems worsen	Can scale smoothly (but still limited by a fixed window)
Word properties	Cannot capture syntactic/semantic similarity between words (film vs movie are treated as two unrelated symbols)	Automatically captures word properties, yielding more robust estimates

2. Recurrent Neural Networks (RNNs)

2.1 Definition

Allows representing arbitrary-length inputs;
Core idea: through a recurrence formula, process one input in the sequence at a time;
Use a state vector to represent the previously processed context.

s_i = f(s_{i-1}, x_i), \qquad s_{i+1} = f(s_i, x_{i+1})

The same recurring function $f$ is reused at all time steps.

2.2 Node

Core idea: input only one word at a time (not the whole sentence), and combine it with the previous state to generate a new state.
The input to each node is the word's embedding, and the state carries position/order information.
Input sequence $X$ has variable length, but the dimension of each input vector $x_i$ and that of the state vector $s_i$ are still fixed by the model.

Two formulas of a Simple RNN

s_i = \tanh(W_s s_{i-1} + W_x x_i + b)

y_i = \text{softmax}(W_y s_i)

$W_s$ : state-to-state weights (how memory is passed)
$W_x$ : input-to-state weights
$W_y$ : state-to-output weights
$b$ : bias

Figure 2-1　A single RNN node: the new state $s_i$ is jointly determined by the previous state $s_{i-1}$ and the current input $x_i$ .

2.3 Network

All time steps share the same set of parameters ( $W_s, W_x, b, W_y$ ), and positions are connected through the state, so the number of parameters does not grow directly with sequence length.

Figure 2-2　RNN unrolled over time: all time steps share the same set of parameters $W_s, W_x, W_y, b$ .

2.4 Training

Input only one $x$ at a time (one word in an LM);
At different time steps, the same word produces different outputs $y$ (because the state is different);
Loss is the sum of losses over all positions:

L = L(y_1) + L(y_2) + L(y_3) + \dots = \sum_t L(y_t)

Use BPTT (Back Propagation Through Time) to backpropagate along the unrolled computation graph, updating the shared weights, biases and trainable embeddings.

2.5 Training Example (Language Model)

Sentence: a cow eats grass

$x_i$ is the current word (e.g. eats), first mapped to an embedding
$s_{i-1}$ contains information from the previous context (a, cow)
$y_i$ predicts the next word (grass)

The network is unrolled over time:

TEXT

a      cow     eats     grass
↓       ↓       ↓        ↓
RNN →  RNN  →  RNN   →  RNN
s1      s2      s3       s4
                ↓
            y3 = P(·|a cow eats) → target grass

Numerical example (vocabulary {a, cow, eats, grass})

Position	Input	State $s$	Output distribution $y$	Target	Loss
1	a	$s_1$	[0.40, 0.30, 0.20, 0.10]	cow	$-\log(0.30)$
2	cow	$s_2$	[0.30, 0.20, 0.30, 0.20]	eats	$-\log(0.30)$
3	eats	$s_3$	[0.50, 0.10, 0.20, 0.20]	grass	$-\log(0.20)$

L_{\text{total}} = L_1 + L_2 + L_3

2.6 Problems with RNNs

Problem	Description
a) Mismatch between training and decoding	During training, teacher forcing is used (feeding the true previous word); at inference, the model is fed its own predictions, causing a distribution mismatch (exposure bias).
b) Error propagation and difficulty locating intermediate errors	Errors propagate and accumulate along time steps, and it is difficult to locate which step went wrong.
c) Tends to generate "blank" or "generic" language	Tends to generate vague, safe high-frequency expressions lacking specific information.
d) Vanishing gradient	Gradients decay exponentially when backpropagated along time, so long-distance dependencies cannot be learned.

2.7 Vanishing Gradient

Cause: after unrolling over time, backpropagation requires multiplying the Jacobians of each step, which include shared weights and activation-function derivatives. When the effective gain of repeated multiplication is small, gradients decay; when it is too large, they may also explode;
Consequence: training signals from distant positions are hard to propagate back, making long-distance dependencies harder to learn; the effective memory length depends on the model, parameters and task and cannot be fixed at a certain number of steps;
Remedies: better initialization, gated structures (LSTM/GRU), residual connections and attention mechanisms. Gradient clipping is mainly used to control gradient explosion, not to restore gradients that have already vanished.

3. Long Short-Term Memory Network (LSTM)

3.1 Definition and Motivation

A vanilla RNN passes the old state together with the new input through a nonlinear transformation at every step, so previous information may be gradually lost;
LSTM additionally maintains a memory cell, and through gated additive updates it makes information and gradients easier to propagate along time.

Memory is controlled by gates:

How much new input should be written into the memory cell (input gate)
How much of the current memory should be forgotten (forget gate)
At output time, how much should be read out (output gate)

The essence of a gate: a vector with values between 0 and 1 (generally generated using sigmoid), which is multiplied element-wise with the vector being controlled, thereby deciding “how much information to retain.” 0 = completely blocked, 1 = completely passed.

3.2 RNN vs LSTM Structure

RNN: the repeating module has only a simple tanh layer (single-layer structure).
LSTM: the repeating module has four interacting layers (3 sigmoid gates + 1 tanh), plus an additional cell state channel running through it $C_t$ .

Figure 3-1　The repeating module of a vanilla RNN has only one tanh layer, and the state is updated in an overwrite fashion.

Figure 3-2　The LSTM repeating module contains four interacting layers (3 sigmoid gates + 1 tanh), plus an additional cell state channel running through it $C_t$ .

3.3 Understanding and Controlling Memory

How to understand memory: $C_t$ provides an additive update path through time steps. When the forget gate is close to 1, information and gradients are easier to retain; long-term memory still depends on the gate values and training results.

How to control memory: four steps

Step	Role
Forget	Decides how much of the old cell state $C_{t-1}$ to discard.
Input	Decides how much new information to write
Update	Updates the cell state using the results of forget and input
Output	Decides how much to extract from the cell state as the current hidden state $h_t$

3.4 Three Gates and State Update Formulas

Forget gate

f_t = \sigma\left(W_f\,[h_{t-1}, x_t] + b_f\right)

$f_t$ controls how much information in the old memory cell $C_{t-1}$ is retained: discard when close to 0, retain when close to 1.
Example: The cats that the boy likes ...
- When predicting likes in the relative clause, the model needs to remember that the current subject boy is singular;
- Later, when predicting the main-clause verb, it may again need the plural information of cats;
- So this cannot be simply understood as “forget all about cats upon seeing boy”: memories in different dimensions can retain different information separately.

Input gate

i_t = \sigma\left(W_i\,[h_{t-1}, x_t] + b_i\right)

\tilde{C}_t = \tanh\left(W_C\,[h_{t-1}, x_t] + b_C\right)

$i_t$ : controls how much new information is written into the memory cell;
$\tilde{C}_t$ : new candidate information (distilled information), e.g. information about boy.

Update the cell state

C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t

Use the forget gate to "discount" the old memory, then add the new memory "admitted" by the input gate.
Key point: this is addition rather than overwriting, so gradients can flow directly back along $C_t$ → mitigating vanishing gradients.

Output gate

o_t = \sigma\left(W_o\,[h_{t-1}, x_t] + b_o\right)

h_t = o_t \odot \tanh(C_t)

Controls how much content is “distilled” from the cell state to form the current time step's hidden state $h_t$ .

$[h_{t-1}, x_t]$ means concatenating the previous hidden state and the current input into one vector.

Figure 3-3　Inside the memory cell: the three arrows correspond to the direct cell state path, the previous hidden state $h_{t-1}$ , and the current input $x_t$ ; $\sigma$ and $\tanh$ are connected in parallel to form the gates.

3.5 Number of Parameters

Let

$\text{Hidden}_{dim}$ = dimension of memory (also the dimension of $h_t$ )
$\text{Input}_{dim}$ = dimension of word embedding

\#\text{params} = 4 \times (\text{Hidden}_{dim} + \text{Input}_{dim}) \times \text{Hidden}_{dim} + 4 \times \text{Hidden}_{dim}

Why 4? Because there are 4 weight matrices: $W_f, W_i, W_C, W_o$ , each with the same dimensions (rows, columns).
With the column-vector convention, the number of rows of each weight matrix is the output dimension $\text{Hidden}_{dim}$ ;
the number of columns is the concatenated input length $\text{Hidden}_{dim} + \text{Input}_{dim}$ ;
Finally, $4 \times \text{Hidden}_{dim}$ accounts for the 4 sets of bias constants.

Example: Hidden = 256, Input = 300

4 \times (256+300)\times 256 + 4\times256 = 569{,}344 + 1{,}024 = 570{,}368

3.6 Applications of RNN / LSTM

Application	Description
Wikipedia Generator	Generates encyclopedic text from a consistently structured training corpus
Code Generator	Generates code (highly structured and easy to evaluate)
Speech recognition	Maps acoustic feature sequences to text
Text Classification	Use the final hidden state or pooling as the document representation
Sequence Labelling	POS, NER (output one label per position)

3.7 Variants

Variant	Key points
Peephole connections	Allow gates to "see" the cell state ( $C_{t-1}$ also participates in gate computation)
GRU (Gated Recurrent Unit)	Simplified version: only 2 gates (reset / update), no separate memory cell, fewer parameters, faster training, often comparable to LSTM
Multi-layer LSTM	Stack multiple layers for layer-by-layer abstraction (but in practice it is not easy to stack too deep)
Bidirectional LSTM	Forward + backward LSTMs, concatenating hidden states from both directions → each position has both left and right context (cannot be used for autoregressive generation)

3.8 Pros and Cons

Pros

Able to capture long-distance context (compared with a vanilla RNN);
Like FFNNs, LSTMs are flexible: can be used for classification, sequence labelling, generation and many other tasks.

Cons

Due to sequential processing, LSTMs are slower than feedforward networks (cannot parallelize over the time dimension);
In practice, not very good at capturing very long dependencies (especially when generating long texts);
In practice, difficult to stack (multi-layer LSTM has limited gains and unstable training);
With the emergence of more advanced architectures (Transformer), they are now less popular.

4. Transformer

4.1 Definition

Instead of passing a recurrent state position by position as RNNs do, it uses attention to directly establish connections between positions;
When the entire input is known, representations at each position in the same layer can be computed in parallel; when training an autoregressive language model, a causal mask can also be used to compute losses at multiple positions simultaneously;
Parallel computation does not mean positions are independent of each other. During autoregressive generation, the next token still depends on the preceding generated text, so decoding must proceed step by step;
This structure shortens the information path between distant positions and is also better suited to matrix operations on GPUs.

4.2 Motivation: From RNN to Attention

For problems of the form "input text → output text", the original approach used an RNN to sequentially process the input and current output;
Attention was originally intended to solve the RNN's information bottleneck problem: an RNN must compress the whole sentence into a fixed-length vector. Attention allows each output node to directly compare with all input nodes, selecting the most relevant input nodes to determine its output;
Since Attention can directly access all inputs, a sequential RNN is no longer required;
Thus Transformer was born: its Encoder uses Self-Attention instead of RNN (paper Attention Is All You Need, 2017).

Figure 4-1　Overall Transformer architecture (Encoder on the left, Decoder on the right, each stacked $N$ layers; at the bottom are Input/Output Embedding + Positional Encoding). Source: Vaswani et al., Attention Is All You Need, NeurIPS 2017, Figure 1.

4.3 Use Cases

Any task of the form "input text → output text":

Language Model: predict the next word
Dialog System: given user input, generate an appropriate reply
Machine Translation: translate from one language to another

Essentially all are "a piece of input text → another piece of output text", which Transformer is very well suited to.

4.4 Self-Attention: Query / Key / Value

Intuitive analogy:

Use the target word's Query to compare relevance with all words' Keys, then use these scores to weight the corresponding Values.

Example: sentence I made her duck, target word her

Role	Content
Query $q$	The query vector of the target word her
Key $k$ / Value $v$	Context words (I, made, duck) and the target word's own key/value

Figure 4-2　Self-attention weight allocation: after comparing the Query of the target word her with the Keys of each word, a weighted sum of the Value vectors gives the new representation of her.

Steps

Generate Q/K/V: In this section, each word's embedding $x_i$ is written as a row vector and multiplied by three projection matrices $W_Q, W_K, W_V$ :

q_i = x_i W_Q,\qquad k_i = x_i W_K,\qquad v_i = x_i W_V

Compute relevance: the target word's $q$ is compared with each word's $k$ using a dot product to obtain relevance scores;
Normalize: apply softmax to the scores to obtain a probability distribution (weights sum to 1);
Weighted sum: use this distribution to compute a weighted sum over all $v$ to obtain the target word's output vector $z$ ;
Repeat the above process for all input words to obtain each $z$ , arranged by position to form the output of one attention head.

Core conclusion: The final representation of the target word = the weighted sum of the value vectors of the context words (including itself), with weights determined by query–key similarity.

Figure 4-3　Left: Scaled Dot-Product Attention (MatMul → Scale → Mask → Softmax → MatMul); right: Multi-Head Attention (multiple sets of linear projections computed in parallel, then concatenated and linearly transformed). Source: Vaswani et al., 2017, Figure 2.

4.5 Formulas

Single-query form

\text{SelfAttention}(q, K, V) = \sum_i \frac{\exp(q \cdot k_i)}{\sum_j \exp(q \cdot k_j)}\, v_i

Matrix form (stack the $q_i$ , $k_i$ , $v_i$ by row into $Q$ , $K$ , $V$ )

\text{SelfAttention}(Q, K, V) = \text{softmax}(QK^\top)\,V

Scaled Dot-Product Attention (used in practice)

\text{SelfAttention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Why divide by $\sqrt{d_k}$ ? $d_k$ is the dimension of the key vector. The variance of the dot product grows with dimension; when the dimension is large, softmax inputs become too large → an extreme distribution (close to one-hot) → extremely small gradients. Dividing by $\sqrt{d_k}$ scales the scores to stabilize the values (avoiding overly large values/vanishing gradients).

Figure 4-4　Computing Q/K/V word by word: each embedding $x_i$ is multiplied by $W_Q, W_K, W_V$ , then the target word's $q$ is compared via dot products with all $k$ vectors, and softmax is applied to obtain weights.

4.6 Multi-Head Attention

Motivation: different words in a sentence can be related in multiple ways:

Relation type	Example
Syntactic	subject–verb agreement (noun–verb agreement)
Semantic	synonyms
Discourse	it refers to the earlier the dog

Multi-head attention allows the model to attend to different relations in parallel in different representation subspaces, therefore:

Independently perform the attention computation $h$ times (different heads have their own $W_Q^{(i)}, W_K^{(i)}, W_V^{(i)}$ );
Take each head’s output $z^{(i)}$ and concatenate them together;
Then multiply by the output projection matrix $W_O$ , mapping the concatenated result back to the model dimension $d$ , fusing multi-head information;
This output is passed as input to the next layer.

\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W_O

\text{head}_i = \text{Attention}(QW_Q^{(i)}, KW_K^{(i)}, VW_V^{(i)})

Multiple Transformer layers repeat these modules. The original Transformer adds positional encoding once to the input embeddings; other positional encoding schemes may inject it differently.

4.7 Transformer Block

Taking the original Transformer's Encoder block as an example, besides self-attention it also contains:

Component	Count	Role
Feedforward layer	1×	A position-wise two-layer MLP (first increase dimension, then reduce it), providing nonlinear transformation capability
Residual connection	2×	Adds the input and the sublayer output ( $x + f(x)$ ), preserving a direct propagation path for information and gradients, making deep networks easier to train
Normalisation layer (LayerNorm)	2×	Normalizes the hidden features at each position and learns scaling and bias, helping stabilize training

Note: this uses Layer Normalization, normalizing over each position's own feature dimension and not depending on statistics from other samples in a batch. The figure below uses the original Transformer's Post-LN order; other variants may also use Pre-LN.

Figure 4-5　Inside a Transformer Block: the two sublayers Self-Attention and Feed Forward, each wrapped with a residual connection + LayerNorm.

4.8 Stack Them Up

Stack more Transformer modules → learn deeper word representations;
GPT-3 (175B) has 96 Transformer modules.

4.9 Positional Information

Problem: without adding position information, self-attention itself cannot distinguish word order. Applying the same permutation to the input only permutes the output accordingly; it does not magically know which word comes first and which comes later. Therefore, additional position representations are needed to distinguish dog bites man from man bites dog.

Learnable Positional Embeddings

Input embedding $X$ = word embeddings + position embeddings (the two are added);
Randomly initialize position vectors and let the model learn them;
Approach: one embedding for position 1, one for position 2, … up to some maximum length;
Limitation: the position lookup table has a preset length, and positions not trained on lack reliable representations; handling longer sequences usually requires additional extension methods.

Static Functions (sinusoidal positional encoding)

Use a static function to map each position to a fixed vector without learning:

PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE_{(pos, 2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)

Advantages: any position can be computed by formula, relative displacements have regular patterns, and no trainable parameters are added; however, “can compute longer positions” does not mean the model will necessarily generalize reliably to longer sequences.
Modern variants: RoPE (Rotary Position Embedding), ALiBi (LLaMA uses RoPE).

4.10 Decoder-only Architecture

Autoregressive language models use next-token prediction to model text, taking the existing preceding text as input;
When there is no separate source-sequence encoder, cross-attention between Encoder and Decoder is not needed either;
Only modules such as self-attention with causal mask and FFN are retained and stacked into a Decoder-only architecture; the GPT series follows this idea. Encoder and Encoder–Decoder structures still suit their respective understanding or sequence-transformation tasks.

5. Word Vectors: Count-based Methods

5.1 Overview of Word Vector Embedding

Encodes a word's contextual information;
Can capture various semantic relations (synonymy, analogy, etc.);
One source: the embedding layer in an FFNN can be trained along the way;
This section compares two major categories of methods:
1. Count-based methods (this section)
2. More efficient neural methods, designed specifically for learning word vectors (Section 6)

5.2 Documents as Context: Vector Space Model (VSM)

Core idea: represent word meaning as a vector;
Treat documents as the context of words;
One matrix, two perspectives:

Perspective	Row/column	Meaning
Document perspective	Each row is a document	A document is represented by the words it contains
Word perspective	Each column is a word	A word is represented by the documents in which it appears (the column vector is the word vector)

Word–document matrix example (rows = document id, columns = words)

	state	fun
425	0	1
426	3	0
427	0	0

Problem: using raw counts directly, high-frequency words (the, of) dominate the vectors; moreover, the vector dimension equals the number of documents, making vectors very sparse and high-dimensional.

5.3 TF-IDF

A standard weighting scheme in information retrieval;
Core role: discounts common words.

Term Frequency

In the matrix, each column represents a word, and each row represents a document;
The simplest TF is the count $tf_{t,d}$ (one may also take $\log(1+tf)$ or normalize).

Inverse Document Frequency

idf_w = \log \frac{\vert D\vert }{df_w}

$\vert D\vert$ = total number of documents
$df_w$ = number of documents containing word $w$ .
Meaning: the number of different documents containing the same word is used to penalize its TF weight—the more documents it appears in, the lower its discriminative power and the smaller its weight.

Combination

\text{TF-IDF} = TF \times IDF

This uses the standard version; in practice there are also smoothed variants (e.g. $\log\frac{1+\vert D\vert }{1+df_w}+1$ );
Implementation: sklearn.feature_extraction.text.TfidfVectorizer().

Example (tf matrix → tf-idf matrix)

tf matrix (rows 425/426/427, columns the/country/hell):

	the	country	hell
425	43	5	1
426	24	1	0
427	37	0	3
$df$	500	14	7

Shown here are three document rows from the full corpus; let the total number of documents be $\vert D\vert=500$ , use base-2 logarithms, and approximate the IDFs of country, hell as 5.2 and 6.2 respectively before multiplying.

→ the appears in all 500 documents, so $idf=0$ ; while hell appears in only 7, so its weight is relatively higher.

	country	hell
425	26.0	6.2
426	5.2	0
427	0	18.6

5.4 Dimensionality Reduction

SVD (singular value decomposition)

A = U \Sigma V^\top

To obtain word vectors by rows, here the previous document–word matrix is transposed; $A$ is the $\vert V\vert \times \vert D\vert$ word–document matrix
$U$ : $\vert V\vert \times m$ , new word matrix (each row is the representation of one word)
$\Sigma$ : $m \times m$ singular-value diagonal matrix (in descending order of importance)
$V^\top$ : $m \times \vert D\vert$ , the new document matrix
$m = \text{Rank}(A)$

When all nonzero singular values are retained, this is an equivalent decomposition; in practice, dimensionality reduction and denoising usually rely on the truncated version below.

Latent Semantic Analysis (LSA)

Keep only the largest $k$ singular values and their corresponding directions, and use the rows of $U_k$ or $U_k\Sigma_k$ as word vectors;
This is lossy compression: the dimensions corresponding to small singular values are discarded (treated as noise/details);
Basis: many tasks only need a small number of relations between words ( $k$ is often 100–300);
By-product: it can capture latent semantics of “same meaning but different words” (because their document co-occurrence patterns are similar).

Complete TF-IDF word-vector pipeline

TEXT

tf-idf matrix  →  SVD  →  U  →  pick k  →  keep the first k dimensions  →  word embedding  →  LSTM (or a downstream model)

Figure 5-1　Apply singular value decomposition to the TF-IDF matrix $A=U\Sigma V^\top$ , take $U$ 's first $k$ columns to obtain $k$ -dimensional word vectors (LSA).

5.5 Word as Context

Idea: according to relations between words, use co-occurrence frequency as context to obtain word meaning;
"How close exactly counts as context" needs to be defined in advance: common window sizes (e.g. ±2, ±5), whether to split by sentence, whether to weight (closer words get higher weights);
The common words problem also exists (the, of co-occur with all words) → correct it with PMI.

5.6 Pointwise Mutual Information (PMI)

\text{PMI}(x, y) = \log_2 \frac{P(x, y)}{P(x)\,P(y)}

Meaning: the ratio of the actual co-occurrence probability to "the co-occurrence probability if the two words were independent".

$\text{PMI} > 0$ : co-occurrence is more than chance, positive correlation (related words)
$\text{PMI} = 0$ : the co-occurrence probability of this word pair equals the probability under the independence assumption
$\text{PMI} < 0$ : co-occurrence is less than chance, negative correlation (complementary/repulsive)

Example: compute PMI(heaven, hell)

Co-occurrence matrix (window co-occurrence counts, excerpt)

	the	country	hell	Σ
state	1973	10	1	12786
fun	54	2	0	633
heaven	55	1	3	627
Σ	1047519	3817	780	15871304

$P(\text{heaven}) = 627 / 15871304$
$P(\text{hell}) = 780 / 15871304$
$P(\text{heaven}, \text{hell}) = 3 / 15871304$ (if the count for this cell is 3)

Substitute back into the formula:

\text{PMI}(\text{heaven},\text{hell})=\log_2\frac{3\times15871304}{627\times780}\approx6.61

Here heaven and hell have a count of 3, so PMI is positive. In the table below, -inf corresponds to fun and hell, whose co-occurrence count is 0: when the marginal probabilities are nonzero, $P(x,y)=0 \Rightarrow \log 0=-\infty$ .

PMI matrix (excerpt)

	the	country	hell
state	1.22	1.78	0.63
fun	0.37	3.79	-inf
heaven	0.41	2.80	6.61

5.7 Three Additional PMI Techniques

Technique	Approach	Reason
Replace $-\infty$ or negative values with 0	$\text{PPMI} = \max(\text{PMI}, 0)$	Retains co-occurrence relations above the independence assumption and reduces the impact of unreliable negative-association estimates for low-frequency pairs
Normalised PMI (NPMI)	$\text{NPMI}(x,y) = \dfrac{\text{PMI}(x,y)}{-\log_2 P(x,y)}$	The numerator and denominator use the same logarithm base; for $0<P(x,y)<1$ , normalize to $[-1,1]$ , making comparisons across word pairs easier
PMI can also be simplified with SVD	Apply SVD to the PMI matrix and take the first $k$ dimensions	Reduce dimensionality and denoise, obtaining dense low-dimensional word vectors

Note: PMI for low-frequency word pairs tends to be inflated, so association strength cannot be judged by the score alone. Minimum co-occurrence frequency filtering or smoothing can reduce noise. If using $\log\frac{P(x,y)^k}{P(x)P(y)}$ as a PMI $^k$ variant, only when $k>1$ does it suppress word pairs with low co-occurrence probabilities more strongly than ordinary PMI.

6. Word Vectors: Neural Word2Vec

6.1 Word2Vec Overview

Neural distributional semantics: using neural networks to implement “distributional semantics” (word meaning is determined by context).
Core idea: use context words to predict the target word (or vice versa).
Learn embeddings through a word-prediction training task: CBOW's input is the context and Skip-gram's input is the center word; the output is a distribution over the words being predicted or a word-pair classification result, and word vectors are the parameters learned during training.
Two implementations (both from Mikolov et al., 2013):

Model	Direction	Description
Skip-gram	Use the target word to predict surrounding words	In practice, better at low-frequency words
CBOW (Continuous Bag of Words)	Use surrounding words to predict the target word	Faster, and better at high-frequency words

The range of “surrounding words” (window $L$ ) depends on the situation; the example below uses $L=2$ .

Figure 6-1　Two Word2Vec structures: CBOW (uses context to predict the center word) and Skip-gram (uses the center word to predict context). Source: Mikolov et al., 2013, Figure 1.

6.2 Skip-gram Model

Objective: maximize the probability that the correct words appear in the context given the target word.

Sentence example: ... Bereft of life he rests in peace! If you hadn't nailed him ..., target word rests, $L=2$ , the context is {he, in, life, peace}.

Probabilistic modeling: the probability of context words = the joint probability conditioned on the target word:

P(\text{context} \mid w_t) = \prod_{i \in [-L, \dots, -1, 1, \dots, L]} P(w_{t+i} \mid w_t)

Assumption: given the center word, context words are treated as conditionally independent, so their probabilities can be multiplied; this basic version does not distinguish the order of context words.

6.3 Implementation with Logistic Regression (Softmax Form)

P(w_O \mid w_I) = \frac{\exp(v'_{w_O}{}^\top v_{w_I})}{\sum_{w \in V} \exp(v'_{w}{}^\top v_{w_I})}

Here, $v$ is the input (center word) vector and $v'$ is the output (context word) vector.

It can be transformed into a neural network:

TEXT

one-hot x (1×|V|)  →  W (|V|×d)  →  h (1×d)  →  C (d×|V|)  →  softmax  →  P(·|w_t)
                    input embedding          output embedding

6.4 Embedding Parameterization: Two Matrices

Key detail: Word2Vec has two sets of word-vector matrices:

$W$ ( $\vert V\vert \times d$ ): target word embedding (center word/input embedding)
$C$ ( $d \times \vert V\vert$ ): context word embedding (context/output embedding)

After training, usually only $W$ (or the average of both) is kept as the final word vector.

How to choose $d$ ? Treat the dimension as a hyperparameter and tune it according to corpus size, downstream tasks and computational budget; 100–300 dimensions is a common starting point.

Training objective: given any input word, obtain “the probability that each word is its context word,” and hope that the correct context word has the highest probability.

Problem: too slow—the denominator must sum over the entire vocabulary ( $\vert V\vert$ can reach millions), and every training sample has to traverse all words.

6.5 Optimization: Negative Sampling

Transform “predicting $P(\text{context} \mid \text{word})$ ” into a binary classification problem:

Use center–context word pairs actually observed in the corpus as positive samples;
Draw words from a noise distribution and pair them with the center word to form negative samples;
Train a binary classifier to distinguish true co-occurring word pairs from sampled word pairs.

Why is negative sampling needed?

The core purpose is to avoid computing a full-vocabulary softmax for every sample;
Each positive sample is paired with only a small number of negative samples; the numbers of positive and negative samples need not be equal;
It changes the training objective to word-pair binary classification, no longer directly computing a normalized next-word probability.

Maximum likelihood estimation

\max \ \sum_{(w,c)\in D} \log \sigma(u_c^\top v_w) + \sum_{(w,c)\in D'} \log \sigma(-u_c^\top v_w)

where $D$ , $D'$ are the positive and negative word-pair sets respectively, $v_w$ is the input vector, and $u_c$ is the output vector.

For positive samples, maximize $P(+\mid \text{word},\text{context})$ ;
For negative samples, maximize $P(-\mid \text{word},\text{context})$ , i.e. suppress the probability that it is judged as a true co-occurrence.

Intuition for embeddings: input words and output context vectors that often co-occur tend to have larger dot products; words with similar contexts may learn similar representations. The dot product is also affected by vector length and cannot be directly equated with cosine similarity.

6.6 Loss function

In practice, take $k$ negative samples for each positive sample (commonly $k=5\sim20$ , adjusted according to the corpus). Below, $T$ denotes the number of positive sample word pairs, $c_t$ is the center word, and $o_t$ is the observed context word:

J(\theta) = -\frac{1}{T}\sum_{t=1}^{T}\left[\log \sigma(u_{o_t}^\top v_{c_t}) + \sum_{i=1}^{k} \mathbb{E}_{w_i \sim P_n(w)}\log \sigma(-u_{w_i}^\top v_{c_t})\right]

Negative samples are drawn from a unigram distribution $P_n(w)$ (usually proportional to word frequency raised to the 3/4 power), preventing high-frequency words from dominating while avoiding too many rare words.

Advantages:

Unsupervised: no manual annotation is needed; any corpus can be used;
Efficient: negative sampling makes the computation per step independent of vocabulary size, allowing scaling to billion-level corpora.

7. Word Vector Evaluation

7.1 Intrinsic Evaluation 1: Word Similarity

Question: how do we know whether a word's embedding is good?

Method: use cosine similarity between the vectors of two semantically similar words (e.g. well & good), then compare these scores with human similarity ratings using correlation (Spearman correlation).

\cos(u, v) = \frac{u \cdot v}{\Vert u\Vert \,\Vert v\Vert }

Why use cosine instead of Euclidean distance? It considers direction rather than length, avoiding the bias caused by “high-frequency word vectors having larger norms.”

7.2 Intrinsic Evaluation 2: Word Analogy

Method: perform vector arithmetic in the embedding space and see whether the nearest neighbors conform to the semantic relation.

\text{king} - \text{man} + \text{woman} \approx \text{queen}

Vector addition can also illustrate semantic composition, but this is not a standard analogy evaluation:

\text{breakfast} + \text{lunch} + \text{dinner} \approx \text{a direction associated with meal} \quad\text{(an intuition for semantic composition, not an identity)}

Common dataset: Google analogy set (including semantic and syntactic categories).

7.3 Extrinsic Evaluation: Downstream Tasks

Method: directly apply the embeddings to real tasks (classification, NER, translation…), and use task performance to measure embedding quality in turn.

Advantages: truly reflects “whether it is useful”;
Disadvantages: slow, heavily affected by the downstream model (conclusions may change if the classifier is changed), and cannot locate which dimension of the embedding went wrong.

Practical advice: intrinsic evaluation is fast and cheap, suitable for rapid iteration; final conclusions must be based on downstream tasks.