NLP Study Notes (3): Pretrained Models and Large Language Models - Bojin Li

These are my notes from COMP90042, the NLP course I took at the University of Melbourne. This installment covers pretrained models and large language models, including GPT, BERT, T5, RLHF, prompting, instruction tuning, PEFT and LoRA.

1. Pretrained models: GPT / BERT / T5

1.1 What is a pretrained model?

In an NLP model, the first layer can be initialized with pretrained word embeddings rather than random weights;
Across many NLP tasks, this provides a simple, readily available performance improvement;
Even with pretrained word embeddings, however, most of the network's other weights are still randomly initialized. This is the problem that pretraining the whole model, rather than only word vectors, addresses.

The progression: word2vec (pretrained word vectors only, independent of context) → ELMo/GPT/BERT (whole-model pretraining, contextual representations) → T5 (a unified text-to-text framework for multiple tasks) and larger language models.

1.2 GPT (Generative Pre-trained Transformer)

GPT-1 (2018)

A Transformer-based language model;
Pretraining objective: next-word prediction (an autoregressive LM);
Paper: Improving Language Understanding by Generative Pre-Training.

Training and model details

Item	Details
Subword tokenization	Byte-Pair Encoding (BPE)
Size	12-layer Transformer, 117M parameters
Pretraining corpus	BookCorpus
Corpus characteristics	Contains 7,000 distinct books in a variety of styles, providing large amounts of continuous text (making long-range dependencies easier to learn)

Adapting to downstream tasks

For classification tasks, process the entire input and then add a classification layer at the end;
- This classification layer is randomly initialized.
Use the downstream task's dataset to continue training the entire network. This is called fine-tuning.

Fine-tuning example: sentiment classification

Initialize a classification layer $W_y$ , using the final word's output $h_n$ as input:

P(y\mid x) = \text{softmax}(h_n W_y)

Training example: <s> I love pizza! <e> → label: positive
Loss $= -\log(0.4)$ (when the model assigns a probability of only 0.4 to positive)

Fine-tuning example: question answering

The GPT-1 paper uses multiple-choice question answering: concatenate the context, question and each candidate answer, score them separately, then apply softmax across the candidates.
Predicting the answer's start and end positions in a passage is a different, extractive QA formulation, used when fine-tuning BERT on SQuAD. The two task formulations should not be conflated.

GPT results: GLUE and other benchmarks

Benchmark	Task type
MNLI, RTE	Textual entailment
QQP, STS-B, MRPC	Sentence similarity
QNLI	Answerability prediction
SST-2	Sentiment analysis
CoLA	Sentence acceptability

1.3 BERT (Bidirectional Encoder Representations from Transformers, 2018)

Key distinction: it uses bidirectional self-attention.

Each target word attends to both its left and right context to build its contextual representation (GPT can only look to the left);
Paper: arXiv:1810.04805.

Figure 1-1 BERT's two stages: unsupervised pretraining on the left (Masked LM + NSP), and fine-tuning for downstream tasks on the right (classification, QA and sequence labeling). Source: Devlin et al., BERT, NAACL 2019, Figure 1.

Pretraining objective: masked word prediction (masked language modeling, MLM)

Randomly select about 15% of tokens as prediction targets; they are not all replaced with [MASK].
Objective: predict the masked words.
Details: of the selected tokens, 80% are replaced with [MASK], 10% with random words, and 10% are left unchanged. Loss is computed only at the selected positions.
The original BERT also uses Next Sentence Prediction (NSP): determine whether two text segments are adjacent in the original text.

Training and model details

Item	Details
Subword tokenization	WordPiece
BERT-base	12-layer Transformer, 110M (110 million) parameters
BERT-large	24-layer Transformer, 340M (340 million) parameters
Pretraining corpus	Wikipedia + BookCorpus
Training cost	Multiple GPUs and several days of training

Adapting to downstream tasks

As with GPT, add a classification layer;
BERT, however, uses a special first token, [CLS], whose representation becomes the input to the classification layer:

P(y \mid x) = \text{softmax}(h_{[\text{CLS}]} W_y)

Example: spam detection

TEXT

[CLS] Congratulations! You won $1000 ... [SEP]
   ↓ (BERT bidirectional encoding)
 h_[CLS] → classification layer → spam / not spam

BERT results: the original paper achieved leading results at the time on language understanding benchmarks such as GLUE and SQuAD. These comparisons concern specific datasets, models and experimental settings.

Encoder

BERT's bidirectional Transformer is also called a transformer encoder;
Each output word depends on bidirectional context (bidirectional attention);
Advantage: bidirectional context suits understanding tasks such as classification, extraction and sequence labeling;
Limitation: the original BERT's MLM objective and bidirectional attention do not directly provide left-to-right autoregressive generation. BERT can serve as the encoder in a generation system, or be adapted for generation with additional design, but cannot directly be used as a GPT-style decoder.

Figure 1-2 Three pretrained architectures: BERT (bidirectional Transformer, encoder-only), OpenAI GPT (unidirectional, decoder-only), and ELMo (bidirectional LSTM). Source: Devlin et al., 2019, Figure 3.

1.4 Decoder and encoder–decoder

Decoder

GPT's unidirectional Transformer is also called a transformer decoder;
Each output word depends only on its left context (unidirectional / causal attention);
It supports both classification and generation;
At each position, its available context differs from an encoder's. Classification performance also depends on model size, training objectives, data and fine-tuning methods; it cannot be determined from attention direction alone.

Encoder–decoder

Motivation: handle input representation and output generation separately, suiting sequence-to-sequence tasks whose input and output lengths may differ.
How they work together:
1. Use the encoder to process the input (bidirectional understanding produces rich source representations)
2. Use the decoder to generate the output (autoregressively, word by word, reading the encoder's output through cross-attention)

Comparing the three architectures

Architecture	Examples	Attention	Strengths
Encoder-only	BERT	Bidirectional	Classification, extractive QA, sequence labeling
Decoder-only	GPT family, LLaMA	Unidirectional (causal)	Generation, few-shot learning, dialogue
Encoder-Decoder	T5, BART, original Transformer	Bidirectional encoding + unidirectional decoding + cross-attention	Seq2seq tasks such as translation and summarization

1.5 T5 (Text-to-Text Transfer Transformer, 2019)

Figure 1-3 T5's text-to-text framework expresses translation, classification, summarization and regression as "input text → output text." Source: Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR 2020, Figure 1.

Pretraining objective: span corruption

Randomly select several contiguous token spans, replace each with a different sentinel, and train the model to generate the removed content;
Example: Thank you <X> me to your party <Y> week → target: <X> for inviting <Y> last <Z>

Figure 1-4 Span corruption: replace contiguous spans with sentinels <X> <Y> <Z> and train the model to generate the removed content. Source: Raffel et al., 2020, Figure 2.

Other possible objectives: a comparison

Objective	Example	Description
Prefix language model (prefix LM)	cows love to → eat grass	Given a prefix, predict what follows (similar to GPT, but implemented with an encoder–decoder)
Masked language model (masked LM)	cows <M> <M> eat grass → cows love to eat grass	Randomly mask words and predict them (similar to BERT)
Shuffling (word-order corruption)	grass cows to love eat → cows love to eat grass	Shuffle the word order and train the model to restore it
Drop tokens (token deletion)	cows love grass → cows love to eat grass	Delete some input words and train the model to reconstruct the complete original text

T5 ultimately uses span corruption: it was competitive in the paper's comparisons and shortened the target sequence to be generated. This is an experimental design trade-off, rather than evidence that it outperforms other objectives on every task.

Pretraining corpus: filtered C4 (Colossal Clean Crawled Corpus)

C4 consists of filtered web text;
Raw web pages contain considerable noise, including nonlinguistic text, non-English content and offensive content;
A series of heuristics filters the data (for example, retaining only sentences ending in punctuation, removing pages with profanity, and deduplicating);
Quality matters alongside quantity (C4 performed much better than a similar amount of raw crawled data).

Fine-tuning

Express all tasks as text-to-text problems;
Fine-tune with a next-word prediction objective;
Examples: translation translate English to German: That is good. → Das ist gut.; classification sst2 sentence: it's great → positive.

T5 results

Item	Details
Model sizes	small (60M), base (220M), large (770M), 3B, 11B
GLUE	A benchmark containing nine language understanding tasks
CNN/DM	Summarization
WMT	Machine translation
Conclusion	The largest version achieved leading results at the time on several benchmarks evaluated in the paper; this does not imply human-level general language ability

2. Large language models and RLHF

2.1 What is an LLM?

A large language model (LLM) generally refers to a large model pretrained on substantial amounts of text; there is no universally agreed parameter threshold. This section focuses on Transformer models and their zero-shot generalization, in-context learning and instruction-following abilities.

The development of these methods can be summarized as: statistical language models → neural language models → whole-model pretraining → scaling models and data → instruction tuning and preference alignment. Word2Vec is a method for word representations, and should not be equated with a complete language model; scaling and alignment are also distinct.

2.2 Generative Pre-trained Transformers and emergent abilities

The role of pretraining and fine-tuning in NLP: pretraining can improve downstream task performance through parameter initialization.

Model	Parameters	Layers	Dimension	Training data	Training approach
GPT-1	117M	12	768	BookCorpus (~1B words)	Self-supervised pretraining + supervised fine-tuning
GPT-2	1.5B	48	1600	WebText (~10B words)	Self-supervised pretraining
GPT-3	175B	96	12288	A mixture of Common Crawl, WebText, books and Wikipedia	Self-supervised pretraining

GPT-2's emergent ability: zero-shot learning

It can perform various tasks without examples or gradient updates;
Without task-specific fine-tuning, it surpassed the state of the art at the time on language modeling benchmarks.

GPT-3 and in-context learning

The models in the GPT-3 paper were trained on about 300B tokens in total, using a weighted mixture of several corpora. File size, word count and token count are different measures.
At some scales, such as 175B, models can follow instructions without demonstrations (zero-shot);
Specify the task by prepending some examples of that task to the input;
This is called in-context learning, emphasizing that adapting to the new task involves no gradient updates.

Figure 2-1 Four ways to use GPT-3: fine-tuning (parameter updates) versus zero-shot, one-shot and few-shot settings (examples without parameter updates). Source: Brown et al., Language Models are Few-Shot Learners, NeurIPS 2020, Figure 2.1.

Traditional fine-tuning vs In-context Learning

Aspect	Traditional Fine-tuning	In-context Learning
Parameters	Model parameters must be updated	No parameters are updated
Data	Many labeled examples are required	Only a few demonstrations are needed (possibly none)
Process	Pretrain → fine-tune separately for each task → deploy multiple models	One model + different prompts
Cost	Training and storage for each task	Higher inference cost with long prompts, but no training cost

Limitations of in-context learning

Highly sensitive to the prompt's format, example selection and example order;
The context length limits how many examples fit;
It can still fail on complex reasoning tasks, such as multistep arithmetic → motivating CoT (Section 3);
Without parameter updates, adaptation occurs within the current context. This does not establish that the model has reliably learned the task's underlying rules; new examples, prompts and demonstration settings still need to be tested.

2.3 InstructGPT and RLHF (Reinforcement Learning from Human Feedback)

Overall idea: collect human demonstrations and preferences, then use supervised learning and reinforcement learning to adjust model behavior. The following example uses the three-stage process published in the InstructGPT paper. It also helps explain the training ideas behind ChatGPT, but InstructGPT's data volumes and model configurations should not be treated as ChatGPT's complete training details.

Figure 2-2 Three RLHF stages: (1) supervised fine-tuning (SFT) → (2) train a reward model → (3) optimize the policy using PPO and rewards. Source: Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT), 2022, Figure 2.

Step 1: supervised fine-tuning (SFT)

1. Collect high-quality data

Create a prompt list: identify the questions or statements to use for training;
Request desired answers: collect the desired responses to these prompts;
Human experts or crowdsourcing produce the data, ensuring quality and diversity;
The InstructGPT paper's SFT training set contains about 13,000 prompts and corresponding demonstrations.

2. Fine-tune the pretrained model

The InstructGPT paper uses pretrained GPT-3 as the base model;
Pretraining provides general language abilities, while SFT adapts the model to the desired response style;
Optimize parameters with the high-quality data above to better support tasks framed as instructions.

Step 2: reward model (RM)

1. Generate outputs

Select a prompt from the prompt list;
Use the SFT model from Step 1 to generate about 4–9 outputs.

2. Human ranking

Annotators rank the outputs by quality, from best to worst;
The paper's RM dataset contains about 33,000 prompts. Each ranking can yield multiple pairwise comparisons; counts of prompts, answers and comparison pairs must be distinguished.

3. Train the reward model

Input: a prompt + its corresponding SFT model output;
Training signal: which response annotators prefer for the same prompt;
Output: a predicted scalar reward. Training assigns higher scores to preferred responses, rather than directly regressing absolute scores given by annotators;
The paper uses a reward model with 6B (6 billion) parameters.

Step 3: proximal policy optimization (PPO)

1. Sample a prompt: sample a prompt from the dataset, such as "Write a story about Melbourne." The paper's PPO dataset contains about 31,000 prompts; this stage does not require a human demonstration answer for each prompt;

2. Generate a response: generate text with the current policy model. The policy is initialized from the SFT model and then updated by PPO during training iterations; its parameters are not updated on the fly every time it responds to a user;

3. Compute the reward: the reward model trained in Step 2 evaluates the generated output and returns a scalar reward;

4. Update the policy: use the reward to update the policy with the PPO algorithm.

PPO is a reinforcement learning algorithm designed to control the size of policy updates, using new reward information to optimize the policy while maintaining training stability (avoiding overly large updates that cause the model to collapse).

Additional constraint: training adds a KL penalty relative to the SFT reference model to limit policy drift and mitigate problems from overoptimizing the reward model. This does not guarantee that all reward hacking is eliminated.

Adapting for dialogue

Dialogue models also need data and evaluation for formats such as multiturn conversations, addressing context continuity, refusal of inappropriate requests and format compliance. These are choices of training objectives and data design, rather than a fixed "fourth step" in the three-stage process above.

References: InstructGPT paper.

2.4 Other LLMs: LLaMA

LLaMA (Large Language Model Meta AI)

This refers to the first generation of LLaMA, released in 2023 with model sizes ranging from 7B to 65B parameters.

Compared with GPT-3 175B, LLaMA explores training smaller models on more tokens to improve performance for a given inference budget. This does not mean every LLaMA model is smaller than every GPT model;
It uses publicly available data from web pages, books, Wikipedia, GitHub, papers and other sources. Including code data does not establish that code is the sole cause of all improvements in reasoning;
It uses a decoder-only Transformer, combining existing design choices:
- Pre-normalisation with RMSNorm;
- SwiGLU activation;
- Rotary Position Embedding (RoPE);
The paper discloses the architecture and training settings. The first-generation weights were made available to researchers under the research license then in effect; publicly available weights must be distinguished from an unrestricted open-source license.

References: LLaMA paper.

2.5 Three core LLM abilities

Ability	Meaning	Example
In-context learning	Adapt to a task using instructions, examples and information in the current input, without updating model parameters; memory across sessions requires additional system mechanisms	Perform a new task after seeing two examples
Step-by-step reasoning	Break a complex problem into smaller steps	Mathematics, programming, logical reasoning
Instruction following	Follow the user's explicit instructions, serving as an effective tool for a range of tasks	Write an email to specification, generate steps from a recipe, execute a sequence of commands in a software environment

3. Prompting and chain of thought

3.1 Zero-shot and few-shot in-context learning

Specify the task by adding examples of it before the actual input;
This is called in-context learning, emphasizing that adapting to a new task involves no gradient updates;
In other words, the model reasons or generates results using only the context in its current input.

Setting	Number of examples	Description
Zero-shot	0	Only an instruction, such as "Translate to French: cheese"
One-shot	1	One example
Few-shot	Several	Several examples, typically improving performance

3.2 Limitations of prompting

If a task is too complex, prompting an LLM may not produce correct reasoning, especially when it involves substantial information or multiple reasoning steps.

Failure example: standard prompting

Q: A cafeteria has 23 apples, uses 20 for lunch and buys 6 more. How many does it have now? A: The model outputs 27 ❌

The incorrect answer in this example fails to compute $23-20+6$ correctly; 27 cannot be explained as $23+6$ , because the latter equals 29.

Alternative: change the prompting style ✅ (using CoT, described next).

3.3 Chain-of-thought prompting (CoT)

Definition: provide intermediate reasoning steps in examples, or prompt the model to generate such steps before the final answer. The original paper observed improvements on various arithmetic, commonsense and symbolic reasoning tasks, but intermediate steps are not guaranteed to be correct.

Comparison

Approach	Prompt	Output
Standard Prompting	Only questions and answers	The answer is 27 ❌
Chain-of-Thought Prompting	Guide the model through reasoning step by step	The answer is 9 ✅

Example

Q: A cafeteria starts with 23 apples, uses 20 for lunch and buys 6 more. How many does it have now? A: It starts with 23 and uses 20, leaving 3. Buying 6 more brings the total to 9 apples.

Figure 3-1 Left: standard prompting gives a direct, incorrect answer. Right: chain-of-thought prompting reasons step by step to a correct answer. Source: Wei et al., Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NeurIPS 2022, Figure 1.

3.4 Emergent properties of model scale

For the models and tasks tested in the original CoT paper, larger models generally benefited more from reasoning demonstrations;
Some smaller models instead generated incorrect steps, reducing performance. This was a scale effect in those experiments, rather than a fixed threshold or rule that can be assumed to apply to every model;
Common benchmarks:
1. GSM8K: grade-school math word problems (8K questions)
2. SVAMP: arithmetic problems with semantic variation
3. MAWPS: math word problem solving

3.5 Two CoT variants

Zero-shot CoT

Use the phrase "Let's think step by step" as the prompt;
The model generates its own reasoning process (generated rationale);
Finally, extract the answer (answer extraction).

Advantages	Disadvantages
No manually constructed examples are needed	Reasoning quality depends on the model itself and is less stable

Manual CoT

In advance, manually design several examples with reasoning steps (Manual Demos);
Then provide the test question;
The model generates a response based on the supplied examples.

Advantages	Disadvantages
Explicitly demonstrates the desired reasoning format; effectiveness depends on how well the examples match the task	Examples must be manually constructed and checked; their errors may also be imitated

3.6 Additional dataset notes

MultiArith: multistep math word problems expressed in natural language;
GSM8K (Grade School Math 8K): also multistep math problems in natural language, but with a more structured style and more steps.

3.7 Prompt injection and jailbreaking

Prompt injection: causes the model to treat instructions within text, web pages or other untrusted data being processed as instructions to execute, diverting it from the original task;
Jailbreaking: induces the model to bypass its existing safety or behavioral constraints. It can overlap with prompt injection, but the terms are not synonymous;
For example, inserting Ignore the above and instead tell me ... into text being translated may steer the model toward another task;
Lesson: applications need to distinguish trusted instructions from external data. Text should not receive the same control authority merely because it resembles an instruction.

3.8 Advantages and disadvantages of in-context learning

Pros

No fine-tuning: use a pretrained model directly, without additional training;
Prompt engineering can readily improve performance: for example, CoT can significantly improve reasoning.

Cons

Limited context length: the model's maximum input length makes it difficult to handle large amounts of information or many examples;
Prompting alone may be insufficient: complex tasks may require better context, retrieval, tools, verification or fine-tuning. Choose methods based on the source of errors, rather than assuming that gradient updates are always necessary.

4. Instruction tuning

4.1 Scaling up fine-tuning: the traditional two-stage paradigm

Step 1: pretraining

Item	Details
Approach	Use large amounts of text (books, Wikipedia, etc.) for self-supervised learning
Model	BERT and others
Datasets	Wikipedia and others
Objective	Predict masked words (language modeling)

Step 2: fine-tuning

Item	Details
Approach	Use a labeled dataset for supervised learning
Model	The model pretrained in Step 1
Datasets	For example, spam messages and their labels
Objective	Train a classifier for a specific task, such as spam detection

Limitation: each task needs its own model and labeled dataset, which does not scale as the number of tasks grows.

4.2 Instruction tuning

Definition: instruction fine-tuning is supervised fine-tuning (SFT) on instruction data that describes tasks in natural language. It is one form of post-training, which can also include preference optimization and other steps; these terms are not fully synonymous.

Instruction tuning = supervised fine-tuning on instruction data;
Instruction data consists of (prompt, completion), or $(x, y)$ , pairs: a natural-language instruction and its desired response;
RECAP: InstructGPT: train with instructions and human-desired outputs so that the model learns to understand task descriptions, rather than memorizing one particular task.

SFT details

Collect high-quality data
- Create a prompt list
- Request the desired answers
- Have humans create the data (experts and crowdsourcing)
- InstructGPT, for example, uses about 13,000 prompts and corresponding demonstrations for SFT; data volumes differ across projects
Fine-tune the pretrained model
- Start with a pretrained model. The InstructGPT paper uses GPT-3; SFT in general should not be equated with GPT-3.5 training

This corresponds to the SFT stage of the InstructGPT process in Section 2. SFT → RM → PPO is the specific process used in that paper; RLHF is not restricted to this combination of algorithms.

4.3 Alpaca (Stanford Alpaca)

Result: in the authors' preliminary, small-scale, single-turn instruction-following evaluation, Alpaca 7B performed similarly to text-davinci-003. This does not establish equivalence across all tasks, safety or reliability.

Training process

Step 1: seed generation (Self-Instruct)

Start with the Self-Instruct seed set, using 175 human-written instruction–output pairs;
Use text-davinci-003 (GPT-3.5) to generate more instructions;
Generate a total of 52,000 distinct instructions and their corresponding outputs.

Step 2 – Fine-tuning

Use the Hugging Face training framework to fine-tune LLaMA;
Fine-tuning the 7-billion-parameter LLaMA model on eight 80GB A100 GPUs took 3 hours;
The compute cost of this fine-tuning run was under US$100. Generating the training data cost under US$500, bringing the combined cost to under US$600, excluding pretraining the base model.

References: Stanford Alpaca project description.

4.4 The T5 perspective: pretraining and multitask training

Approach: pretrain first, then fine-tune on multiple specific tasks (Tasks A, B, C…) and evaluate on those same tasks.

A trend in newer language models: GPT-3 and LLaMA

Move away from emphasizing separate fine-tuning for each task;
Instead, after large-scale pretraining, specify tasks through instructions or examples (instruction prompting), without task-specific fine-tuning.

4.5 Three paradigms: pretrain–fine-tune, prompting and instruction tuning

Paradigm	Examples	Steps	Characteristics
Pretrain–Finetune	BERT, T5	Pretrained LM → fine-tune on Task A → inference for Task A	Train a dedicated model for each task; usually requires many task-specific examples
Prompting	GPT-3	Pretrained LM → few-shot prompting or prompt design → Task A	No parameter updates, enabling flexible use; sensitive to prompts, with performance depending on prompt quality
Instruction Tuning	FLAN	Pretrained LM → instruction-tune using natural-language instructions across multiple tasks → generalize to a new Task A	Language instructions teach the model to perform multiple tasks and generalize to unseen tasks

Figure 4-1 Three paradigms: (A) pretrain–fine-tune uses one model per task; (B) prompting changes the prompt without changing parameters; (C) instruction tuning across multiple tasks can support generalization to unseen tasks. Source: Wei et al., 2022, Figure 2.

4.6 FLAN (Instruction Fine-Tuning)

Figure 4-2 Instruction tuning overview: express multiple tasks as natural-language instructions, fine-tune, then use instructions for zero-shot inference. Source: Wei et al., Finetuned Language Models Are Zero-Shot Learners (FLAN), ICLR 2022, Figure 1.

Core approach: collect examples of (instruction, output) pairs spanning multiple tasks and fine-tune a language model.

The original FLAN expresses tasks as natural-language instructions and performs multitask instruction tuning on the 137B LaMDA-PT model;
The later Flan-T5 applies an expanded instruction-tuning approach to T5, improving performance on various tasks tested in the paper. Their base models and data collections differ, so the experimental settings should not be conflated.

Task setup

Item	Details
Task type	12 types (including classification and generation)
Datasets	62
Constructing unseen tasks	Tasks in the same cluster must appear in either the training or test set, never both (preventing leakage between related tasks and testing generalization)
Templates	For each task, manually create multiple templates to increase instruction diversity

Results: after instruction tuning, zero-shot FLAN's performance on three unseen task types, compared with zero-shot and few-shot GPT-3, improved substantially.

Generalization to new tasks

In the original FLAN experiments, effectiveness varied with model scale, with larger models benefiting more from instruction tuning;
Insufficient model capacity is one possible explanation, but tasks, data and training settings also matter. These results do not justify a claim that all small models are unable to understand instructions.

Scaling the number of tasks: later FLAN versions

Item	Number
Scale of expansion	Later work expanded instruction tuning to about 1.8K tasks and examined the effects of model scale and CoT data
Held-out tasks	Excluded from fine-tuning and used to test generalization

Other findings

There are different fine-tuning data formats (for example, with or without explanations or answer choices);
Joint fine-tuning on non-CoT and CoT data can improve performance on both types of evaluation tasks;
On the unseen task types evaluated in the original paper, zero-shot FLAN was comparable to few-shot GPT-3. This is an experimental finding under specific benchmark and model settings.

References: Original FLAN, Scaling Instruction-Finetuned Language Models.

4.7 Benefits and limitations

Benefits

Simple, straightforward method (supervised learning + instruction formatting);
Can generalize well to unseen tasks.

Limitations

High-quality data has a cost: human demonstrations, existing datasets or model-generated synthetic data can be used, but all require curation, deduplication and quality control;
Some tasks have no single correct answer: for open-ended creative generation, such as "Write a story about my dog and child," a correct answer is hard to define;
Objective mismatch: generative models typically undergo SFT by predicting the next token in demonstration responses. Imitating demonstrations does not directly optimize every human preference.

We may therefore go on to use RLHF to optimize model behavior.

4.8 Discussing capabilities and limitations

Fluent text does not mean reliability and generalization have been solved. The following distinguishes research observations from broader conclusions.

Topic	Key points
Facts and planning	Models can assist with writing, but may also fabricate facts or propose infeasible plans. Instruction tuning and human feedback provide no guarantee of correctness; specific abilities require empirical evaluation
Self-correction	Research finds that, in certain reasoning settings without external feedback, merely asking a model to check and revise its answer may fail to improve accuracy and can even turn a correct answer into an incorrect one. This does not establish that all correction methods using tools or verifiers are ineffective
Reversal Curse	In some experiments, a model trained on "A is B" failed to answer the corresponding "B is A" query automatically. Fitting one form of expression does not necessarily produce generalization to its reverse; this does not mean every natural-language relation is directly reversible

References: Large Language Models Cannot Self-Correct Reasoning Yet, The Reversal Curse.

5. Parameter-efficient fine-tuning and LoRA

5.1 Why PEFT?

PLM and LLM: the former emphasizes pretraining, the latter scale. They are not mutually exclusive categories, and there is no universal parameter boundary. This section asks how to reduce task adaptation costs as models grow.

Fine-tuning becomes more expensive as models grow

Step 1: pretrain: learn general language abilities from large amounts of text;
Step 2: fine-tune: adapt to specific tasks, often with few labels.

Larger models bring greater costs and risks for full fine-tuning:

Issue	Description
Scalability	High computational cost (a complete model copy must be stored for each task)
Training instability	For example, overfitting and gradient interference
Catastrophic forgetting	After fine-tuning on a new task, the model loses its original general abilities

In-context learning and fine-tuning are complementary forms of adaptation. ICL requires no parameter updates, but still has limitations:

Limitation	Description
Inefficiency	Each inference must process the entire prompt again (long prompts are costly)
Performance	Prompting alone may not meet the target task's performance needs; comparisons with fine-tuning require the same settings
Sensitivity	Prompt wording, example order and similar choices can make performance highly sensitive
Lack of clarity	Success with one prompt does not establish stable generalization to other inputs and prompts

→ This motivates a shift from fine-tuning to parameter-efficient fine-tuning.

5.2 Historical context

Computer vision (CNNs): early approaches often fine-tuned only the last few layers, or updated only the final layer;
NLP: although fine-tuning only the final layer was once common, in NLP fine-tuning all parameters performs better, at greater cost, motivating parameter-efficient fine-tuning methods.

5.3 Fine-tuning only some parameters

1. Full fine-tuning is difficult for large models

For models with publicly reported sizes, such as GPT-3 175B, full fine-tuning requires substantial computation and GPU memory. Unverified parameter counts should not be supplied for models whose sizes have not been disclosed;
Fine-tuning all parameters is computationally costly and also difficult to deploy and update (one set of weights per task).

2. State-of-the-art models are heavily overparameterized

In practice, we do not need to update every parameter to achieve strong downstream task performance.

3. PEFT can approach full fine-tuning performance

For some tasks and settings, using a small number of trainable parameters, as in adapters, LoRA or prompt tuning, can approach full fine-tuning performance. There is still a trade-off between efficiency and adaptation capacity.

5.4 Modular and compositional representations

a) Models are increasing in size

As parameter counts grow, as with GPT-3 175B, full fine-tuning becomes computationally expensive and resource-intensive;
Researchers therefore turn to PEFT methods such as adapters, LoRA and prefix tuning, fine-tuning only a small subset of parameters to adapt to new tasks and substantially reduce training costs.

b) Unseen scenarios

In real applications, models often encounter languages, tasks or domains unseen during training, such as minority languages or cross-modal tasks;
Module Composition combines skill modules, such as language and task modules, to investigate generalization to unseen combinations. A compositional structure alone does not guarantee out-of-distribution performance.

c) Catastrophic interference

In conventional fine-tuning, new tasks can easily overwrite existing knowledge, causing catastrophic forgetting;
Using modular structure as an inductive bias can place different tasks or knowledge in separate modules, reducing interference and improving stability and long-term memory.

d) Efficient model updates

When knowledge changes rapidly, as with current affairs or changes in a person's position, retraining the whole model is impractical;
Updating small modules or using external knowledge can be explored to reduce retraining costs. Whether knowledge is updated correctly, and whether other behaviors are affected, still requires verification.

5.5 Architecture overview: three computation functions

PEFT methods can be grouped into three ways of modifying a frozen pretrained model:

Type	Approach	Examples
Addition-based	Attach small trainable modules to the model	Adapter, Prefix Tuning, Prompt Tuning
Selection-based	Select a subset of parameters or sparse updates to train	BitFit (bias terms only), Diff Pruning
Reparameterization	Learn low-rank or compressed changes to the weights	LoRA, Compacter

5.6 Sparse subnetworks and pruning

Sparsity is related to parameter-efficient adaptation, but ordinary pruning is not equivalent to PEFT.

Pruning usually focuses on model compression: select connections to keep, remove the others, and potentially retrain;
Sparse fine-tuning methods can instead select a small number of parameters to update, or learn sparse parameter changes;
Both use the idea of selecting a subset, but their training objectives, frozen weights and final deployment methods must be explained separately.

Sparse subnetworks

A common inductive bias for module parameters is sparsity;
The most common way to implement sparsity is pruning;
Pruning can be viewed as applying a binary mask that determines whether each connection is retained, producing a subnetwork;
During pruning, some connections with the lowest-magnitude weights are removed;
Weights that have not been pruned are re-trained;
Pruning can be done once, or in repeated rounds of pruning and retraining. Results depend on the model, pruning ratio and training settings.

TEXT

Initial training → pruning(mask) → retraining → pruning again → retraining again → … → sparse subnetwork
   (one-shot pruning: prune once; iterative pruning: multiple rounds)

Figure 5-1 One iterative pruning workflow: train → prune with a binary mask → retrain, repeating as needed.

5.7 LoRA (Low-Rank Adaptation)

Core idea

Fine-tuning a large model is expensive → do not directly update the full weight matrix $W$ ; instead, learn only its change $\Delta W$ :

W' = W + \Delta W

Express the trainable update as the product of two small matrices:

\Delta W = B \cdot A

The commonly used scaling factor $\alpha/r$ is omitted here to emphasize the structure of the low-rank update.

What does "low-rank" mean?

$W$ is the original weight matrix; it need not itself be full-rank;
LoRA learns only a low-rank change;
"Low-rank" means a compressed representation of change, with rank $r \ll d$ .

Efficiency: counting parameters

Suppose the original matrix $W$ has shape $1024 \times 1024$ (a total of 1,048,576 ≈ 1 million parameters);
LoRA uses two smaller matrices, $A\,(r \times d)$ and $B\,(d \times r)$ , where $r \ll d$ ;
For example, when $d = 1024,\ r = 8$ :
- $A$ : $8 \times 1024$ , $B$ : $1024 \times 8$
- Total trainable parameters: $2 \times 1024 \times 8 = \mathbf{16{,}384}$
Compared with the full parameter count, the number of parameters updated is reduced by about 98.4%:

1 - \frac{16{,}384}{1{,}048{,}576} \approx 98.4\%

Figure 5-2 LoRA reparameterization: freeze $W$ and train only the low-rank matrices $B$ and $A$ , giving the forward computation $h = Wx + BAx$ . Source: Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, ICLR 2022, Figure 1.

Forward computation: low-rank composition

\text{Output} = (W + B\cdot A)\,x = W\cdot x + B\cdot(A\cdot x)

Symbol	Meaning
$W$	Frozen pretrained weights (not updated)
$A, B$	Only these two matrices are trainable
$x$	Input vector, usually a token embedding or the previous layer's output

Backpropagation: compute and store gradients only for trainable parameters; the optimizer updates only $A$ and $B$ ; $W$ remains frozen. Gradients may still propagate through the frozen linear layer; this does not mean the layer is entirely excluded from backpropagation.

Advantages

Train only matrices $A$ and $B$ , rather than the entire $W$ ;
No additional inference latency (at inference time, $BA$ can be merged into $W'$ without changing the architecture);
Fewer parameter gradients and optimizer states need to be stored. In experiments on GPT-3 175B, the LoRA paper reports GPU memory requirements reduced to about 1/3 of full fine-tuning and a training speed improvement of about 25%. Actual gains depend on the implementation and training settings;
Small, separate LoRA weights can be saved for each task and switched as needed. The original weights are not overwritten, but behavior with a particular adapter still needs evaluation.

LoRA and its variants

Variant	Characteristics
LoRA	Standard low-rank adaptation, placed alongside linear layers in attention or FFNs
LoRA-FA	Freeze the down-projection matrix $A$ and train only the up-projection matrix $B$ , reducing activation memory
QLoRA	4-bit quantization of the base model + LoRA fine-tuning, enabling large-model training on a single GPU
HydraLoRA	Use an asymmetric structure with a shared $A$ and multiple $B$ matrices, routing between components to handle different constituents of mixed data

5.8 Comparing PEFT methods

The parameter proportions below indicate typical orders of magnitude. Actual values depend on the model, module sizes and number of trainable layers.

Method	Trainable parameters	Architecture changes?	Inference overhead
Full Fine-tuning	100%	No	None (but a separate model per task)
LoRA	Depends on rank and insertion points; usually far fewer than all parameters	Adds a training branch that can be merged for deployment	Usually none after merging
Adapter	1%–5%	Yes (inserted layers)	Slight increase (sequential computation)
Prefix/Prompt Tuning	<0.1%	No	Uses context length
BitFit	~0.01% (bias terms only)	No	None
Sparse selection / pruning	Method-dependent; ordinary compression pruning is not necessarily PEFT	Selects connections or updates	Fewer sparse parameters do not automatically produce practical speedups

References: LoRA, LoRA-FA, QLoRA, HydraLoRA.