Finetuned Language Models are Zero-Shot Learners
Jason Wei presented his 2021 ICLR paper Finetuned Language Models Are Zero-Shot Learners at the Stanford NLP journal club:
The idea is simple: finetune a language model on a collection of NLP tasks, described using natural language. The result is that the model performs tasks it hasn’t seen before via instructions.
The proof
Background
“Instruction tuning” .
Motivation
Training
62 NLP datasets, with 12 task clusters
Generate natural language instruction templates for each task
137B parameter pretrained checkpoint. Instruction tune all parameters for 30 k steps on 62 datasets, spanning 12 task clusters.
Results
More effective on tasks where verbalizing instructions is easy
Performs better from prompt tuning
Scaling Laws
Ablation
Clusters don’t have same number of datasets and different datasets have different sizes
10 templates per dataset, trying different ways of rephrasing the same goal. Additional templates has little effect
Natural language instructions critical to zero-shot learning
Contact: jasonwei@google.com