Broad-coverage finetuning improves instruction following

Pretrain on B tasks, then finetune with one demonstration per instruction; test on the tasks held out from pretraining.

Pretrained only (OOD)
+ 1 demo / instruction (OOD)

OOD Success: Before vs. After 1-Demo Finetuning

Takeaway: Sparse pretraining already learns the skills; one demonstration per instruction steers them to new combinations (0.4% → 54.7%).