Outcome Prediction Quiz: Stability

40 minutes · 100 points · 6 questions

How to answer

This quiz has six questions. Use your assignment experiments to predict the outcomes. Detailed experimental evidence and sound reasoning can recover up to 75% of missed points. Adjacent loss-regret intervals earn half credit. Each question points to its code diff in Appendix A.

Reference recipe

The released A4 recipe uses width 512, vocabulary 4096, context 1024, batch 64, 600,000 fresh training sequences (614.4M token slots; 9,375 updates), AdamW, linear LR decay with 1% warmup, masked weight decay .1, clipping at 1, and model/data seed 42. Lower validation loss is better.

The transformer is the default 8-block model with RMSNorm, QK normalization, RoPE, and SwiGLU MLPs. The Modern tanh baseline is the released four-block ModernRNN: each block contains a tanh recurrence and a SwiGLU MLP, both with depth residuals and pre-LayerNorm. The GRU keeps this depth stack and changes only the recurrent update.

Unless stated otherwise, train each variant from scratch for all 9,375 updates. Select the run with the lowest final validation loss on that variant’s released/reference LR grid; do not select an earlier training checkpoint. All runs use linear decay and the other defaults above.

Diagnostics use common held-out inputs and FP32. Depth activations are complete block outputs, before the readout norm; embeddings are excluded. Loss is mean cross-entropy per next-token target.

Code reference

The diffs apply independently to the released A4 starter at 4350166. Questions 1–3 change the measurements; Questions 4–6 add model variants. Training and execution backends remain those of the release.

Submitting reveals every answer and explanation. Your work is graded locally in this browser and is not sent to the course staff.