Outcome Prediction Quiz: Optimization 2

Instructions

Predict the outcomes of the following experiments. Each question will have 3 instructions:

• From the assignment provides some context, providing any needed information from the assignment.
• This question changes provides a natural language description of what is different from the assignment.
• The diff is the exact code that we run when creating the solution key (printed in Appendix A). It is a patch to the released A3 starter (a3_release at 57c60bb). Each question below includes its diff.

Check one box in every row. Options that are given as sketches are judged on matching shape; their axis values show rough magnitudes. Scoring: In rows whose options are ordinal choices, the adjacent option will receive half the maximal points. Each question has three rationale fields: the assignment problem you cite, what you observed in that experiment, and your rationale for choosing your answer based on those observations. Detailed proper citation, with experimental details and sound rationale can return 75% of missed points on a question.

Default setup

The standard setup of the Optimization 2 assignment: Llama-style architecture with 8 layers, hidden size 512, 8 attention heads, QK-norm, a 4k vocab size tokenizer, and about 35M parameters (d8); 614.4M training tokens (9,375 updates) of fresh data; AdamW with $\beta_1, \beta_2 = (0.9, 0.95)$ and $\epsilon = 10^{-8}$, linear decay learning rate schedule with 1% linear warmup (93 updates), peak learning rate .003, context length 1024, batch size 64, weight decay 0.1 (masked: hidden and readout matrices decay; embeddings and normalization gains do not), and 1.0 grad clipping; model and data seed 42. Lower validation loss is better. Seed-to-seed noise in the final validation loss is about .003.

Conventions

We follow the same conventions as the rest of A3.

• wsdX schedule: after the warmup, the peak LR is held constant and then decays linearly to zero over the final fraction $X$ of training (wsd0.2 decays from update 7,500).
• Regret (Problem 2(c)): a run’s validation loss minus the control run’s at the same update.
• Sharpness: $\lambda_{\max}$ is the top eigenvalue of the loss Hessian, measured in FP32 on the 64-sequence held-out probe. A family block is the Hessian restricted to one parameter family (embedding, attention, MLP, RMSNorm gains, LM head), as in Problem 4(a).
• Full-batch runs (Problem 3(a)): plain gradient descent on the first $N$ training sequences, embedding initialized $16\times$ larger, no clipping, no weight decay, constant LR $\eta$.