CS 312: Deep Learning Alchemy

Logistics

• Lectures: Tues/Thurs, 1:30-2:50pm in CoDa B90
• Recordings: TBA (All class materials will be made public)
• Contact: TBA

Coursework

Structure

The goal of this class is to develop human intuition for the variety of strange deep learning phenomena that exist in the real world. Our two core beliefs are:

• Alchemy is learned by hand: Deep learning has many mysteries and surprises that are not well understood by the best researchers, let alone deep learning classes. As such, many classes teach mental models that are not supported by an empiricist's experience (e.g. classic bias-variance tradeoff disagrees with the current scaling era). This class will teach students how to design clean experiments that enable them to build intuition for themselves.
• Prediction is understanding: To build a scientific understanding is to be able to predict the outcome of an experiment before running it. Therefore, this class assesses human understanding via quizzes where students have to predict experiment outcomes.

In this class, students will alternate between running experiments to understand a given unit (e.g. architecture stability) and taking quizzes testing their understanding. Course staff will support student learning with lectures on each unit and tutorials for interactive discussion.

class Block(nn.Module):
    def __init__(self, width, ...):
        super().__init__()
        self.attn_norm = RMSNorm(width)
        self.attn = Attention(width, ...)
        self.mlp_norm = RMSNorm(width)
        self.mlp = MLP(width, ...)

    def update(self, x):
        attn = self.attn(self.attn_norm(x))
        mlp = self.mlp(self.mlp_norm(x + attn))
        return attn + mlp

    def forward(self, x):
        return x + self.update(x)


class Transformer(nn.Module):
    def __init__(self, vocab_size, depth=8, ...):
        super().__init__()
        self.token_embed = nn.Embedding(vocab_size, ...)
        self.position_embed = nn.Embedding(...)
        self.blocks = nn.ModuleList(
            [Block(...) for _ in range(depth)]
        )
        self.final_norm = RMSNorm(...)
        self.lm_head = nn.Linear(..., vocab_size)

    def forward(self, tokens):
        positions = torch.arange(
            tokens.size(1), device=tokens.device
        )
        x = self.token_embed(tokens)
        x = x + self.position_embed(positions)
        for block in self.blocks:
            x = block(x)
        return self.lm_head(self.final_norm(x))

@@ -10,4 +10,4 @@ class Block.update
 def update(self, x):
-    attn = self.attn(self.attn_norm(x))
+    attn = self.attn(x)
-    mlp = self.mlp(self.mlp_norm(x + attn))
+    mlp = self.mlp(x + attn)
     return attn + mlp

@@ -16,2 +16,2 @@ class Block.forward
 def forward(self, x):
-    return x + self.update(x)
+    return self.update(x)

@@ -10,4 +10,3 @@ class Block.update
 def update(self, x):
     attn = self.attn(self.attn_norm(x))
-    mlp = self.mlp(self.mlp_norm(x + attn))
-    return attn + mlp
+    return attn

@@ -10,4 +10,3 @@ class Block.update
 def update(self, x):
-    attn = self.attn(self.attn_norm(x))
-    mlp = self.mlp(self.mlp_norm(x + attn))
-    return attn + mlp
+    mlp = self.mlp(self.mlp_norm(x))
+    return mlp

@@ -22,2 +22,1 @@ Transformer.__init__
         self.token_embed = nn.Embedding(vocab_size, ...)
-        self.position_embed = nn.Embedding(...)
@@ -30,8 +29,4 @@ Transformer.forward
     def forward(self, tokens):
-        positions = torch.arange(
-            tokens.size(1), device=tokens.device
-        )
         x = self.token_embed(tokens)
-        x = x + self.position_embed(positions)
         for block in self.blocks:
             x = block(x)

@@ -34,3 +34,9 @@ Transformer.forward
 x_two_back = None
 for i, block in enumerate(self.blocks):
-    x = block(x)
+    update = block.update(x)
+    residual = (
+        x if i == 0
+        else 0.5 * x + 0.5 * x_two_back
+    )
+    x_two_back, x = x, residual + update

Assignments

Current assignment topics are tentative. Assignments 1 through 4 will study language model pre-training at a small scale reflective of larger-scale phenomena. Assignments 5 through 7 will switch to DNA.

1. Basics: Hyperparameter tuning and scaling
2. Optimization 1: Sharp and flat basins
3. Optimization 2: Hyperparameter invariants
4. Architecture 1: Extending model capacity
5. Architecture 2: Stability across depth/time
6. Generalization: Data-efficient algorithms
7. Exotic: ???

Grading

• 15% will come from tutorial participation (with 2 permitted absences)
• 85% will come from quizzes (with 2 drops out of 7 quizzes)
• 0% will come from assignments (except for their use as partial credit when cited in a quiz)

Attendance is mandatory on quiz and tutorial sessions. There will be no rescheduling or make-up opportunities except for those covered by accommodations.

Honor code

• Collaboration: We encourage students to discuss experiment ideas with each other over the week and at tutorial. However, since we believe that running experiments confers deeper understanding, we require all submitted experiments to be run by the student. If you discussed experiments with anyone, please put their names at the top of your assignment.
• AI tools: We view assignments as instrumental to improving human understanding. Therefore, we support AI usage on assignments for implementation, visualization, mock quizzes, etc. However, since grades will be primarily determined by quiz performance, we encourage students to be thoughtful about what AI usage enhances their ability to digest phenomena (as measured by their ability to predict experiment outcomes).

Submitting coursework

• All coursework is submitted via Gradescope by the deadline (not by email). You can submit as many times as you'd like until the deadline: we will only grade the last submission.
• Since assignment reports are not necessary to take the quiz, we will not accept any late reports.
• If you believe that the course staff (and their models) made an objective error in grading, you may submit a regrade request on Gradescope within 3 days after the grades are released.

GPU compute for self-study

If you are following along at home, you can access GPU compute from a cloud provider to complete the assignments.

Here are a few options (public pricing for a single B200 GPU on March 28, 2026):

• Modal: $6.25/hour. Offers $30 of free monthly compute. You are only charged for actual compute (no idle resources) and their UX makes switching between local dev and large-scale gpu experiments simple. (Modal Pricing)
• Lambda Labs: $6.69/hour (Lambda Pricing)
• RunPod: $4.99/hour (RunPod Pricing)
• Nebius: $5.50/hour ($3.05/hour preemptible) (Nebius Pricing)
• Together: $7.49/hour, minimum 8 GPUs, cheaper for longer commitments (Together Pricing)

Sponsor

We would like to thank Modal for sponsoring compute for this class.