A First Taste of Algorithms · Lab 9: The Engine

knobs: a score for each (previous letter -> next letter) pair

✦ ✦ ✦

W = [[Value(random.gauss(0, 0.1)) for _ in range(V)] for _ in range(V)]

data = [] for name in names: cs = [“.”] + list(name) + [“.”] for prev, nxt in zip(cs, cs[1:]): data.append((stoi[prev], stoi[nxt]))

for step in range(40): loss = Value(0.0) for prev, nxt in data: scores = W[prev] m = max(s.data for s in scores) # for numerical stability exps = [(s - m).exp() for s in scores] Z = exps[0] for e in exps[1:]: Z = Z + e p = exps[nxt] / Z loss = loss + (-(p.log())) # cross-entropy, Chapter 8 loss = loss * (1.0 / len(data))

for row in W:
    for w in row:
        w.grad = 0.0                             # reset blame each step
loss.backward()                                  # YOUR sweep prices every knob
for row in W:
    for w in row:
        w.data = w.data - 1.0 * w.grad           # gradient step, Chapter 8

if step % 10 == 0:
    print("step", step, " loss", round(loss.data, 3))

```

The loss falls — and every gradient driving it down came from the sweep you wrote, firing each operation’s blame-rule across a graph the computation built itself. This is the entire chapter, working, in pure Python. (Why reset w.grad = 0.0 each step? Predict what the += in every blame-rule does across steps if you don’t — then hold it for Break-it 1.)

5. NumPy, and a model that can speak

Pure Value is for understanding; numpy is for scale — arrays of numbers operated on all at once, the same mathematics a thousand times faster. The full name model (a 3-letter-context network) is provided as one block; read it for the shape you already know — embedding, hidden layer with tanh, scores, softmax, cross-entropy — and run it:

```python import numpy as np rng = np.random.default_rng(1)

names = “”“ada gauss euler noether hilbert euclid fermat cantor godel turing lovelace ramanujan poincare riemann hardy erdos klein hopper church kolmogorov galois abel jacobi lagrange laplace fourier cauchy leibniz newton pascal bernoulli dirichlet weierstrass dedekind peano banach markov bayes nash hamilton cayley boole russell minkowski hausdorff lebesgue hermite chebyshev legendre maxwell dirac feynman fermi bohr planck einstein pauli wiener shannon hamming huffman dijkstra hoare knuth karp cook tarski kleene curry milner tarjan floyd kruskal dantzig weyl cartan grothendieck serre deligne tate wiles taylor mazur selberg renyi turan lovasz tao gowers”“”.split() names = list(dict.fromkeys(names))

chars = sorted(set(“”.join(names))) + [“.”] stoi = {c: i for i, c in enumerate(chars)}; itos = {i: c for c, i in stoi.items()} V = len(chars); CONTEXT = 3

All 32 chapters of A First Taste of Algorithms · Foundations · All courses