A First Taste of Algorithms · Building the Learner

From engine to learner

The engine prices knobs; a learner needs knobs arranged into something that computes. Three small classes, each a few lines, each built from the Value engine — and the design idea is the one classes exist to express: composition, small pieces that snap together.

Figure 9.2 — Composition at three zoom levels: a neuron, a layer of neurons, a network of layers — the same shape repeated, which is the whole reason classes exist.

A neuron holds a weight per input and a bias (all Values, all knobs), multiplies each input by its weight, sums, adds the bias, squashes with tanh. A layer is a row of neurons sharing inputs. A network is layers chained, each feeding the next — the same shape at three zoom levels, which is exactly what composition buys. Because every operation inside is a Value operation, the whole network — however deep — builds one giant graph as it runs forward, and the single backward() you wrote prices every weight in it. You do not write new calculus for a bigger network. The engine already handles arbitrary graphs; a deeper network is just a bigger graph.

That is the entire principle behind every deep learning framework in the world. The ones running in data centers differ from yours in three ways and only three: they store numbers in big arrays instead of one-at-a-time (the next section’s instrument), they run on hardware built for exactly this graph, and they have a decade of engineering in the constants — Chapter 6’s lesson, that the shape is yours and the constants are an industry, returning for its final bow. The idea on your screen is complete.

The instrument, and the model

To train on real data at any speed, one concession — the same kind Chapter 6 made for clocks and Chapter 7 for dice: an instrument earned by need. Pricing Values one at a time is correct but slow; a network’s thousands of multiply-adds want to happen together. NumPy is Python’s tool for holding numbers in arrays and operating on whole arrays at once — and it is to this chapter what the stopwatch was to Lab 6: adopted for scale, framed honestly, the same mathematics at a thousandfold the speed. The pure-Value engine stays on the workbench as the thing that explains; numpy becomes the thing that runs. (Lab 9 keeps both: it trains a tiny model with your own Value engine, end to end, to prove the engine is real — then rebuilds the same idea in numpy to train at a size that can actually speak.)

The model itself is the simplest one that generates: it reads the last few characters and predicts the next, then feeds its own prediction back as input and predicts again, spinning out a name one letter at a time. The training data is this book’s own cast — a list of mathematicians’ names. (The example lists that have seeded this book since Lab 1 — Ada, Erdős, Gauss — were always meant to arrive here.) The loss is the one Chapter 8 built, averaged over every next-letter prediction across every name: how surprised was the model by the letter that actually came next? Train by the loop you ran on the bowl — gradient down, repeat — and the model’s surprise falls, its letters grow wordlike, and eventually it produces names like Riemanujan, Dantor, Wilevin: nobody’s name, everybody’s plausible mathematician. The machine learned the shape of the names without memorizing the names, which is the whole of what learning is.

And here the parrot exam from Chapter 8 returns to keep everyone honest, because it has teeth in this chapter. As the model trains, its training loss keeps falling — but its loss on held-out names it never saw will, past a point, start to rise. That gap is overfitting, live, in your own run: the model beginning to memorize its specific training names rather than learn the general music of names. Lab 9 makes you watch both curves and name the moment they part ways. The most expensive lesson in modern AI, the one Chapter 8 said no one fully understands at scale, is reproducible on your laptop in under a minute — and seeing it small is the beginning of taking it seriously large.

All 32 chapters of A First Taste of Algorithms · Foundations · All courses