Lecturized
EN
Library · CC BY-SA 4.0

Alice's Adventures in aDifferentiable Wonderland

Simone Scardapane · 2024 · retold from the English original

Alice's Adventures in a Differentiable Wonderland, Volume I: A Tour of the Land

Cover illustration of the book

Choose your version

The same text, told at every level, in every language and look shown here. Pick, then listen or download.

LanguageEnglishRussianSpanish
Level1 · As written2 · Formulas explained3 · Formulas in words4 · Ideas and examples5 · Big picture
VoiceMale · Device
DrawingsGraphic inkWhite on black

Listen and watch

The same player the product gives for your own book: the voice, the drawings as they are drawn, the text.

A modern textbook on how neural networks really work, from tensors and gradients to transformers. Chapter 6 shows how a program finds its own gradients: automatic differentiation, the reverse walk behind every trained model.

This is a sample: 1 of 13 chapters.

From the lectures

A small stack of picture frames feeds a grid of reactions that grows from a few cells into an enormous table filling the canvas, marked too big.
A Recipe Made of Small Steps

A Table Too Big to Keep

Some steps have a sharp corner where the slope is not defined, such as the popular ReLU, which lets positive numbers through and turns negative ones into zero.

Four stations in a row; under each a growing stack of small tables, one under the first station, two under the second, three under the third and four under the fourth, each new stack updated by the station's input Jacobian.
Why the Obvious Ways Are Too Slow

Forward Mode Keeps Tables

So the book turns to the first genuine method, forward mode, and it rests on the chain rule. If one station feeds another, the reaction of the pair is the reaction of the second multiplied by the reaction of the first.

Four blocks in a row, the first, second and fourth marked as kept checkpoints and the third left empty; the backward arrows stop at the third block, a rerun arrow goes from the second block to the third, and a label says plus a quarter.
Walking Back From the Score

Checkpoints Save Memory

The book shows a way to trade some of that memory back for time, called gradient checkpointing. Picture four blocks in a row. On the forward pass, only the outputs of the first, second and fourth blocks are kept; these are the checkpoints.

A layer joins three inputs, valued 2, 1 and 3, to an output that matters by 4; the three links show their gradients 8, 4 and 12, each the output's importance times the input.
Asking the Table One Question

A Weight's Share of Blame

Why is this so much better? Take the most common station of all, a layer that mixes its inputs with a grid of weights, as every layer of a chat model or a photo model does many times over.

A row of copies of a page, each half as wide as the one before, shows a gradient shrinking by a factor of a quarter per layer, falling below one millionth after ten layers.
How Frameworks Walk Back, and Why ReLU Won

Factors That Multiply

Now back to the photocopier. Remember that when the backward walk passes an activation function, the travelling row is scaled by the function's slope. A deep model has many such stations, so the row is scaled again and again, and the factors multiply exactly like the copier's reductions.

The ReLU, flat to the left of zero and rising at 45 degrees to the right, with a fan of lines through the corner whose slopes run from 0 to 1, the flat one marked as the usual choice; above, three ways of writing ReLU give slopes 0, 1 and 0.5 at zero.
The Corner of the ReLU

Every Slope From 0 to 1

The careful answer uses the idea of the ruler. For a function that curves upward like a bowl, any line that touches it at a point and stays below the whole function is a subgradient there, and its slope is a valid substitute for the missing slope.

How it was made

Examples for: a curious adult with school maths who uses AI apps every day; add one modern instance per idea, such as how a chat assistant or a phone's photo app is trained

The source

The lectures retell the English text of arXiv 2404.17625 (version 3, 2025) by Simone Scardapane, published under an open licence that allows retelling and translation. The retelling, the drawings and the voice are made by AI and checked against the book by a second model. How AI is used.

The lectures adapt the original and are released under the same licence, free to share and adapt with credit to the author and a link to this page: CC BY-SA 4.0

Your own hard book, the same way

Upload a textbook or any difficult book. The first five minutes are free.

Upload a book