Alice's Adventures in a Differentiable Wonderland, Volume I: A Tour of the Land
The same text, told at every level, in every language and look shown here. Pick, then listen or download.
The same player the product gives for your own book: the voice, the drawings as they are drawn, the text.
A modern textbook on how neural networks really work, from tensors and gradients to transformers. Chapter 6 shows how a program finds its own gradients: automatic differentiation, the reverse walk behind every trained model.
This is a sample: 1 of 13 chapters.
On our notation and higher-order Jacobians. We only consider vector-valued quantities for readability, as all resulting gradients are then matrices. In practice, existing primitives may have inputs, weights or outputs of higher rank.
Consider the first instruction in our program, 𝐡₁ = f₁(𝐱, 𝐰₁). Because nothing has been stored up to now, we initialize the tangent matrix for 𝐰₁ as its weight Jacobian: Ŵ₁ = ∂_𝐰₁ 𝐡₁. We now proceed to the second instruction, 𝐡₂ = f₂(𝐡₁, 𝐰₂).
We can put these observations together to develop a second approach to automatic differentiation, which we call reverse-mode automatic differentiation, or R-AD, outlined next.
Hence we can implement an entire automatic differentiation system by first choosing a set of primitive operations, and then augmenting them with the corresponding VJPs, without having to materialize the Jacobians in memory at any point.
As we mentioned, in practice our models are implemented as compositions of objects whose parameters are encapsulated as properties. One possibility is to “purify” the object, turning it into a pure function.
Subdifferentiability and automatic differentiation. There is a small detail we avoided discussing until now: the ReLU is not differentiable at 0, which makes the overall network non-smooth. What happens in this case?
Examples for: a curious adult with school maths who uses AI apps every day; add one modern instance per idea, such as how a chat assistant or a phone's photo app is trained
The lectures retell the English text of arXiv 2404.17625 (version 3, 2025) by Simone Scardapane, published under an open licence that allows retelling and translation. The retelling, the drawings and the voice are made by AI and checked against the book by a second model. How AI is used.
The lectures adapt the original and are released under the same licence, free to share and adapt with credit to the author and a link to this page: CC BY-SA 4.0
Upload a textbook or any difficult book. The first five minutes are free.
Upload a book