Lecturized
RU
Библиотека · CC BY-SA 4.0

Приключения Алисы в Странедифференцируемых чудес

Симоне Скардапане · 2024 · пересказ с английского оригинала

Alice's Adventures in a Differentiable Wonderland, Volume I: A Tour of the Land

Иллюстрация к книге

Выберите версию

Один и тот же текст на каждом уровне, на всех языках и во всех оформлениях, что здесь есть. Выберите, потом слушайте или скачивайте.

ЯзыкАнглийскийРусскийИспанский
Уровень1 · Как в книге2 · Формулы с объяснением3 · Формулы словами4 · Идеи и примеры5 · Общая картина
ГолосМужской · Device
РисункиГрафика тушьюБелым по чёрному

Слушать и смотреть

Тот же плеер, который продукт даёт для вашей книги: голос, рисунки по мере того, как они рисуются, текст.

Современный учебник о том, как на самом деле устроены нейросети: от тензоров и градиентов до трансформеров. Глава 6 показывает, как программа сама находит свои градиенты: автоматическое дифференцирование, обратный проход, на котором обучается любая модель.

Это пробная часть: 1 из 13 (главы).

Из лекций

A small stack of picture frames feeds a grid of reactions that grows from a few cells into an enormous table filling the canvas, marked too big.
A Recipe Made of Small Steps

A Table Too Big to Keep

Some steps have a sharp corner where the slope is not defined, such as the popular ReLU, which lets positive numbers through and turns negative ones into zero.

Four stations in a row; under each a growing stack of small tables, one under the first station, two under the second, three under the third and four under the fourth, each new stack updated by the station's input Jacobian.
Why the Obvious Ways Are Too Slow

Forward Mode Keeps Tables

So the book turns to the first genuine method, forward mode, and it rests on the chain rule. If one station feeds another, the reaction of the pair is the reaction of the second multiplied by the reaction of the first.

Four blocks in a row, the first, second and fourth marked as kept checkpoints and the third left empty; the backward arrows stop at the third block, a rerun arrow goes from the second block to the third, and a label says plus a quarter.
Walking Back From the Score

Checkpoints Save Memory

The book shows a way to trade some of that memory back for time, called gradient checkpointing. Picture four blocks in a row. On the forward pass, only the outputs of the first, second and fourth blocks are kept; these are the checkpoints.

A layer joins three inputs, valued 2, 1 and 3, to an output that matters by 4; the three links show their gradients 8, 4 and 12, each the output's importance times the input.
Asking the Table One Question

A Weight's Share of Blame

Why is this so much better? Take the most common station of all, a layer that mixes its inputs with a grid of weights, as every layer of a chat model or a photo model does many times over.

A row of copies of a page, each half as wide as the one before, shows a gradient shrinking by a factor of a quarter per layer, falling below one millionth after ten layers.
How Frameworks Walk Back, and Why ReLU Won

Factors That Multiply

Now back to the photocopier. Remember that when the backward walk passes an activation function, the travelling row is scaled by the function's slope. A deep model has many such stations, so the row is scaled again and again, and the factors multiply exactly like the copier's reductions.

The ReLU, flat to the left of zero and rising at 45 degrees to the right, with a fan of lines through the corner whose slopes run from 0 to 1, the flat one marked as the usual choice; above, three ways of writing ReLU give slopes 0, 1 and 0.5 at zero.
The Corner of the ReLU

Every Slope From 0 to 1

The careful answer uses the idea of the ruler. For a function that curves upward like a bowl, any line that touches it at a point and stays below the whole function is a subgradient there, and its slope is a valid substitute for the missing slope.

Как это сделано

Примеры для: a curious adult with school maths who uses AI apps every day; add one modern instance per idea, such as how a chat assistant or a phone's photo app is trained

Источник

Лекции пересказывают с английского оригинала книгу arXiv 2404.17625 (version 3, 2025), автор — Симоне Скардапане; книга вышла под открытой лицензией, которая разрешает пересказ и перевод. Пересказ, рисунки и голос сделаны ИИ и сверены с книгой второй моделью. Как используется ИИ.

Лекции — переработка оригинала и распространяются под той же лицензией: их можно свободно распространять и перерабатывать с указанием автора и ссылкой на эту страницу: CC BY-SA 4.0

Ваша трудная книга — так же

Загрузите учебник или любую сложную книгу. Первые пять минут бесплатно.

Загрузить книгу