Neural Networks: Zero to Hero

Overview

We start with the basics of backpropagation and build up to modern deep neural networks, like GPT. In my opinion language models are an excellent place to learn deep learning, even if your intention is to eventually go to other areas like computer vision because most of what you learn will be immediately transferable.

This is why we dive into and focus on languade models.Prerequisites:

solid programming (Python), intro-level math (e.g. derivative, gaussian).

Syllabus

• The spelled-out intro to neural networks and backpropagation: building microgradThis is the most step-by-step spelled-out explanation of backpropagation and training of neural networks. It only assumes basic knowledge of Python and a vague recollection of calculus from high school.
• The spelled-out intro to language modeling: building makemoreWe implement a bigram character-level language model, which we will further complexify in followup videos into a modern Transformer language model, like GPT. In this video, the focus is on (1) introducing torch.Tensor and its subtleties and use in efficiently evaluating neural networks and (2) the overall framework of language modeling that includes model training, sampling, and the evaluation of a loss (e.g. the negative log likelihood for classification).
• Building makemore Part 2: MLPWe implement a multilayer perceptron (MLP) character-level language model. In this video we also introduce many basics of machine learning (e.g. model training, learning rate tuning, hyperparameters, evaluation, train/dev/test splits, under/overfitting, etc.).

---

Taught by

Andrej Karpathy

---

Subjects

Artificial Intelligence