Build a Neural Network From Scratch: Training, Fine-Tuning, and RL (Part 1 of 2)

Two Colab notebooks accompany the session; copy them to your own Drive and run them at your own pace: Notebook 1 and Notebook 2.

Why engineers have to become statisticians

Aaron rarely says “AI” when he teaches. It’s a big, general concept, and what’s actually running under the hood is a machine learning model — an LLM is just one type. The explosion of LLMs into everyday software work has forced everyone to become an applied statistician whether they realize it or not, because a core piece of the workflow now depends on a statistical model rather than a deterministic algorithm. Run a program twice and you expect the same result; ask a model twice and you may not. Most software engineers never studied statistics, and it’s a field worthy of its own study. The goal of this session isn’t to make you a statistician or to build production models from scratch — it’s to make you an informed consumer who knows when to pick which model, what guardrails to put up, and what to measure.

Meet the cassowary: what learning is

Aaron asks the room to picture a cassowary from the word alone, then shows one — a dinosaur-like, ostrich-shaped bird with a bony casque on its head, blue neck, red accents, and a black body (advice from chat: don’t go near one). Now picture it again. You just learned from training data. Machine learning is loosely inspired by that, not the other way around, and we still don’t fully understand neuroscience. What ML actually is: the application of statistical principles to real-world data via algorithms, because no one can whiteboard a closed-form formula for messy, noisy, real-world data. Every ML model has two steps, training and inference, and this session focuses on supervised learning — where you have labeled data, like pictures tagged “cassowary” or “cat” — because that’s where almost all the action is. None of this is new to LLMs. Machine learning has mattered for decades; what changed is scale.

Attention, transformers, and why scale keeps winning

The last big conceptual breakthrough was Google’s 2017 paper “Attention Is All You Need,” which laid the foundation for the transformer architecture — an accessible paper Aaron recommends reading. The transformer isn’t necessarily the “best” architecture; it’s the most trainable, the one that scales horizontally across compute. And an okay architecture with enormous data beats a theoretically better one you can only train small. That’s the recurring lesson: the answer is rarely a brilliant new algorithm and usually more data and cheaper chips. Whether that keeps holding is a crystal-ball question, and there are arguably diminishing returns already, but bigger models are still better. Aaron also notes that binary classifiers — spam or not, fraud or not — are probably still the most-used models on the planet by inference calls, even in the age of LLMs.

A model is a function

Remember y = mx + b? The x is what you know, the y is what you want to know, and the function turns one into the other. Machine learning is function approximation: there’s some ideal function that maps email headers to “spam or not,” and we learn the best approximation we can from labeled examples. In the notebook, a linear regression fed inputs 0–6 through a hidden function recovers a slope of 4 and an intercept of 5 — exactly 4x + 5 — using linear algebra on a matrix of those pairs. Add random noise (instrument bias, the noise of society) and the fit is close but not perfect, and the plot shows red predictions drifting from blue truth. That gap is the error, or residual, and minimizing it is the optimization problem every ML model solves. Constrained to a line, the model gets very close to five points at the cost of two; an LLM is doing the same thing with billions of parameters in billion-dimensional space, finding the weights that make predictions closest to the training data.

Why training data has to be balanced

If you train a spam detector only on spam, it will learn to say everything is spam — the model is trying to “win” with the data you gave it. You need a control group, the way a medical study needs varied demographics and regions rather than proving a treatment works for college students in Southern California. Balanced, representative data is the job, and in any real ML project almost all the work is collecting, curating, cleaning, and labeling it. Training the model is clicking a button and paying for compute. A perfect function that generalizes to unseen data is generally unrealistic; how much accuracy you need depends on the stakes, and a spam filter can afford mistakes a cancer detector can’t.

Logistic regression and the decision threshold

Linear regression is mostly a teaching and history model now, but logistic regression is still useful: it constrains the output to the unit interval, 0 to 1, which makes it a solid MVP for spam, fraud, churn, and other binary classification. Since the output is a probability, you set a decision threshold — naively 0.5 — and that threshold becomes a tunable dial for how confident the model must be before it commits. The notebook’s churn example makes the feature lesson concrete: predicting churn from how many times a customer coughed on calls is no better than random, while predicting it from net promoter score is accurate. Use the features that are relevant. That simple model is already close to something real — the “we’d love to keep you as a customer” email you’ve received may well have been triggered by exactly this kind of churn model.

Correlation, causation, and looking at your data

Net promoter score correlating with churn doesn’t mean it causes churn. Aaron’s favorite example: rising global ice cream consumption correlates with the decline of piracy in the Caribbean, and both are really driven by refrigeration, maritime law enforcement, and history — ice cream does not prevent pirates. A viewer asked what exploratory data analysis techniques help find complex features that don’t obviously contribute to accuracy. Aaron’s answer: the instinct to look at your data first is the right one; visualizations help but break down past two or three dimensions, so use dimensionality reduction (eigenvalues, violin plots, and the rest) — and talk to people in the domain. A lot of statistics is formalizing what practitioners already know tribally, and staying humble about that is part of being a good engineer. Causal and structural inference tools exist for the weak-correlation problem too; statisticians have been working this space for a long time.

The Python toolkit: NumPy, pandas, scikit-learn, PyTorch

Statistics happens in Python (a few holdouts still use R). NumPy represents matrices and, like any good Python library, shells out to Fortran and C primitives for speed — built-in lists won’t scale. pandas gives you data frames, essentially programmable spreadsheets with typed columns and named rows, and once pandas landed people stopped needing R. scikit-learn is a huge, well-documented collection of models with built-in datasets (the notebook fits a model on the classic iris flower data); it’s used more for teaching than production, though it can ship. For scale you go to PyTorch (from Facebook) or Google’s TensorFlow. The notebook builds a roughly 100-line feed-forward neural network in PyTorch — flatten, linear layer, ReLU, linear layer — and it takes about a minute to train because there are far more parameters. Each neuron is a small math function, like a logistic regression; layers combine them; and a one-megabyte cassowary picture is a million numbers, which y = mx + b can’t capture and a bigger model can.

Why we can build models we can't explain

A viewer noted that black-box solutions are cutting edge right now, and Aaron agrees: our ability to make models currently outruns our ability to explain them. His analogy is ancient architecture. Pyramids, aqueducts, and the Great Wall were huge, but nobody built a Burj Khalifa — not for lack of an emperor who wanted one, but because building that tall requires math and physics they didn’t have. They knew what stood up without always knowing why. Machine learning, LLMs very much included, is in that state today.

Representations, bits, and where embeddings come from

Early AI tried symbolic reasoning and triples (“John is father of Tom”), which still has value and may see a resurgence, but writing down the whole world that way can’t keep pace with observationally collecting data from computers, phones, and sensors and feeding it to a learning algorithm. Everything digital reduces to zeros and ones, and a one-megabyte file has two-to-the-million possible values — a number that dwarfs the atoms in the universe — which is why wrapping a model around that space takes billions or trillions of parameters. Inside a neural network, the early layers compress a million pixel numbers down toward a single output, and along the way they extract the structure of what makes a cassowary a cassowary. That intermediate representation — often the penultimate layer, before the final label — is the embedding. Stored as tensors, these vectors carry semantics: the famous demo is that the vector for king plus the vector for woman lands near the vector for queen. It’s a snapshot of what the model “thinks” before it speaks, with the usual caution about anthropomorphizing.

Q&A: activation functions and hyperparameter search

Asked what experiments decide which activations to use between layers, Aaron draws the line between parameters (the inscrutable numbers inside the model that training learns) and hyperparameters plus architecture (the design decisions humans make). PyTorch offers ReLU and its variants (ReLU6, leaky ReLU), sigmoid, tanh, softmax, and more. None is universally best; when in doubt he starts with ReLU because it performs well and is efficient, and speed and ease of training are real criteria too. The real answer is to sample your data representatively, train multiple models with different configurations, run a hyperparameter search, and let the experiment decide — the optimum depends on your data.

Get the reps in

Aaron’s closing advice: you won’t retain any of this from watching him move fast through the notebooks. Go back, run them slowly, change the noise scale, swap models, try a function like y = x² that linear regression can’t learn, and type it yourself. Part 2 continues in two weeks with two more notebooks, and the next Gauntlet cohort starts September 14, with a live Q&A the following day for program questions.