Back to basics: The Hundred-Page Languages Models Book, Chapter 1
July 22, 2026
As I wrote in my earlier entry, I left my first foray into open-source LLMs with the strong conviction that I need to go back to basics and understand what is happening under the hood with these models. When I was in grad school in the early 2000s, we studied neural networks mostly as a historical curiosity, an evolutionary dead end that had been entirely evicted from use by statistical machine learning. When deep learning burst onto the scene in the 2010s, the idea that bigger models, bigger training data, more computation will combine to yield dramatically better results felt self-evident enough that I didn't bother trying to understand it more, well, deeply.
So maybe it's embarrassing to admit this, but I never really bothered to update my mental model of LLMs beyond my (already cursory) understanding of deep learning. I did at least wrap my head around the colloquial concept of embeddings and the idea that you can represent semantic similarity between words in some machine-readable way, but I didn't go any further than that. I definitely did not do the work of understanding the math.
But now, my goal is to be able to read a HuggingFace model card, and in order to do that, I have to learn all of the vocabulary of LLM architectures and hyperparameters.
I've been steadily working my way through Andriy Burkov's phenomenal The Hundred-Page Languages Models Book. I'm finding it very easy to read, and as long as I force myself to go slowly, and especially if I make sure that I am understanding all of the sample code snippets provided, the concepts that the book presents feel like they are resting in my brain in a way that feels like I'll actually retain them over time.
Chapter 1 – Machine Learning Basics
Just like it says on the box, the first chapter of the book is a review of machine learning basics, written for an audience that is being exposed to these ideas for the very first time. Since I am not encountering machine learning for the very first time, I was able to go through most of the chapter quite quickly. For me, it got interesting when moving from generic classification algorithms to neural networks in particular.
The first bit of novel vocabulary that I encountered is the difference between a dense neural network, one in which each perceptron in a given layer is connected to every perceptron in the subsequent layer, and a convolutional neural network, in which the layers are not fully connected. The author notes that convolutional networks were originally designed for image processing, but have proved useful for certain kinds of text processing such as document classification. For me, this is interesting because one of the first bits of unfamiliar vocabulary that I encountered when browsing models in HuggingFace is that some LLMs are described as “Dense” whereas others are “Mixtures of Experts”.
I'm trying not to skip ahead, so I haven't gone looking for an explanation of why a convolutional network might perform better in some domains or applications than a dense network. I can see the biological analogy; our brains are composed of clusters of neurons that are densely connected to their close neighbors, but only loosely connected to other clusters. I can imagine an image processing network working best when allowed to develop similar clusters of expertise, for example a set of neighboring perceptrons that together perform edge detection and another cluster that handles perspective and scale. I don't really understand why a convolutional network would be better for document classification, though, and I wonder whether the clusters need to be trained independently, or whether “experts” appear emergently, given large-enough training data and a convolutional network with enough subneighborhoods.
The other novel concept for me from this chapter is the RELU activation function. RELU, or rectified linear unit, is the function:
RELU(x) = max(0, x)What RELU does is take an input and truncate it at 0, so that the output is always positive. Apparently, RELU has been the predominantly-used activation function for neural networks since 2012. I genuinely can't fathom why. When I first studied neural networks, the default activation function was the sigmoid, which makes perfect intuitive sense if you imagine each perceptron as a baby classifier. RELU makes no sense to me at all. However, the book promises that we will come back to RELU in Chapter 4 – Transformers, so I am, for now, suspending disbelief.