A large language model, exemplified by Llama 2 70B, fundamentally consists of two files: a large parameters file (e.g., 140GB for 70B parameters stored as float16) containing the neural network's weights, and a smaller code file (like run.c) that executes these parameters. This self-contained package allows for offline operation on standard hardware like a MacBook, though performance varies with model size. The true 'magic' resides in the parameters, not the execution code.
Training LLMs: Compressing the Internet
Obtaining the LLM parameters involves a computationally intensive training process, akin to lossy compression of vast internet text (around 10TB). This requires massive GPU clusters (e.g., 6,000 GPUs for 12 days for Llama 2 70B), costing millions. The resulting parameters, though much smaller than the original data, encapsulate a 'gestalt' of the internet's knowledge, enabling next-word prediction.
From Internet Sampler to Assistant: Fine-Tuning
Base LLMs, trained on internet data, are primarily 'internet document generators.' To become helpful assistants, they undergo fine-tuning. This involves training on curated, high-quality Q&A datasets, often created by human labelers following specific instructions. This process aligns the model's behavior, teaching it to respond helpfully to prompts, while retaining the vast knowledge acquired during pre-training.
Tokenization, the process of converting text into numerical tokens for LLMs, is a fundamental yet often problematic stage. Many LLM issues, from spelling errors to difficulties with non-English languages and arithmetic, can be traced back to the design and limitations of tokenizers. The speaker expresses a strong personal dislike for this component, highlighting its 'hairy' and 'gnarly' nature, and hopes for future tokenization-free models. This foundational step is crucial for understanding LLM behavior.
The Tiktokenizer Demo: Visualizing Tokenization Quirks
The Tiktokenizer web app visually demonstrates how different tokenizers, like GPT-2 and GPT-4, break down text. It reveals arbitrary tokenization of numbers, case sensitivity issues (e.g., 'egg' vs. 'Egg'), and inefficiencies in handling non-English languages and Python code (due to spaces). The GPT-4 tokenizer shows improvements, particularly in representing Python indentation more efficiently, by using larger tokens for whitespace sequences. This visualization underscores the non-intuitive nature of tokenization.
The Byte Pair Encoding (BPE) Algorithm Explained
The Byte Pair Encoding (BPE) algorithm addresses the limitations of raw byte sequences by compressing them into a more manageable vocabulary. It works by iteratively identifying the most frequent pair of adjacent tokens (initially characters or bytes) in the corpus and merging them into a new, single token. This new token is added to the vocabulary, and the process repeats, effectively creating a hierarchy of tokens that represent common sub-word units. This compression allows for a larger effective vocabulary while keeping sequence lengths reasonable.
The primary objective is to reproduce the GPT-2 124 million parameter model from scratch, leveraging insights from both the original GPT-2 and GPT-3 papers. This involves understanding the architecture, training process, and hyperparameters to achieve comparable or better performance. The speaker aims to build confidence by first loading the official OpenAI weights and then training a model from random initialization.
Karpathy: Deconstructing the GPT-2 Architecture
The GPT-2 architecture is a decoder-only Transformer, differing from the original Transformer by omitting the encoder and cross-attention. Key modifications include the placement of layer normalizations (pre-normalization) and an additional layer norm before the final classifier. The implementation mirrors the structure used by Hugging Face Transformers to facilitate weight loading.
Karpathy: Loading Hugging Face Parameters
To verify the custom GPT-2 implementation, parameters from a Hugging Face pre-trained model are loaded into the newly created module. This process involves careful mapping of variable names and, in some cases, transposing weights to match PyTorch's expected format, confirming the code's fidelity to the original model.
Andrej Karpathy: The Internet as an LLM Training Ground
The foundational step in training Large Language Models like ChatGPT involves processing a massive dataset, primarily sourced from the internet. This data, curated through processes like those used for the FineWeb dataset, aims for quantity, quality, and diversity. It involves extensive filtering to remove undesirable content, extract clean text from raw HTML, classify languages, and deduplicate information, ultimately resulting in terabytes of text data.
Andrej Karpathy: The Inner Workings of a Neural Network
Neural networks, while conceptually simple as mathematical functions transforming inputs to outputs, are parameterized by millions or billions of weights. These 'neurons' are far simpler than biological ones and operate stateless-ly, with information flowing through them to generate predictions. The core process involves adjusting these parameters to align model predictions with patterns found in training data.
Andrej Karpathy: The Art of Inference
Inference is the process of generating new data from a trained model. It begins with a prefix of tokens, which are fed into the network to produce a probability distribution for the next token. By sampling from this distribution, a token is selected, appended to the sequence, and the process repeats. This stochastic sampling means generated text is a remix of training data, not a verbatim reproduction, leading to unique outputs.
Large Language Models are akin to a one-terabyte zip file containing compressed knowledge from the internet, learned roughly six months ago. Their 'personality' is shaped by human-labeled conversations during post-training, making them probabilistic and potentially outdated recallers of information.
Context Window Management
The conversation history, or 'context window,' is a precious resource. Users should start new chats when switching topics to avoid distracting the model and incurring unnecessary costs, thereby improving performance and accuracy.
The Evolution of LLM Training
LLMs undergo a multi-stage training process: pre-training, supervised fine-tuning, and reinforcement learning (RL). RL is crucial for developing 'thinking strategies' that mimic human problem-solving, leading to improved performance on complex tasks like math and code.
![[1hr Talk] Intro to Large Language Models](/_next/image?url=https%3A%2F%2Fi.ytimg.com%2Fvi%2FzjkBMFhNj_g%2Fhqdefault.jpg&w=3840&q=75)
