Karpathy: Transformer Architecture Remains King

The 2017 landmark paper, "Attention Is All You Need," marked a significant departure from incremental research focused on Recurrent Neural Networks (RNNs). Instead of building upon existing RNN structures, the paper introduced a novel architecture that entirely eliminated RNNs, placing attention mechanisms at its core. This architecture proved to be remarkably resilient and has since become the standard for modern Large Language Models (LLMs).

Most of the original Transformer structure, first introduced in 2017, remains largely unchanged in today's sophisticated Generative Pre-trained Transformers (GPTs). Hyperparameters, such as the 4x expansion factor within the feed-forward networks, have consistently proven effective over time. Attempts to modify detailed aspects of the Transformer, such as its internal mechanisms or scaling factors, have largely failed to produce superior results.

Andrej Karpathy conceptualizes attention as a communication phase within a directed graph, interleaved with a computation phase. In this framework, the communication phase is handled by multi-headed attention, allowing different parts of the network to exchange information. The computation phase then involves multi-layer perceptrons (MLP) that process this information locally within each node. Each node in the Transformer utilizes its private data to generate Query, Key, and Value vectors, which facilitate this dynamic communication.

Multi-headed attention is simply the parallel application of the same attention scheme multiple times, each with different learned weights. This allows the model to capture diverse relationships and focus on different parts of the input simultaneously. Self-attention, a specific form of attention, generates its Key and Value vectors from the input sequence itself, enabling tokens within the sequence to relate to each other.

Andrej Karpathy introduced NanoGPT as a pedagogical tool designed to demystify the complex workings of GPT-2. This minimalist implementation is compact, spanning approximately 300 lines of code, making it highly readable and accessible for educational purposes. Despite its brevity, NanoGPT is capable of reproducing GPT-2's behavior on the OpenWebText dataset.

Transformers cannot process raw text directly; they require numerical representations of input data. This necessitates the encoding of text into integer sequences through tokenization and embedding. Characters or subword units are converted into unique integers, forming a numerical sequence that the Transformer can understand.

Block size defines the maximum contextual window a Transformer can process, meaning it dictates how many previous tokens are considered for predicting the next one. Batch size determines how many such sequences are processed simultaneously, significantly impacting parallelization efficiency. The effective batch size is a product of batch size and block size, reflecting the total number of tokens processed in parallel at any given time.

Transformer blocks are structured to alternate between distinct communication and computation phases. During the communication phase, facilitated by self-attention, tokens within the sequence can exchange information and establish relationships with each other. This allows the model to integrate contextual understanding across the input. Following this, the computation phase involves a Multilayer Perceptron (MLP), which processes each node independently based on the aggregated information from the communication phase.

The fundamental distinction between encoder and decoder attention models lies in the application of masking. Decoder models employ masking to prevent tokens from accessing information about future tokens in a sequence, ensuring that predictions are made based only on preceding context. Removing this mask allows all tokens to communicate freely with each other, effectively transforming the architecture into an encoder.

Andrej Karpathy asserts that Transformer architectures maintain static connectivity patterns, independent of the input data. He notes that he has not observed any examples where the network's connections change dynamically based on the data it processes. For encoders, the connectivity is typically fully connected, allowing all tokens to interact with each other, while decoders employ a triangular or causal connection pattern to prevent future tokens from influencing current predictions.

Andrej Karpathy highlighted the historical irony surrounding the original Transformer paper, noting that its authors were largely unaware of the monumental impact their architecture would eventually have. The seminal paper, released in 2017, was framed primarily as a contribution to machine translation research. At the time of publication, the profound scalability, versatility, and universal applicability of the Transformer architecture were not fully anticipated.

Andrej Karpathy pointed out the somewhat 'ridiculous' yet effective method of applying Transformers across various domains, often involving converting input data into 'patches' or small segments. For Vision Transformers (ViT), images are segmented into small squares, which are then fed into the Transformer. A similar slicing logic is applied to audio spectrograms for speech processing and even to reinforcement learning trajectories.

Andrej Karpathy attributes the Transformer's remarkable success to its simultaneous optimization of three crucial properties: high expressivity, optimizability, and extreme hardware efficiency. The architecture's design allows it to represent complex functions effectively during the forward pass. Its structure, including residual connections and layer norms, makes it highly optimizable, facilitating stable and efficient training.

Recurrent Neural Networks (RNNs), despite their theoretical ability to implement arbitrary programs, suffer from significant practical limitations due to serial processing bottlenecks. This sequential nature makes them difficult to optimize and highly inefficient for modern hardware. In contrast, Transformers provide shallow and wide compute graphs, which are ideal for efficient gradient flow during training.

Transformers handle multimodal data by converting diverse input types into a unified tokenized format. Images, for instance, are divided into patches, which are then treated as individual tokens. Similarly, other data types like radar signals or audio are segmented and encoded. Special learnable embedding tokens are often used to identify the source of the data, allowing the Transformer to differentiate between image, text, or other modalities.

Andrej Karpathy proposed a method to overcome the fixed context length limitations of Transformers by allowing the network to utilize an external 'scratch pad.' This approach involves keeping the context length fixed but teaching the Transformer to interact with a supplementary memory area. The network would learn, through examples embedded in the prompt, to read from and write to this scratch pad using special tokens.