Stanford Details LLM Architecture & Training

By leveraging the multi-head attention layer from the encoder only, BERT introduced the CLS token to capture sequence-wide context, facilitating downstream tasks like sentiment extraction without full model retraining.

This architecture strategically utilizes the multi-head attention layer from the encoder to enable self-attention across all input tokens, allowing them to attend to one another bidirectionally.

BERT also introduced the CLS (classification) token, a special input token designed to capture aggregated contextual information from the entire input sequence, which proved crucial for various downstream tasks like sentiment extraction without requiring full model retraining.

To learn document-level coherence and force the deduction of corrupted tokens, the model undergoes Masked Language Modeling and Next Sentence Prediction as a one-time, computationally intensive pre-training step.

MLM involves randomly masking a percentage of tokens in the input and training the model to predict the original, corrupted tokens based on their context, thereby forcing a deep understanding of token relationships.

NSP, on the other hand, trains the model to understand coherence at the document level by predicting whether two sentences naturally follow each other.

While pre-training is a computationally intensive, one-time process, the resulting pre-trained model can be efficiently adapted to specific downstream tasks with relatively small labeled datasets through fine-tuning.

The BERT framework, with its reliance on bidirectional self-attention mechanisms, delivered strong performance across various natural language understanding tasks.

Its design allowed for efficient fine-tuning, often requiring only hundreds of labeled data samples, which significantly reduced the effort for adapting the model to new tasks.

However, fine-tuning still represented an additional engineering effort, and generating text using an encoder-only model like BERT was not as straightforward as with decoder-based architectures.

Test loss serves as the primary metric for model quality, as 2020 research established that performance gains remain predictable when scaling training data and parameter counts.

These studies showed that larger models exhibit higher token efficiency, meaning they achieve superior results with a comparatively smaller number of training tokens.

Performance is typically measured via test loss, where lower values signify a better quality and more accurate model output.

FLOPs, or floating-point operations, measure the total volume of computation performed by a model.

Conversely, FLOPS (with a capital 'S') or FLOPs/s denotes the performance capacity or speed of the underlying hardware, representing how many floating-point operations can be executed per second.

These distinctions are crucial for understanding the computational demands and capabilities involved in training and running large language models.

While later training stages may prioritize data quality, the field currently considers raw data volume the most impactful factor for the pre-training phase, the instructor notes.

Training regimes prioritize a larger, noisy corpus over a smaller, perfectly curated dataset during the initial learning stages.

However, the instructor notes that later stages of the training process, specifically discussed in a future lecture, may introduce a shift where data quality becomes a more critical factor.

In decoder-only Transformer architectures, the multi-head attention layer primarily facilitates communication between tokens, while feed-forward networks are typically where meaningful embeddings are computed.

Mixture-of-Experts (MoE) integrates a router into the feed-forward network, which, for a given input, decides which expert (a specialized feed-forward neural network) to activate, thereby handling compute-intensive operations more efficiently.

Routing collapse is a significant challenge in training Mixture-of-Experts (MoE) models, where the model disproportionately relies on a small subset of experts, diminishing its effective capacity.

This phenomenon occurs because early activation of certain experts leads to more frequent updates and stronger weights, creating a feedback loop where these 'richer' experts are increasingly favored.

To counteract this, auxiliary loss functions, such as load balancing auxiliary loss, are introduced to incentivize a more uniform distribution of tokens across all available experts.

These routing decisions, crucial for expert utilization, are made at the individual token level rather than at the sentence level, ensuring granular control over expert activation.

The self-attention mechanism within Transformers inherently causes a loss of positional information, making explicit positional encoding necessary.

The original Transformer paper explored two main approaches to address this: learned position embeddings, which assign a unique vector to each index and add it to the token embedding.

However, learned embeddings suffer from a significant drawback: they cannot generalize to sequence lengths that exceed the training data, limiting their applicability to novel, longer inputs.

To overcome this limitation, the paper also introduced a deterministic, formula-based method for positional encoding, which offers better generalization across varying sequence lengths.

The core concept behind deterministic positional encoding is to represent each position in a sequence as a high-dimensional vector, akin to the multi-hand mechanism of a watch.

Each 'hand' or component within this vector operates at a distinct frequency, with lower indices corresponding to higher frequency components.

Conversely, higher indices are associated with lower frequency components, causing their sine and cosine values to vary more slowly across positions.

This intricate design ensures that positions with similar characteristics yield high similarity scores when their dot products are computed, allowing the model to discern relative positions effectively.

This design ensures that when 'm' is equal to 'n', the sum of the dot product is maximized, indicating the highest similarity between identical positions, thereby allowing the model to interpret proximity effectively.

However, despite its mathematical soundness, this specific original method for positional encoding is not the standard implementation used in most modern large language models today.

The original method for positional encoding, though innovative, is not widely adopted in today's large language models.

Modern architectural approaches increasingly favor applying positional information surgically to only the attention layer, rather than injecting it across all Transformer layers.

This shift stems from the understanding that the attention layer is the primary site of token-to-token communication and where positional biases can have the most direct impact.

By injecting positional bias specifically into the attention layer, models can prioritize interactions between closer, more relevant tokens while still maintaining the capacity for broader contextual connections.

The current industry standard for positional encoding is Rotary Position Embedding (RoPE), which fundamentally differs from previous methods like ALiBi or T5 bias.

RoPE operates within the attention layer by directly modifying the query and key vectors through rotation, with the rotation amount being a function of their respective positions.

This rotation ensures that the dot product between queries and keys inherently becomes a function of the difference between their positions (n minus m), effectively encoding relative positional information.

A key advantage of RoPE is its extensibility to any position without the need for learning new biases, providing a robust and flexible solution for positional encoding.

By restricting tokens to a limited range, sliding window attention seeks to decrease the heavy computational load typical of global attention mechanisms.

This approach significantly reduces the heavy computational burden associated with full global attention, where every token interacts with every other token in the sequence, by narrowing the scope of interactions.

Despite the localized constraints imposed by mechanisms like sliding window attention, stacking multiple attention blocks enables information to propagate further within the model than a single window would allow.

Deep layers allow tokens to indirectly interact with information from distant sequence positions, despite individual layers being limited to a local context.

The behavior of these stacked local attention layers effectively mimics the accumulation of receptive fields observed in deep convolutional neural network architectures.

Consequently, while local attention reduces immediate computational cost, the overall model can still build a comprehensive understanding of distant dependencies through its multi-layered structure.

Older Transformer architectures commonly employed Post-Normalization, applying normalization after sub-layer operations, which could interfere with the direct input to subsequent layers.

Modern architectures have largely shifted to Pre-Normalization (Pre-Norm), where inputs are normalized before entering a layer, ensuring that residual connections can function more effectively.

This change in normalization strategy is crucial for improving gradient flow and the overall stability of training in very deep neural networks, leading to more robust model performance.

Greedy search, a common decoding strategy, selects the most probable next token at each step, but this localized optimization often fails to find the globally optimal sequence.

Beam search improves upon greedy search by tracking multiple (k) potential paths, yet it remains a deterministic method, still prone to generating predictable outputs.

For applications like chatbots, deterministic generation is problematic because it lacks the human-like variation and spontaneity required for engaging interactions.

Shervine Amidi notes, "you might feel that the chatbot is robotic and robotic chatbots I don't know about you but it's not so appealing because it feels like it's still heavily engineered."

A temperature setting below one sharpens the distribution to emphasize high-probability indices, while a setting above one flattens it for more uniform token selection.

When T > 1, the distribution flattens, leading to a more uniform selection among available tokens and thus increasing the randomness or creativity of the output.

Conversely, when T < 1, the distribution sharpens, emphasizing higher-probability indices and making the model's output more focused and deterministic.

Afshine Amidi notes that while a temperature of zero might imply complete determinism, practical LLM inference engines often batch requests, introducing slight variations.

Standard large language model requests for structured data, such as JSON, frequently lack 100% reliability, making downstream parsing unpredictable and often prone to errors.

Guided decoding is a technique designed to overcome this by restricting the span of available tokens at each step of the generation process, based on the syntactic rules of the desired output language.

This method ensures that the model's output adheres strictly to the specified structural rules, guaranteeing valid format generation and eliminating parsing ambiguities.

Afshine Amidi explains the concept as one where "the span of available tokens to be predicted at the next step is restricted depending on the syntax of the language you want to decode this in."

Needle-in-a-haystack tests demonstrate that retrieval performance suffers as context volume grows, leading advocates to recommend context compaction or new session starts to mitigate noise.

Needle-in-a-haystack tests consistently demonstrate that models experience greater difficulty retrieving specific pieces of information when they are embedded within a longer, more complex context.

A higher volume of context introduces additional noise, making it harder for the model to isolate and focus on the relevant information.

To mitigate this, strategies such as 'context compaction' and initiating new conversational sessions are advocated to maintain optimal information retrieval performance.