Learning ML: 10 Core Concepts for Engineers

Training involves the model adjusting its internal numerical parameters, known as weights, to identify and fit patterns within the provided data. This phase is where the model learns from historical information.

Conversely, inference is the stage where the model, with its weights already fixed or 'frozen,' receives new input and subsequently generates outputs. It is important to note that models do not continue to learn from real-time user input during the inference phase; their learning is confined to the training period.

A 2026 AI index from Jamfer indicated that the cost of inference for models of GPT-3.5 class quality became approximately 280 times cheaper over a span of two years.

This reduction in cost coincides with a substantial increase in infrastructure, as total AI data center capacity climbed to around 29.6 gigawatts (GW). This capacity is comparable to the peak electrical load of an entire state, such as New York, highlighting the immense scaling of AI operations.

A core concept in machine learning training is the 'loss number,' a single numerical value that quantifies how incorrect the model's predictions are. This metric is essential because it provides a clear target for improvement.

Gradient descent is an optimization algorithm that uses this loss number to guide the model's learning. For each weight within the model, gradient descent determines the direction that would make the loss number smaller and then subtly adjusts the weight in that direction. This iterative process gradually minimizes the model's overall 'wrongness.'

It is crucial to understand that this mechanism operates purely mathematically; the model does not possess inherent understanding, intention, or awareness of what it is doing, but merely follows the gradient to reduce error.

Overfitting describes a scenario where a machine learning model learns the training data excessively well, to the point of memorizing it. This leads to poor performance on any data outside of the training set, as the model cannot apply its 'knowledge' to new, unseen examples.

In contrast, generalization refers to a model's ability to make accurate predictions or draw valid conclusions from new, previously unobserved data, such as a test set. This capability indicates that the model has learned underlying patterns rather than simply memorizing specific instances. A common strategy to mitigate overfitting and promote generalization is to split the dataset into three distinct parts: training, validation, and test sets.

An embedding is a numerical representation, typically a list of numbers or a vector, that positions a piece of semantic meaning within a multi-dimensional space. The core principle is that items with similar meanings are located closer to each other in this embedding space.

This concept underpins various functionalities in AI, including semantic search, where queries are matched by meaning rather than exact keywords, and recommendation systems that suggest similar items. Embeddings also play a crucial role in tasks like fraud detection by identifying anomalous patterns in vectorized data.

Large Language Models (LLMs) do not directly process words or individual letters. Instead, they operate on 'tokens,' which are chunks of text. In English, these tokens typically consist of three to four characters.

This token-based processing explains why LLMs often struggle with tasks that require precise character counting or manipulation, as their internal representation is not character-level. Furthermore, tokenization has implications for cost, as non-English texts often require more tokens to represent the same amount of information, making them more expensive to process.

Despite advertised large context windows, such as one million tokens, these do not necessarily translate to high reasoning capacity throughout the entire window. Research indicates that the accuracy of models often degrades significantly before reaching the stated context limit, and this degradation is not gradual but abrupt, with models hitting performance 'cliffs.'

This suggests that while models can theoretically hold a vast amount of information, their ability to effectively utilize or reason over all of it diminishes. The practical, effective context window for complex tasks is often much smaller than the theoretical maximum.

Furthermore, there is a distinction between a model's ability to simply retrieve facts from its context and its capacity to perform complex reasoning based on those facts, with retrieval generally having a higher success rate than deeper reasoning tasks.

Retrieval Augmented Generation (RAG) is a technique that enhances model accuracy by dynamically altering the context at query time. This process involves embedding a collection of documents into a searchable format, retrieving relevant chunks of information based on the user's query, and then integrating these retrieved chunks directly into the model's prompt.

Effective RAG relies heavily on the quality and precision of the retrieval system, as its ability to find the most accurate and pertinent information directly impacts the model's output. By grounding responses in specific source material, RAG significantly reduces the occurrence of 'hallucinations' or generated content that is factually incorrect or unsupported by real-world data.