MiniMax's M3 Model Integrates Million-Token Context

MiniMax is increasingly recognized as one of China's top AI startups, often grouped with other major players like DeepSeek, Moonshot (Kimi), and GLM. The company is highlighted as part of a highly competitive landscape, with teams vying for the leading position in AI innovation. This strong competitive environment pushes MiniMax to continuously develop cutting-edge models and technologies.

Olive Song detailed the model, stating that it features 400 billion total parameters with 20 billion activated parameters. This configuration enables a functional one-million-token context window, making it suitable for handling extensive data inputs.

This innovative design not only facilitates enhanced context processing but also enables multimodal capabilities, allowing the model to understand text, video, and images. Olive Song emphasized that this integrated approach supports advanced coding and complex agentic workflows, moving beyond the limitations of typical open-source models that often focus solely on code.

Olive Song recounted that these early models were primarily used for static tasks, such as analyzing long books. However, the team soon realized that simple retrieval was insufficient for complex agentic tasks, necessitating longer context windows to manage tool responses and multi-round user interactions effectively.

The MSA architecture employs a sophisticated two-branch system to maintain efficiency while processing vast contexts. Olive Song explained that the design is both scalable and simple, featuring an index branch that identifies the most relevant information at a high level. Following this, a sparse attention branch performs calculations specifically on the selected blocks, ensuring precise task execution without unnecessary computational overhead.

Olive Song highlighted the deliberate effort to create an 'elegant architecture' that would facilitate continuous scaling. This foresight ensures that the M3 model can adapt to increasing demands and more complex applications in the long term.

The development of MiniMax's attention mechanisms involved a fundamental re-evaluation of how attention works and how it can be made more efficient. Thomas Wolf noted the historical shift from N-square and linear attention to more modern techniques like flash attention. The team focused on making attention more efficient to handle the exponential growth in context limits, moving from GPT-2's 1,024 tokens to the current 1 million, which he described as "crazy."

Olive Song mentioned this unique contribution, noting that such an event is uncommon in many research labs. MiniMax's open culture allows interns significant access to data and the opportunity to contribute to core models, fostering an environment where efficiency for task-sensitive applications is prioritized, regardless of seniority.

MiniMax cultivates a bottom-up approach to project organization, encouraging all employees to contribute to model improvements. Olive Song explained that the company ensures a robust foundation and infrastructure, allowing anyone to experiment with the model and identify areas for enhancement. Following model releases, employees are free to propose projects to address identified weaknesses, often forming cross-functional teams to iterate on solutions. These collaborative projects can span from weeks to months, and successful improvements are ultimately integrated into the final training, showcasing MiniMax's commitment to internal innovation and continuous development.

MiniMax employs a 'native multimodality' approach, integrating vision understanding from the initial training stages rather than adding it post-text pre-training. Olive Song explained that many labs typically add vision adapters after text pre-training, which they found can negatively impact text performance. By training vision and text together from step one, MiniMax avoids common convergence issues associated with staged training, ensuring a more deeply integrated and robust multimodal capability.

Olive Song highlighted that the 'recipe' for staged training is highly sensitive, varying for different architectures, data mixtures, and learning rates. This makes it difficult to control results and scale experimental conclusions from smaller models to larger ones effectively, often leading to unpredictable outcomes.

Olive Song confirmed that the current model stands at 428 billion total parameters with 23 billion active parameters. She expressed certainty that they would go past a trillion, driven by the belief that many complex tasks require larger parameter counts for optimal performance.

From its inception, MiniMax has been guided by a foundational vision centered on Artificial General Intelligence (AGI) and multimodality. Olive Song revealed that the company's CEO envisioned a model capable of understanding and outputting all modalities even before the company officially started, predating the public release of ChatGPT. This long-term goal of AGI drives their development, with applications serving as crucial gateways for users to directly experience the model's evolving capabilities.

They highly value feedback from the open-source community, noting that contributions and pull requests directly inform future versions and help improve the models.

Olive Song stated that they are particularly interested in any issues users encounter, especially since this is their first time combining these functionalities. She indicated that the team plans to be even more ambitious with multimodality in the future, with new features like 'thinking effort' being considered based on user demand and input.

She provided examples such as a model's ability to read unstructured data like PowerPoint presentations or reports, and to understand very long videos. This capability allows agents to trigger subsequent tool use based on complex visual and textual information, a field she described as currently underexplored but highly impactful for future agent workflows.

MiniMax leverages its own research harnesses to automate a significant portion of its internal workflows. Olive Song explained that models like M3 are used to optimize kernels and generate data, thereby accelerating the iteration speed for new versions. This automation is a critical factor in their rapid development cycle, reflecting a trend among frontier models towards optimizing kernel operations for increased efficiency.

Multi-agent architectures also help in clarifying the specific strengths and weaknesses of different models, optimizing their deployment for diverse applications.