MiniMax's M3 Model Integrates Million-Token Context

MiniMax is increasingly recognized as one of China's top AI startups, often grouped with other major players like DeepSeek, Moonshot (Kimi), and GLM. The company is highlighted as part of a highly competitive landscape, with teams vying for the leading position in AI innovation. This strong competitive environment pushes MiniMax to continuously develop cutting-edge models and technologies.

MiniMax released the M3 model in June, which is characterized by its substantial parameter count and efficient design. Olive Song detailed the model, stating that it features 400 billion total parameters with 20 billion activated parameters. This configuration enables a functional one-million-token context window, making it suitable for handling extensive data inputs.

The M3 model achieves its functional one-million-token context window through a novel architecture known as MiniMax Sparse Attention (MSA). This innovative design not only facilitates enhanced context processing but also enables multimodal capabilities, allowing the model to understand text, video, and images. Olive Song emphasized that this integrated approach supports advanced coding and complex agentic workflows, moving beyond the limitations of typical open-source models that often focus solely on code.

MiniMax's journey with long context models began with earlier iterations like M1, which could process up to 10 million tokens for retrieval tasks. Olive Song recounted that these early models were primarily used for static tasks, such as analyzing long books. However, the team soon realized that simple retrieval was insufficient for complex agentic tasks, necessitating longer context windows to manage tool responses and multi-round user interactions effectively.

The MSA architecture employs a sophisticated two-branch system to maintain efficiency while processing vast contexts. Olive Song explained that the design is both scalable and simple, featuring an index branch that identifies the most relevant information at a high level. Following this, a sparse attention branch performs calculations specifically on the selected blocks, ensuring precise task execution without unnecessary computational overhead.

The design philosophy behind MiniMax M3 emphasizes a scalable architecture to accommodate future growth in both context length and model size. Olive Song highlighted the deliberate effort to create an 'elegant architecture' that would facilitate continuous scaling. This foresight ensures that the M3 model can adapt to increasing demands and more complex applications in the long term.

The development of MiniMax's attention mechanisms involved a fundamental re-evaluation of how attention works and how it can be made more efficient. Thomas Wolf noted the historical shift from N-square and linear attention to more modern techniques like flash attention. The team focused on making attention more efficient to handle the exponential growth in context limits, moving from GPT-2's 1,024 tokens to the current 1 million, which he described as "crazy."

Remarkably, MiniMax's sparse attention architecture, a core component of the M3 model, was designed by an intern. Olive Song mentioned this unique contribution, noting that such an event is uncommon in many research labs. MiniMax's open culture allows interns significant access to data and the opportunity to contribute to core models, fostering an environment where efficiency for task-sensitive applications is prioritized, regardless of seniority.

MiniMax cultivates a bottom-up approach to project organization, encouraging all employees to contribute to model improvements. Olive Song explained that the company ensures a robust foundation and infrastructure, allowing anyone to experiment with the model and identify areas for enhancement. Following model releases, employees are free to propose projects to address identified weaknesses, often forming cross-functional teams to iterate on solutions. These collaborative projects can span from weeks to months, and successful improvements are ultimately integrated into the final training, showcasing MiniMax's commitment to internal innovation and continuous development.

MiniMax employs a 'native multimodality' approach, integrating vision understanding from the initial training stages rather than adding it post-text pre-training. Olive Song explained that many labs typically add vision adapters after text pre-training, which they found can negatively impact text performance. By training vision and text together from step one, MiniMax avoids common convergence issues associated with staged training, ensuring a more deeply integrated and robust multimodal capability.

Staged training, a common approach in AI model development, presents significant challenges when attempting to scale. Olive Song highlighted that the 'recipe' for staged training is highly sensitive, varying for different architectures, data mixtures, and learning rates. This makes it difficult to control results and scale experimental conclusions from smaller models to larger ones effectively, often leading to unpredictable outcomes.

MiniMax has ambitious plans for the future scaling of its models, aiming to surpass the trillion-parameter mark. Olive Song confirmed that the current model stands at 428 billion total parameters with 23 billion active parameters. She expressed certainty that they would go past a trillion, driven by the belief that many complex tasks require larger parameter counts for optimal performance.

From its inception, MiniMax has been guided by a foundational vision centered on Artificial General Intelligence (AGI) and multimodality. Olive Song revealed that the company's CEO envisioned a model capable of understanding and outputting all modalities even before the company officially started, predating the public release of ChatGPT. This long-term goal of AGI drives their development, with applications serving as crucial gateways for users to directly experience the model's evolving capabilities.

MiniMax maintains a strong commitment to open-sourcing its models, viewing community collaboration as vital for development. Olive Song emphasized that it is the plan for both herself and the model research team to open source their work. They highly value feedback from the open-source community, noting that contributions and pull requests directly inform future versions and help improve the models.

MiniMax is actively seeking and prioritizing feedback on the multimodal capabilities of its new models. Olive Song stated that they are particularly interested in any issues users encounter, especially since this is their first time combining these functionalities. She indicated that the team plans to be even more ambitious with multimodality in the future, with new features like 'thinking effort' being considered based on user demand and input.

Multimodality is expected to unlock a wide range of new capabilities and agent applications, according to Olive Song. She provided examples such as a model's ability to read unstructured data like PowerPoint presentations or reports, and to understand very long videos. This capability allows agents to trigger subsequent tool use based on complex visual and textual information, a field she described as currently underexplored but highly impactful for future agent workflows.

MiniMax leverages its own research harnesses to automate a significant portion of its internal workflows. Olive Song explained that models like M3 are used to optimize kernels and generate data, thereby accelerating the iteration speed for new versions. This automation is a critical factor in their rapid development cycle, reflecting a trend among frontier models towards optimizing kernel operations for increased efficiency.

Olive Song finds the emergence of multi-agent systems and model routing particularly exciting, noting that many AI applications are adopting these approaches. These systems enable more complex tasks by chaining together the capabilities of multiple models, leading to greater functional depth. Multi-agent architectures also help in clarifying the specific strengths and weaknesses of different models, optimizing their deployment for diverse applications.