Einstein Arena: Collaborative AI Agents Set New Records in Complex Problem-Solving and Kernel Optimization

"it turns out that in higher dimensions, it actually becomes really hard to compute what's the maximum number of over non-overlapping spheres."

"it turns out that in higher dimensions, it actually becomes really hard to compute what's the maximum number of over non-overlapping spheres."

In the 11-dimensional kissing number problem, AI agents in Einstein Arena achieved a new record of 604 spheres, surpassing previous human and AI efforts. This significantly improved upon the 1980 record of 582 spheres, a mathematician's 2022 improvement to 592, and DeepMind's 2023 advancement to 593 spheres.

This breakthrough demonstrates the power of active collaboration among AI agents, which were able to construct the new solution within days.

"with Alpha Zero, we know by having these agents able to collaborate actively, right, in the wild, within a few days they were actually able to construct a new solution that shows that for the first time you can create 604 spheres in 11 dimensions that do not overlap."

Collaboration among multiple AI agents is crucial for solving problems that single models, such as GPT-5.5, cannot handle alone. The Einstein Arena facilitates this by allowing agents to share information and iteratively refine each other's solutions.

Agents utilize a 'lineage trace' to track and optimize collaborative efforts, enabling them to build upon previous attempts and achieve breakthroughs.

"the collaboration among multiple agents is really critical. And here we're actually able to show that there's like this sort of a lineage trace of how the agents are able to collaborate and then basically take each other's solutions and refine that and further optimize it to arrive at this breakthrough."

The system provides real-time feedback through live leaderboards, guiding agents toward better solutions.

The use of specialized agent personas—such as those focused on profiling, memory, or precision—further enhances team efficiency in identifying and implementing optimal kernel improvements.

"we'll provide a feedback to the agents in real time in the form of these leaderboards. In these kernel settings, we also found it to be quite useful to have different agents with different personas"

AI agents successfully achieved substantial speedups, often exceeding two-fold, in production-grade AI kernels, including those used in page attention mechanisms. These improved kernels are now actively deployed in Together AI's production environment.

The framework is designed for generalized application across various hardware types and configurations, ensuring broad applicability of the optimized solutions.

"the agents were also able to collaborate and lead to really quite substantial speed ups, uh including sometimes over two two x two-fold speed ups in some of these production kernels."

"we call this DS Gym, which stands for data science gym, which is sort of like a unified environment that we created for both for evaluating and for training data science agents"

This setup enables parallel testing through Docker containers.

"So, here in this DS Gym environment, we also curated and created a unified list of different data sets and tasks, right? So, these data sets can combine uh spans across many different settings. And the agents are then able to interact with these different data sets that we have through a unified uh interface and through code execution."

Many widely-used data science benchmarks are vulnerable to 'shortcuts,' where tasks can be solved without engaging with the underlying data, revealing a significant reliability problem.

Researchers found that 20% to 50% of tasks in existing benchmarks were solvable through these shortcuts, undermining the true evaluation of agent capabilities.

"But we actually quickly realized that many of the existing widely-used benchmarks actually have many problems. And one big problem is that they're actually very vulnerable to shortcuts."

To counter benchmark vulnerabilities, the DS Gym team carefully curated their own benchmarks, including tasks for scientific analysis derived from recently published papers and predictive modeling tasks from active Kaggle competitions.

These tasks undergo rigorous review by human scientists and experts to ensure robustness and prevent shortcut solutions.

"So, to address that, we actually carefully curated at our own our own benchmarks, right, for both for scientific analysis and also for predictive modeling."

Currently, frontier models achieve less than 50% accuracy on these tasks, indicating the benchmark is not yet saturated.

This low accuracy highlights the challenging nature of the curated tasks and the significant room for improvement and development within the data science agent community.

"So all together in the DS Gym, we actually have created over a dozen different tasks. They span across dozens of different scientific domains ranging from biology to physics to economics."

This process ensures that the generated trajectories accurately reflect successful code execution by agents.

Through this fine-tuning, smaller models can achieve best-in-class performance and operate efficiently on local machines, democratizing access to advanced AI capabilities.

"So one thing we did here is actually generate in the DS Gym actually the gym itself will actually create all these execution verified trajectories, which means that these are trajectories generated by the agents that have been verified through the through through actually executing the code from the agents."

This approach motivates agents to tackle increasingly challenging problems more effectively.

This evolution, moving from model design to agent workflows and now to environment design, is unlocking greater creativity and problem-solving capacity in AI agents.

"But what our research shows is that I think we're already moving towards the next stage, where you're not then trying to design workflows or specific or specific agents, what we really want to do is to design environments, which is a set of infrastructure and incentives that in that motivates the agents that you solve more and more challenging problems."