Google DeepMind Unveils New AI Agent Sandbox

With the release of Omni Flash for programmatic video, Google DeepMind's Philipp Schmid introduced Imagen 3, a new image generation model available at 4 cents per image.

The company also shipped Omni Flash on the Interactions API, which provides capabilities for programmatic video creation and editing, further expanding Google's suite of AI models, including Gemini and the open-source Gemma models.

The field of large language models (LLMs) has undergone a significant transformation, moving beyond early models like BERT and GPT from 2018-2019 that focused primarily on basic text completion.

The introduction of instruction tuning allowed models to directly answer user questions, a capability further popularized by ChatGPT's turn-based conversational approach.

Today's development centers on creating 'black box' goal-oriented agents that can think, utilize functions, and execute complex tasks autonomously over extended periods, ranging from hours to days.

Web developers building agents benefit from the Interactions API, which streamlines multimodal input handling and offloads state management to the server.

This API supports multimodal inputs, allowing agents to process various data types, including strings and media data, through a unified interface for both basic models and advanced agents.

It incorporates server-side state management to alleviate client-side burdens and introduces asynchronous 'long-running operations' to handle tasks like video generation without encountering HTTP connection timeouts.

The server now manages context by concatenating previous inputs, ensuring the model retains state across interactions and supports various modalities such as text, Gemini TTS for audio, and image generation.

This architectural shift enables the system to handle function calling and tool execution, granting models the autonomy to decide whether to perform tasks like Google Search independently.

Traditional LLM providers often rely on a 'user-model' turn structure, which presents challenges when handling the complex reasoning and tool-calling capabilities of modern AI agents.

Philipp Schmid highlighted the conceptual awkwardness of returning function outputs within a 'user' role, as this information originates from the environment rather than the human user. The Gemini API addresses this by adopting a 'synchronous timeline of steps', where each action is assigned a unique type, thereby enhancing extensibility.

Google has integrated a managed sandbox environment into Gemini API agents, leveraging the backend architecture originally developed for Project Astra.

Google offloads the burden of infrastructure management, allowing agents to execute code, install dependencies, and create files within a stateless yet persistent cloud environment.

These managed sandboxes maintain state across multi-turn interactions, enabling agents to leverage information from previous steps to inform subsequent actions.

Multiple agents operating in the same environment can work effectively by accessing a shared file system to pick up and continue tasks initiated by other sub-agents.

This setup allows for the creation of sub-agents that can seamlessly continue tasks initiated by other agents, provided the model is equipped with appropriate tools and data sources like GCS buckets or GitHub repositories.

Additionally, if an 'agents.md' file is present, its contents are appended to the system instructions, ensuring agents are aware of their capabilities and context without requiring manual configuration updates.

By utilizing an environment ID system, developers define base agents independently of complex infrastructure like Kubernetes or Terraform through the Agents API.

Developers can perform environment scaffolding, including installing CLI tools and running sanity checks, once to obtain an environment ID. This ID can then be used to create new agents that will automatically spawn from the pre-loaded and pre-configured environment, ensuring consistency and efficiency.

Programmatic iteration of agent systems is now possible via a dedicated CLI that allows agents to interact with Gemini model endpoints to generate assets.

This CLI supports calling various models, such as Nano, Flash, or Omni, to generate assets, and facilitates iterative improvement of agent systems. GitHub repositories offer dedicated skills for both the interactions and agent APIs.

The system autonomously performs research, generates speech, music, and visual thumbnails for the radio segments.

An example topic discussed during the demonstration was the common 'cereal first vs. milk first' debate, illustrating the agents' capability to engage in nuanced discussions.

To customize tools like talk show radio applications for Gemini, developers can explore the open-source gallery provided within the 'build' section of AI Studio.

Users can remix existing applets, such as the talk show radio application, to customize them for specific needs. These applets are open-source, allowing full transparency into their underlying code structure.

Gemini agents are constructed from file sets, notably an 'agents.md' file that contains system instructions loaded upon environment startup, defining the agent's core tasks such as research, script writing, music generation, and speech synthesis.

Skills are implemented as Python files, which allow the Gemini agent to make calls to other models via the Interactions API, providing a modular and extensible approach to agent behavior.

The 'anti-gravity agent' template offers six pre-configured setups for comprehensive testing of agent capabilities within a sandboxed environment.

This sandbox allows agents to execute bash commands to identify host system specifics, such as the Ubuntu version, vCPU count, and memory, providing detailed operational context.

Users can download files generated by the agent or interact with them via an API, while the system manages the agent loop server-side, removing the need for client-side function call management.

With the managed agents, everything runs inside a remote sandbox after a single API call triggers the process.

The environment supports connections to both remote and local MCP servers, and comprehensive documentation for building and deploying these agents is available in the Gemini API 'Agents' section.