tldraw Elevates AI Agents from Text to Canvas

He instructed the agent to locate the specified document and then construct the demo within the tldraw desktop application. The agent began processing the request in the background, setting the stage for a later reveal of its work.

Max Drake, a product engineer at tldraw, introduced himself and his work, focusing on agents on the canvas. His involvement with canvas-based agent interactions began even before the public release of ChatGPT, highlighting his long-standing commitment to this specialized field.

This SDK handles intricate functionalities such as object selection, resizing, and complex matrix mathematics, streamlining the development process for canvas-based tools.

The SDK's robust capabilities simplify the development of sophisticated canvas-based tools, demonstrating its value in various development environments.

tldraw, both as an SDK and an application, incorporates built-in multiplayer functionality with live synchronization. Features like shared cursors, selections, and viewports facilitate seamless collaboration, allowing humans and AI agents to work together efficiently within the same canvas environment.

Current successful coding agents primarily operate in text-based environments, leveraging their training on text input and output. The presentation emphasizes a shift from these traditional text-only agents to those capable of spatial interactions on a visual canvas, marking a new frontier for AI agent development.

The agent can successfully identify objects, their relative positions, and even environmental effects like wind or smoke within the canvas. This interpretation allows actions to be precisely mapped and executed on the canvas through iterative data processing.

Agents can now independently set goals based on the current state of the canvas, mirroring the self-directed behavior observed in coding agents. The system allows agents to search and traverse the canvas environment, enabling more complex and continuous tasks.

The design of tldraw shifted to emphasize multiplayer and collaborative interactions, moving beyond single-user agent systems. The 'Fairies' project specifically allows humans and agents to interact within the same shared canvas space, fostering a truly collaborative environment. Individual agent customization is crucial for maintaining clear identity and tracking contributions in a busy collaborative canvas.

Users can summon several agents and assign them to specific tasks, or group them for coordinated efforts. This system facilitates complex multi-agent workflows within the visual interface.

This system enables efficient oversight of large groups of agents without the need to read extensive text. An orchestrator agent delegates tasks, monitors progress, and manages review requests, all visually represented on the canvas.

Integrating them with real-world, external tasks proves challenging due to their canvas-dependent architecture. Users must actively 'opt in' to the full canvas harness to access and utilize these specific features.

Each node in the graph represents a coding agent capable of performing tasks autonomously. This system enables users to manage intricate, multi-step projects that are entirely generated and executed by these agents.

The tldraw desktop app supports multiplayer functionality, allowing teams to coordinate and work together efficiently. This includes direct management and merging of pull requests within the canvas interface. Users can assign tasks to AI agents, such as Claude, and sync progress across multiple collaborators in real-time.

The tldraw desktop app functions as a local scripting environment, exposing the editor's instance to external agents. This setup allows agents, such as Claude Code, to execute plain JavaScript directly against the canvas, automating various workflows. This functionality integrates seamlessly with 'code mode' capabilities, enhancing the canvas's programmatic control.

Agents with computer access can transcend the canvas environment to interact directly with the operating system, allowing tldraw to function as a window manager. For instance, Claude Code, having direct access to the computer, is not confined to the canvas, enabling it to control external applications. An example demonstrated agents creating ephemeral UIs to control the desktop version of Pong.

Agents integrated with tldraw can process real-world data sources, including emails and project specifications from Notion documents. This capability allows agents to automatically implement technical specifications directly onto the canvas. It effectively bridges the gap between abstract agent logic and tangible visual output.

This shared spatial environment serves as a crucial bridge, enabling seamless collaboration between humans and AI agents.