VS Code Shifts to Weekly Releases with AI

The primary goal of this acceleration was not solely to increase shipping speed, but to enhance the effectiveness of releases by fundamentally rethinking development systems, rather than simply incorporating AI into day-to-day tasks.

Initially, GPT-4.1-generated code had a 55% survival rate, but with advancements in testing harnesses and newer models such as Claude Opus 3.6, this rate climbed significantly to 86%.

The integration of AI led to a substantial increase in reported issues, as AI assists users in filing them, according to Kirschner.

The team now manages a higher volume of issues, including both critical quality concerns and automated, lower-quality submissions, with AI helping to categorize them.

Notably, the number of community-merged pull requests (PRs) also increased, indicating AI's potential to facilitate greater community contributions to the project.

Harald Kirschner outlined a three-stage framework for successful AI-enabled development, emphasizing that simply shipping code faster, which he called the "easy part," is only the beginning.

The framework includes Stage 1: achieving faster shipping through agent-ready codebases and specialized skills; Stage 2: maintaining high quality despite increased velocity; and Stage 3: learning faster to ensure the development of the right features.

For AI agents to be effective, codebases must be 'agent-ready,' meaning they provide clear, navigable maps for AI systems.

The VS Code team achieves this by creating `agents.md` files, which serve as dynamic documentation that evolves as agents encounter and resolve errors, providing up-to-date guidance.

Additionally, expert knowledge, such as accessibility best practices, is codified into reusable 'skills' that AI agents can leverage, helping to unblock contributors and ensure adherence to standards.

Traditional build and linting processes often create significant bottlenecks for developers in continuous integration/continuous deployment (CI/CD) pipelines.

When multiple AI agents run concurrently, these existing delays compound exponentially, severely hindering development efficiency.

To address this, VS Code transitioned to TypeScript Go, which yielded a tenfold improvement in build speeds.

Faster build times are crucial for enabling the rapid, automated pull request feedback loops that AI agents require for efficient operation.

AI models frequently encounter difficulties with user interface (UI) layout and the precise alignment of elements.

To mitigate this, VS Code implements a 'component browser' that automatically captures screenshots of every UI component whenever changes are made.

This system helps developers identify 'ripple effects,' where a modification in one area inadvertently causes breakage in another part of the UI.

Requiring videos or screenshots in pull requests (PRs) eliminates the need for developers to build an application from scratch merely to assess UI changes, streamlining the review process.

VS Code, being fundamentally a web application, can be extensively automated using Playwright, a framework for browser automation.

This capability allows AI agents to utilize a `/launch` skill, enabling them to open VS Code, navigate through specific scenarios, and validate fixes without requiring manual intervention.

This approach significantly offloads testing tasks to AI agents, which can then provide detailed logs upon completion, enhancing diagnostic capabilities.

Similar automated feedback loops, described as "magical," are also found in tools like Xcode MCP for iOS development.

Harald Kirschner stated that GitHub Copilot for PRs has matured significantly, evolving to a point where its use is now mandatory for every pull request opened in VS Code.

Teams can adjust the AI's review 'effort' level based on the risk associated with the repository, ensuring appropriate scrutiny.

A critical aspect of this system is that human reviewers are not permitted to examine a pull request until all comments and suggestions from the AI have been addressed by the developer.

This enforcement mechanism is vital for maintaining high code quality amidst the increased frequency of releases.

VS Code receives a high volume of GitHub issues, often submitted by users from diverse linguistic backgrounds, making manual triage challenging.

AI now automates the filtering of spam, enriches content by adding context, and translates reports, significantly improving the quality of incoming issue data.

The system automatically assigns issues to specific area owners, streamlining the internal response process.

Despite automation, human oversight remains essential for providing feedback when the AI agent misinterprets an issue, ensuring accuracy and continuous improvement.

Harald Kirschner noted that VS Code filters 51 billion raw telemetry entries daily, utilizing a fully automated pipeline to fingerprint errors, group them for area owners, and generate relevant pull request fixes.

The system collects raw telemetry, filtering 51 billion daily entries down to actionable error stacks that include full stack trace information.

It then groups and buckets these errors using a fingerprinting exercise, assigning them to relevant area owners.

An error dashboard monitors impact, and the system automatically generates PRs to address identified issues, closing the loop from error detection to resolution.

Previously, VS Code employed what Kirschner termed "YOLO releases," where new versions were immediately shipped to 100% of users upon release day, assuming full stability after testing.

With the increased frequency of releases facilitated by AI agents, the team has shifted to staged rollouts, incrementally deploying updates to monitor error logs and issues as they emerge.

This change was implemented because rolling back an installed application is a costly and complex process, making a cautious, phased approach essential for risk mitigation.

AI agents are utilized to bootstrap new test scenarios from existing templates, accelerating the creation of comprehensive evaluation environments.

Experimentation using VSC-Bench revealed substantial variations in token usage across different AI models for identical tasks, underscoring the need for careful model selection and optimization.

Models undergo rigorous testing both offline and through online experimentation to refine performance and ensure product quality.

VS Code transitioned from monthly release cycles to daily sprints, significantly accelerating the pace of development and feedback.

This approach prioritizes rapid prototyping to establish early feedback loops, allowing for quicker iteration and refinement of features.

Smaller, dedicated squads manage specific work streams, which helps reduce bottlenecks and enables more agile decision-making within the development process.