Optimizing Open Source Documentation for AI Agents to Boost Recommendation

Christopher Burns introduced Lead Type, a framework-neutral tool designed to automate documentation for AI agents by processing `.mdx` files into an optimized format. The primary goal of this tool is to enhance the agent experience on websites, ensuring that AI agents can efficiently parse and utilize documentation.

Burns stated, "it basically does is take your .mdx files, you run Lead Type generate, and it will spit out everything for optimized agent experience for your websites."

Burns underscored the significance of the `LLM.txt` file, which functions as a sitemap specifically for documentation sites, guiding language models. He emphasized that manual creation of this file is far superior to automated generation, a finding supported by their research.

He noted that a meticulously crafted `LLM.txt` with about 40 high-quality lines significantly outperforms a version with 1,000 lines of automatically generated noise. Burns explained, "What we found in our research is that it's much better not to just generate this. It is much better to write your LLM.txt from hand."

A key recommendation for optimizing agent interaction is to switch from HTML to Markdown. HTML is computationally expensive for agents to parse, leading to higher token costs. By shipping Markdown directly to agents, these costs can be significantly reduced.

Burns stated, "HTML is expensive, and why can't we just ship markdown to the agents?" The implementation involves creating 'twin MD' files by simply appending `.md` to standard documentation page URLs, making them accessible in a more agent-friendly format.

Implementing header signals is another crucial step for agent discoverability. Many leading documentation websites, including those from Vercel, use these signals to inform agents about alternative Markdown versions of their content.

These header signals tell agents that a more efficient Markdown version of the page is available. Burns explained, "If you look at all the best documentation websites... they all have this in the header. This is saying to the agents whenever they visit the website that there is an alternative version of this in mark markdown." However, the reliability of agent support for these headers is currently inconsistent.

To ensure comprehensive accessibility for AI agents, documentation should be available through multiple methods. This caters to diverse agent configurations and capabilities.

Burns outlined three primary methods: direct `.md` links, redirects based on an agent's acceptance of specific headers, and URL query parameters (e.g., `mode=agent`) for agents that cannot modify or interpret header tags. Burns stated, "you need to make sure that they're available through multiple methods. So, one of them is like the .md... another one is just taking the normal... link and then adding a redirect... and then the third one is that not all agents can append header tags. So, there's also a URL query of mode equals agent."

Integrating with Web MCP (Web Machine Communications Protocol) is essential for future-proofing agent interactions through standardized protocols. Lead Type is already designed to expose three distinct tools to Web MCP: `Search docs`, `Get pages`, and `Ask docs`.

This integration provides agents with the structured context they need to formulate relevant questions and interact effectively with documentation. This proactive approach ensures that as Web MCP gains wider adoption, Lead Type-optimized documentation will be readily accessible and understandable by agents.

Burns revealed an "uncomfortable truth" about coding agents: they often bypass websites when dealing with libraries, instead visiting local repository files, specifically `node_modules`. This is largely because current training data can be stale, compelling agents to rely on compiled source files for up-to-date information.

Following the lead of companies like Vercel, the strategy involves bundling Markdown documents and placing them within the `node_modules` directory as an `agents.md` file. Burns explained, "the uncomfortable truth is that coding agents are actually never visiting the website if you have a library. They're actually visiting the node modules... So, again, following what people like Vercel are doing... we take the bundled markdown documents and then we also put them in the node modules with an agents.md file."

The `agents.md` file serves as a central repository for library documentation, making it readily available for LLMs. This approach significantly streamlines the process for agents when they need to resolve problems or answer questions related to a library.

Burns stated, "If you've got a problem, if you've got a question, all the documents are here. Grab them." By having all necessary documentation consolidated within `agents.md`, LLMs can directly access information, leading to substantial token savings by circumventing the need for web searches.

Embedding documentation directly within the codebase, rather than forcing agents to search the web, results in significant efficiency gains in token usage. Many models have shown an almost 50% token reduction with this method.

Burns highlighted, "We can see that between many different models almost 50% token saving on instead of trying to search the web, find the right tools, pulling the markdown files from your code base." This approach is particularly effective for libraries that undergo frequent changes, ensuring agents always access the most current documentation directly from the codebase.

While the basic implementation of bundled documentation works effectively without any additional skills, developers can extend its functionality by adding custom skills. These skills provide specific instructions for agents, such as directing them to look at the Node modules for information.

Burns noted, "This is also working without any skills, but if you want as well, you can add skills to it to say, 'Look at the node modules and go from that.'" This flexibility ensures compatibility with various frameworks, including Next.js, and allows developers to fine-tune how agents interact with the documentation bundles to verify their contents.

Burns addressed the current lack of established test suites specifically designed to measure how 'agent-ready' websites are. He mentioned Cloudflare as an early provider of evaluation tools, indicating a nascent but growing interest in this area.

He enthusiastically recommended Aura AI as a new and effective tool for testing agent readiness, stating, "But, my favorite is actually one called Aura AI. Um this is brand new and it tests a lot." The emergence of such tools suggests a growing recognition of the need for standardized metrics to assess agent-friendly documentation.

Burns emphasized that in the rapidly evolving landscape of AI and agent technologies, iterative improvement is paramount, and perfection is an unattainable goal. He candidly shared his own experience, stating, "There is no such thing as perfection. When I started making these slides, I got so caught up of like, everyone expects me to be the expert here, but I've just been hacking on this problem a little more than you guys have so far."

This perspective underscores that expertise in this domain is derived from continuous experimentation and adapting to market changes, rather than a one-time, flawless implementation.

Burns strongly recommended providing a `.md` file for every single webpage, acknowledging that many Content Management Systems (CMSs) are not inherently built to support this. This strategy is driven by the observed trend of AI agents visiting websites more frequently than humans, making agent-friendly formats critical for discoverability.

He stated, "Trying to provide a dot md file for every single page. A lot of CMSs are not built in this way." Proactive documentation in Markdown leads to higher token efficiency and ensures that agents can quickly and cost-effectively process information, enhancing the overall discoverability of content.

For new websites looking to optimize their documentation for LLMs, Burns recommended starting with the `llms.txt` file as a foundational step. He stated, "I would say llms.txt is your first shout." This file acts as a crucial sitemap for AI agents.

If a CMS lacks native support for generating Markdown files, he suggested manually creating them. Additionally, he encouraged creative, on-the-fly file generation to ensure that documentation is available in an agent-friendly format, even if a direct `.md` file for every page isn't immediately feasible.