GitHub Trending Graft: What are the Actual Measurement Results?

When a session begins, the coding agent has no information about the codebase and only refers to rule files and harness definitions.

This structure forces the agent to understand the codebase through loop exploration using tools like grep, read, and bash.

If file exploration fails, the number of LLM calls unnecessarily increases, and each tool call result continuously accumulates in the context window.

As a result, as the number of conversation turns increases, an overhead occurs where the amount of input tokens increases proportionally.

This is because existing code exploration results are deleted when a session ends, leading to the inefficiency of having to restart codebase exploration from scratch in the next session.

This knowledge graph allows the agent to automatically inject relevant context with each prompt, thereby reducing repeated code exploration and increasing efficiency.

Graft has a lightweight architecture that does not require a separate graph DB or vector DB.

The folder structure mirrors the source code, with one markdown file created for each source file in a 1:1 structure within the Graft folder.

This markdown file records the line-by-line location information of functions and classes, and queries this markdown file instead of the actual source code to obtain code information.

During an actual Graft build, 508 markdown files were created, and it was confirmed through JSON files that 2,851 nodes and 9,094 edges were generated.

Graft maps the codebase to markdown files to construct a knowledge graph.

Questions asking about relationships, such as 'Who calls this function?', are explored by following the edges of the graph, while questions about other content are ranked based on the overlap of specific words.

This method has the advantage of fast search speed without using LLM and consistent results.

Knowledge graph generation consists of two layers.

The first layer uses Tree-sitter to analyze only code structures such as functions, classes, and call relationships, and does not incur separate costs as it does not use an LLM model.

The second layer is activated when the `--deep` option is provided, in which case the LLM summarizes files and bundles them into sub-system units to create markdown nodes. Graft also provides real-time comparison and updating of work tree changes.

Additionally, it suggests that a 54% to 66% improvement in correctness can be expected.

Specifically, it provides a codebase summary at the start of a session, adds relevant nodes to the backend upon prompt submission.

It also analyzes impact immediately after tool use and rebuilds the graph at the end of a session, maximizing coding agent efficiency.

Only hook files and schema files are included in the commit target; the local cache is excluded.

This is because it is designed to use individual build methods for each local environment.

However, there is a risk of exposing corporate code or sensitive information, so developers are advised to disable the telemetry feature before using Graft.

To disable telemetry after installing Graft, enter the `graft telemetry disable` command.

Two types of projects were configured for the Graft benchmark.

The first was a YouTube analysis CLI tool of approximately 2,000 lines, and the second was a video editing app project of approximately 100,000 lines.

The LLM model was fixed to Claude 3.5 Sonnet for all test conditions.

Task A was a question about multi-file flow, asking 'which files does the report command go through to execute?'.

Task B was to explore the scope of impact when changing a function signature, and Task C was about a small modification within a specific single file.

Actual measurement results for tool call reduction showed that in small projects, tool calls decreased from 7 to 3.6 and from 3.8 to 1.8 when Graft was not used.

In large projects, there was also a tendency for calls to decrease from 19 to 14 and from 2 to 1.2.

Combining these four cases, an average 41% reduction in tool calls was confirmed, which is generally similar to Graft's claimed 46% reduction.

However, in large projects, time showed a pattern of increasing.

For example, what took 15 seconds increased to 26 seconds, and what took 75 seconds increased to 95 seconds, confirming that applying Graft to large projects resulted in unchanged or slightly increased time.

Regardless of whether Graft was applied, the accuracy rate remained similar or slightly lower.

This supports the conclusion that Graft is suitable as an auxiliary tool to improve agent efficiency by enhancing the existing full-file reading method to a graph query method, rather than making the agent smarter.

Ultimately, Graft's claimed reductions in tool calls and accuracy were somewhat reproduced in the benchmark figures, but the time reduction effect varied depending on project size.