The Bottleneck in Financial Agent Development is 'Developers' Themselves; The Solution is Full Automation of the Development Loop

Siddhant Imani, a data scientist at Auditoria AI, pointed out that the main culprit behind bottlenecks in the financial agent development process is the developers themselves, the internal harnesses used for development, and the process of building production agents through them. In this lecture, he discussed ways for developers to automate the development loop to increase productivity.

Imani stated that he would delve into the bottlenecks in financial agent development, specifically addressing the problems arising from the internal harnesses currently used by developers and the process of building production agents.

This presentation will be delivered by Siddhant Imani, a data scientist at Auditoria AI, a company that builds production agents in the financial sector. He will provide in-depth insights into financial agent development through this session.

At the AI Engineer World Expo, numerous beautiful demos are showcased, but problems arise when these demos are promoted to pilots in real production environments and begin onboarding new clients. Agents encounter future data they haven't experienced before, leading to a surge in production bugs. While it's easy to write code and display impressive demos, in a real operational environment, one faces continuous bug occurrences.

Siddhant Imani explained that while many developers can easily create and present beautiful demos, when these demos transition to pilots in a real production environment and begin to serve new customers, agents encounter new data they haven't seen before, leading to numerous bugs that tend to increase over time.

Siddhant Imani emphasized that the reason these demos fail in production environments is not better models, faster GPUs, or better frameworks. He stated that new models are released every three and a half months, new chips and GPUs about every year, and frameworks emerge daily, making these elements easily replaceable. The real problem, he pointed out, lies in how to solve these production bugs in real-time.

Imani asserted that better models, GPUs, or frameworks are not the root cause of production failures. He stressed that models, hardware, and frameworks are rapidly evolving and easily replaceable elements, making it far more crucial to effectively handle bugs that occur in a production environment.

Siddhant Imani concluded that the true problem lies in the speed of the development loop. While the capabilities of models are increasing exponentially, developers are still spending a lot of manual time daily trying to automate the development loop.

Imani pointed out that developers' efforts to automate the development loop are insufficient compared to the exponential growth in model capabilities, emphasizing that improving the speed of the development loop is essential.

Siddhant Imani suggested that four core primitive elements must be considered for development loop automation. He expressed his hope that attendees could apply these principles to their production code after this session.

Imani mentioned four fundamental elements for efficiently automating the development loop, expecting improvements in production code through these elements.

The first primitive element of development loop automation is sub-agents. In modern harnesses, numerous sub-agents can be created, and it is crucial to manage them through a worktree. A worktree acts like an isolated folder, containing all the code generated by agents and allowing sub-agents to perform independent tasks in parallel without colliding with each other.

Siddhant Imani emphasized the use of sub-agents, explaining that the worktree serves as a key tool for managing independent work units, enabling parallel processing, and contributing to building an efficient development environment.

The second primitive element is skills. Skills are like the unique secret recipes of each organization, and by endowing agents with these skills, agents can always use the correct and appropriate workflows to resolve all production bugs. Therefore, he emphasized the importance of acquiring many skills.

Imani defined skills as core organizational know-how, explaining that it's crucial for agents to leverage these skills to efficiently solve problems and maintain accurate workflows.

The third primitive element is MCP (Multi-Cloud Platform) tools. He stated that various MCP tools are currently being released, and they claim to support agents connecting to all MCP tools and third-party servers. This means client data can reside in any system, ensuring the scalability of agents.

Siddhant Imani explained that the ability of agents to seamlessly connect to various MCP tools and external systems is crucial, as it enables flexible management of client data.

The final primitive element is minimal UX (User Experience). As the number of sub-agents increases, more tasks need to be coordinated, so a minimal UX that reduces complexity plays a key role. This maximizes efficiency by allowing developers to handle all tasks without opening multiple windows.

Siddhant Imani emphasized that a minimal user experience that reduces complexity is essential for efficiently managing multiple sub-agents. This allows developers to handle multiple tasks in an integrated manner on a single screen.

Siddhant Imani provided an example where agents can automate almost every step of the development workflow. This begins from the point when QA reports a bug ticket, and the agent takes charge of requirements analysis, root cause analysis, collecting trace information and logs, performing TDD, implementing fixes, writing test scripts, running local end-to-end tests, creating PRs and requesting reviews, merging to the master branch, building Docker images, and deploying and testing in development and staging environments. Throughout this entire process, he asked the audience when human intervention would be needed.

He questioned at which of the 9 steps in this process—bug ticket reporting, requirements analysis, root cause analysis, log collection, TDD execution, code modification, test script writing, PR creation, code merging, Docker image building, and deployment and testing in development and staging environments—human interaction would be necessary.

Siddhant Imani argued that in the 9-step development workflow mentioned earlier, human involvement is only necessary in step 1 (initial review) and step 9 (final validation). He stated that agents can perform intermediate tasks much more efficiently and accurately, so humans should focus on verifying what the agents are doing and finally validating the work after it has been deployed.

He emphasized that humans should be limited to reviewing the agent's work and finally validating the deployed results. This is based on the belief that agents can perform intermediate tasks better.

When developers perform various tasks in the software life cycle, opening multiple windows and using multiple monitors can actually be inefficient. Siddhant Imani explained that by handling many tasks using a single, integrated widget, similar to a Mac OS widget, the number of times one has to turn their head to deploy a single change—that is, context switching—can be significantly reduced.

He mentioned that developers can handle many tasks with just this one widget, thereby reducing unnecessary movements and context switching during the development process to increase efficiency.

The financial sector, where Auditoria AI operates, is strictly regulated. Siddhant Imani explained that in the absence of artificial intelligence, when coordinating sub-agent teams within the financial sector, it is typical for human auditors to review code and for controllers to verify SOX (Sarbanes-Oxley Act) compliance. This demonstrates that human accountability is essential given the nature of the financial sector.

Imani mentioned the regulatory environment of the financial sector, stating that without AI, it is common for human auditors to review code and controllers to handle SOX compliance when coordinating sub-agent teams.

Siddhant Imani emphasized the importance of clearly defining accountability when production issues arise during an agent-to-agent review process. He pointed out that since responsibility cannot be attributed to the cloud or the agent itself, human accountability ultimately remains essential. This means that even if an agent performs tasks, the ultimate responsibility lies with humans.

Imani asserted that when production issues arise during an agent-to-agent review, responsibility cannot be shifted to the cloud or the agent, and thus, ultimately, humans must bear that responsibility.

Siddhant Imani stated that while he has previously argued that human review and validation of agent work are necessary, he now claims that developers can be completely removed from the loop thanks to the self-improvement capabilities of agents and models. He believes that the advancements in cutting-edge models like Fable 5, Mythos 5, and GPT 5.6 enable this transformation.

Imani explained that with the self-improvement features of the latest agents and models, a higher level of automation is now possible, eliminating the need for developers to directly review and validate every step.

Siddhant Imani explained that every production failure can be used as valuable input for the agent's self-improvement. By automating development tools daily and instructing agents to upgrade themselves, an agent performs tasks and runs the loop for about 1-2 days, resolving 5-6 bug tickets. In this process, the agent analyzes all bottlenecks, creates a list, and automatically improves by progressively eliminating these bottlenecks daily.

Imani emphasized that agents can learn from production failures, automate development tools, and upgrade themselves. He explained that a recursive automation system can be built where agents analyze and eliminate bottlenecks, improving daily.

Siddhant Imani stated that the era has arrived where one can set a goal for an agent and then forget about it. An agent can be given the goal of being notified about data inconsistencies in a specific report, and instructed to fix and investigate issues where agent-generated data does not match original data in production bugs. Combining such goals with loops allows agents to automatically resolve problems and improve the system.

He explained that by setting goals for agents and running automated loops, agents can autonomously resolve production issues such as data inconsistencies. This highlights the agent's autonomous problem-solving capabilities.

Siddhant Imani explained that agents can automatically upgrade the system through a function that allows them to 'dream' like humans. For example, if many customers are experiencing the same type of problem pattern, an agent can collect all sessions used by customers in the background and compress this into data points that the system can utilize. This essentially allows the agent to upgrade itself and proactively learn and solve problems that customers repeatedly face. This enables active system improvement.

He emphasized that agents can learn from recurring customer problem patterns, 'dream' like humans, accumulate data, and upgrade the system themselves. By collecting and analyzing customer session data for self-improvement, agents gain the ability to preemptively address problems before they occur.

Siddhant Imani concluded that human attention is very limited, so humans should always act as validators but should not become a bottleneck that limits the agent's throughput. Agents should handle most of the processing, and humans should focus on final verification and approval.

He emphasized that given limited human attention, humans should remain in the role of validators for tasks performed by agents and should not become a factor that hinders the agent's processing capabilities.