FriendliAI's Inference Cloud for AI Agents

FriendliAI positions itself as the frontier AI inference cloud for agents, specializing in running inference for agents at scale. The company aims to provide faster, cheaper, and more reliable services for this growing segment of AI applications.

FriendliAI originated from a research team at Seoul National University, maintaining a strong research-oriented identity. The company pioneered continuous batching, an inference optimization now standard across the industry, and its Orca work inspired 3LM, a widely used open-source framework. Headquartered in San Francisco with a team in Seoul, FriendliAI operates globally to scale frontier inference, aligning with the exponential growth of AI agents and the maturation of open-weight models.

AI agents are experiencing explosive adoption across software operations and knowledge work. Open-weight models have reached frontier capabilities, now rivaling closed frontier models in performance, which makes agents more economical. This allows for running frontier-quality agents on open models at significantly lower token costs. An example demonstrated GLM 5.2, an open-weight model, completing a tower defense game task, showcasing its capability for real agentic workflows.

The significant difference lies in economics: Opus 4.8 cost approximately $1.50, whereas GLM 5.2 on FriendliAI cost 27 cents, making it about 5.6 times cheaper. This demonstrates the promise of open-weight models to provide frontier quality agents at a fraction of the cost, but the total cost of agents extends beyond just model expenses.

The inference stack needs to evolve to support agentic workloads, which differ significantly from traditional chat usage. In chat, the basic unit is a single request where a user asks a question and receives an immediate answer, with latency defined by response speed. Agents, however, operate on tasks that may involve numerous model and tool calls, running autonomously over time. Users prioritize the completion of the entire task rather than the latency of individual requests, necessitating optimization for end-to-end task completion.

Agent workflows consist of sessions made up of multiple tasks, each typically running in a loop. This loop involves planning, often an LLM call, followed by acting, which might include calling a tool. The agent then observes the result, adds it to the context, and repeats the cycle until the task is complete. This process involves constant alternation between LLM inference and non-LLM tool execution, with the added complexity of agents creating and running sub-agents in parallel.

A critical pattern observed is that consecutive agent steps often share a substantial prefix. Recomputing this shared prefix repeatedly wastes computational resources, highlighting a major optimization opportunity in agentic inference. This implies agents are highly token-hungry, requiring efficient handling of growing context windows.

An example involved explaining the spec decoding framework in VLM using code with GLM 5.2 on FriendliAI, which progresses through multiple stages. Each stage is composed of sub-agents performing numerous inferences and tool calls, potentially running for minutes or hours with tens or hundreds of inference steps. Throughout this process, the shared context continuously expands, and the user's primary concern remains the completion of the entire task, not individual token or core latency.

The challenges in agentic inference include context growth over time, the interleaving of tool work with model cores, and a variable number of model cores that prevents fixed request rate planning. The critical performance metric shifts from single request latency to end-to-end task latency, signifying that agent inference is fundamentally different from chat. FriendliAI addresses these challenges by rebuilding its frontier inference cloud specifically for agentic workflows, focusing on optimizing this end-to-end task latency.

The architecture is built around four key pillars: prefix caching, key-value (KV) cache management, cache-aware routing, and agent-aware optimization. Beneath these pillars, model layer optimizations like sparse attention for long contexts, error reduction techniques, fast kernels, and resilient serving are also integrated to enhance overall performance.

Subsequent steps reuse this cached KV data, processing only the new suffix, which is significantly cheaper than recomputing the prefill. This method improves time to first token and reduces overall compute, becoming more valuable with longer agent tasks. Effective KV cache management is essential for this, utilizing frugal memory management to pack more active context onto each GPU, KV contiguation to reduce memory footprint, and hierarchical caching across GPU memory, host memory, and disks.

FriendliAI extends its caching capabilities beyond GPU limits by employing distributed caching, allowing prefixes to be served across multiple replicas, not just within a single instance. At a global cluster scale, routing becomes critical; a naive load balancer might spread requests evenly but destroy cache locality. FriendliAI's cache-aware router intelligently directs requests to parts of the system that already hold the necessary prefix in cache, transforming a cold prefill into a cache hit. This strategy maintains efficient cache utilization while preventing any single part from becoming a bottleneck, as demonstrated by routing all requests for Task A to the same part for optimal cache locality.

The next frontier in agentic inference is agent-aware optimization, which goes beyond treating each LLM call as independent. Current systems often schedule individual LLM cores without understanding their role within a larger agent program. However, by providing the optimizer with agent-level context, FriendliAI can make more informed decisions. This includes preempting the right work, speculatively prefilling context for anticipated next steps, or making better cache eviction decisions based on the agent's overall progress and requirements. The ultimate goal is to reduce end-to-end task latency, not merely to make individual calls appear fast.

Integrating these optimizations leads to significant performance improvements. FriendliAI demonstrated this by using the GLM 5.2 model with Kilo Code to create a simple mobile game. When running the same task, FriendliAI's model APIs completed the end-to-end task much faster compared to a well-known inference provider. This efficiency is attributed to FriendliAI's agent-centric cloud design, which provides a more robust production agent stack. By combining preferred agents with open-weight frontier models like GLM 5.2, Minimax, and Kimi served on FriendliAI, users gain frontier-quality capabilities with enhanced economics.

FriendliAI currently supports various teams in production, ranging from AI-native startups to large global enterprises, demonstrating its versatility and impact.

FriendliAI's client base includes diverse companies such as Hilo, a popular AI coding tool serving millions, and LG, a global enterprise spanning electronics, healthcare, and energy. Despite their varied businesses, both require fast, reliable, and cost-effective agentic inference. A testimonial from Kilo Code highlighted that after testing several inference providers for both open and closed models over the past year, FriendliAI emerged as a core component of their stack due to its superior performance in GLM 5.2 usage comparisons against other third-party providers and direct usage.

FriendliAI offers flexible consumption models to fit various stack requirements. The Model API provides the fastest way to start, offering core frontier open-weight models via a serverless API. For production workloads requiring isolated deployments and guaranteed SLAs, dedicated endpoints are available. Additionally, the 'Bring Your Own GPU' (BYOG) option allows users to run FriendliAI's inference stack on their own infrastructure, ensuring the same robust performance across three distinct deployment methods.

He encouraged those building agents to try frontier open-weight models on FriendliAI, which can be accessed at friendly.ai within minutes.