AI Token Costs Must Be Managed Like Labor Costs

When building an AI agent, both CAPEX (capital expenditure for facilities) and OPEX (operating expenses) incurred during operation must be considered. It was particularly emphasized that the token fees used by AI agents should be recognized as part of labor costs and managed at the executive level. This indicates the increasing necessity to optimize and efficiently utilize token costs from a tokenomics perspective.

AI token costs can be managed efficiently using a quadrant model. The X-axis represents whether token usage was low or high, and the Y-axis represents the performance generated by those tokens, specifically their contribution to the business model. If performance is high and token usage is high, it should be utilized more actively. If performance is low but token usage is high, efficiency improvement measures should be sought.

While the token economy refers to the entire ecosystem that generates AI-related tokens, tokenomics signifies a management approach to saving and efficiently utilizing tokens to contribute to business innovation. Therefore, companies should approach it from a tokenomics perspective, which can be applied to actual management, rather than the broad concept of the token economy.

The process by which tokens enter data centers and are ultimately processed or generated on the user's screen is divided into four main stages. The first is the Prefill stage, where the input prompt is broken down into units that the AI model can understand. The second is the Reproduction stage, where probabilistically related tokens are generated in vector space. The third is the Decode stage, where processed tokens are output in a human-readable format. The final fourth stage is the Agent Control stage, which manages tool calls and execution.

For stages 1 through 3 of AI models, token usage can be controlled to some extent by adjusting inference depth. However, stage 4 agents consume the most tokens because they autonomously decide which tools to call and how many times to repeat. This makes predicting and controlling agent token costs a very difficult area for companies.

Token cost management should be carried out through a three-stage cyclical structure: measurement, evaluation, and management. First, it is crucial to accurately measure how many tokens each department used for what tasks. Based on the measured data, token usage should be evaluated in connection with performance. It is important to note that high token usage does not necessarily equate to high performance. Thus, only with prior measurement can reasonable evaluation and efficient management be achieved.

The token operating matrix manages by dividing the X-axis into token consumption (low/high) and the Y-axis into business contribution performance (high/low). If token usage is high but significantly contributes to company value, more usage should be encouraged, and optimization and efficiency should be considered. Conversely, if token waste is severe regardless of performance, immediate management and improvement are needed. This matrix allows companies to set a strategic direction for token usage.

It is important to quantitatively measure how many tokens were used by which department. However, work performance often involves qualitative aspects that appear over time or are difficult to measure precisely with numbers. Therefore, while token usage should be fundamentally tracked, the value of the work should be assessed through qualitative judgment. A flexible approach is needed, where value is judged after basic measurement, rather than attempting to quantify everything.

As unpredictable token usage leads to cost management issues, technical departments must consider token governance and implement necessary support measures. By adopting AI orchestration and branch processing techniques to enhance token usage efficiency, and developing cost optimization strategies for various models like Foundation Models and vertical Small Language Models (sLLMs) based on business value.

A redeployment strategy for idle personnel resulting from the replacement of existing tasks by AI agents is crucial. If work gaps filled by agents are addressed, these idle personnel should be able to quickly transition to other roles. To help adapt to new roles, agents can be used as auxiliary tools, and ultimately, overall cost efficiency should be improved by integrally analyzing labor costs and token costs.

In the early stages of AI adoption, a division of roles between CTO/CIO and CAIO (Chief AI Officer) is necessary. The CTO/CIO can handle traditional IT infrastructure and security, while the CAIO can be dedicated to AI-specific infrastructure construction and operational management. In the long term, organizational integration may occur as technology evolves, similar to data professional organizations.

In AX (AI Transformation), AI should be managed by distinguishing between AI used by internal employees and AI used by external customers. For tools used commonly by all employees across the company, usage should be encouraged before token management. Conversely, for agents specialized in specific tasks or tools used by individual departments, token usage should be meticulously managed and efficiency improved. The management methods for general-purpose tools and specialized agents must differ.

Currently, most AI token costs are paid to US companies. Therefore, the goal of Korean sovereign AI is to expand domestic token usage and export them. This requires the establishment of independent Foundation Models (Dokpamo) for national infrastructure and key industries. Using Korean tokens is more efficient and aligns with national interests than using foreign tokens.

When using AI models like ChatGPT, costs are paid to OpenAI. OpenAI, in turn, needs to lease data centers for AI model training and inference serving operations, making data center rental costs a major factor in rising expenses. This is why OpenAI is moving to internalize its infrastructure through the 'Stargate Project.' Even if data centers are attracted to Korea, token payments occur to the operating entity, so proprietary technology and infrastructure construction are crucial.

Since exporting tokens themselves is realistically difficult, prioritizing the competitiveness of AI services is essential. Developing and exporting AI services like ChatGPT that appeal to the global market is the most desirable approach. Especially, vertical AI services combined with K-culture (webtoons, K-pop, etc.) can have great potential in overseas markets. A proprietary export strategy based on reasonably priced and reliable Korean LLMs (Large Language Models) is needed.

Excellent data center operational capabilities that can reduce operating costs directly translate into export competitiveness in the global market. Efficient data centers with lower operating costs can be a strong advantage for exporting Korea's AI infrastructure and services to overseas markets.

China is exporting AI by utilizing 'full-stack' infrastructure, combining Chinese-made semiconductors and data centers to efficiently operate affordable AI services. This strategy is particularly effective in expanding markets, especially in developing countries. Thanks to OpenAI models, China's AI infrastructure is also being exported.

Data centers are not just clusters of semiconductors; they drive the co-growth of various related industries such as power facilities, cooling solutions, energy (SMR, renewables), and water resource management. In particular, efficient cooling technology to dissipate the immense heat generated by data centers directly impacts token production efficiency, which is a crucial factor in enhancing the overall competitiveness of AI infrastructure.