Kimi K3, One-Third the Price, Could Actually Cost More

Even when the same prompt is entered into AI models, different model-specific settings critically influence the outcome. Lee Jong-beom, an adjunct professor of AI, emphasized that the differences observed when the same prompt is used across various models now stem from these settings. Since output can vary significantly depending on each model's default settings, it is crucial for users to understand and adjust them.

Most demo videos published on the official websites of AI model developers are 'cherry-picked' results, showcasing only the best outcomes. Kang Su-jin stated, "On the official website, there are some pretty, cherry-picked demos that make you wonder if this is really possible." She pointed out that it is highly probable that the same results as in the demos will not be achieved when the same prompt is entered, which can mislead users about the model's actual performance.

Before the experiment, Kang Su-jin set two hypotheses. The first hypothesis was that if given identical prompts, the three models would respond according to their default settings. The second hypothesis was to verify whether the desired output could be achieved under the same conditions through prompt correction, regardless of model ranking. These hypotheses aimed to confirm the essential performance differences between AI models and the importance of prompt utilization strategies.

Each AI model exhibited differences in setting adjustments based on its inference method. Claude Fable 5's 'Thinking' function could not be turned off and was always active by default. In contrast, GPT-4o allowed control over the use of the 'Thinking' function. Kimi K3's 'Thinking' function was active by default, but detailed adjustments were only roughly possible. These unique characteristics of each model made it difficult to control experiments under identical conditions.

Kang Su-jin's team built their own platform, 'Compare LRMG,' to compare AI model performance. This platform was developed using Claude Code and designed to demonstrate Claude Fable 5, GPT-4o, and Kimi K3 simultaneously in real-time. It was utilized as a key tool for objective comparative experiments.

Detailed prompt configuration was crucial for successful game output. Beyond simple feature implementation, game operability such as steering and camera switching was added, and it was set to be implemented as a single file within the browser without external tools. Kang Su-jin emphasized the importance of language, UI, and sound generation and UI control, explaining that these elements significantly impact the fun and completeness of the game.

The comparative experiment was conducted using only pure LLMs (Large Language Models) without external tool integration. The models under investigation were GPT-4o, Claude Fable 5, and Kimi K3. Notably, Kimi K3 only offered three options—Low, High, and Max—making subtle adjustments difficult. The same prompt was entered into each model to compare the differences in output.

In the identical prompt experiment, Kimi K3 exhibited errors during output generation. In contrast, GPT-4o produced results in a 2D game format, and Claude Fable 5 yielded relatively decent outputs. Kang Su-jin noted, "GPT-4o, then Claude Fable 5, Kimi K3... Kimi K3 now has an error," evaluating Kimi K3 as having disappointing quality compared to the cherry-picked demos released by the developer.

Kimi K3 was found to undergo about 2 to 10 times more inference processes compared to other AI models. The number of tokens required for output generation exceeded 70K, which is a highly inefficient figure compared to other models that complete inference at levels between 0.9K and 5.5K. Kang Su-jin pointed out, "It generally increased by two to three times, or even five to ten times, compared to other models," emphasizing Kimi K3's issue of excessive inference volume.

Kimi K3 boasts the advantage of a token price that is one-third of other models. However, due to excessive inference volume, the actual cost could be even higher. Reporter Hong Jae-ui questioned, "The token price was said to be about one-third. But if inference is 70 times more, then it's more expensive." This illustrates that if the inference volume differs by 70 times, even with a cheaper token unit price, the total cost will increase significantly. This case exemplifies that model optimization and inference volume control are key to cost-efficiency.

In this experiment, attempts were made to keep the prompt identical and control the output format to a single HTML. Kang Su-jin explained, "What we controlled was that the prompts were the same, and the output format could be seen as a single HTML." However, due to policy differences among model providers, 'Thinking mode' control was not possible, and it was acknowledged that there were areas beyond control due to the unique characteristics of each model.

The conclusion was that even with the same prompt, making the best use of a model's existing settings is the most efficient way to achieve results. Kang Su-jin stated, "If the prompt is the same, the best approach is to utilize the model's settings as effectively as possible to suit your needs." She added that by writing detailed and comprehensive prompts, sufficient results can be obtained with low inference intensity, and combining optimal settings for each model is currently presented as the most efficient way to use AI.

Kimi K3 lacked flashy images or visual effects when implementing games. However, Kang Su-jin evaluated, "While it couldn't implement flamboyant images or refreshing visual effects, a precise game was completed that aligned with my design and the evaluation rubric I had set." This suggests that Kimi K3 might have strengths in terms of game's fundamental design and logical completeness rather than visual elements.

For presentation creation, Kang Su-jin formulated a prompt requesting an interior proposal PPT on the theme of 'Mid-Century Modern Curated Goods Store.' She specified nine sections for a 16:9 slide deck in a final editable format and requested a premium architectural interior studio style. Additionally, she instructed that the text on the slides should be very short and stylish, emphasizing visual effects by making it image-centric.

In the presentation creation experiment, clear performance differences were observed among the models. GPT-4o produced high-quality PPTs using external real-image URLs, while Claude Fable 5 and Kimi K3 implemented them primarily with shapes without external tools, resulting in lower quality. Kang Su-jin asked, "Why is that? Why… is it the difference in tool usage?" explaining that the ability to use external tools proved to be a significant distinguishing feature among models. GPT could create better results by utilizing design source website URLs.

Kang Su-jin emphasized that AI models do not automatically handle everything and that specific instructions explicitly mentioning the use of external tools are necessary to obtain better results. She explained, "If you just tell it to use external URLs or design tools to make it prettier, even a frontier model might not give us the desired output. So, you need to fill that in the prompt. It doesn't do everything automatically." She stressed the importance of actively instructing the model to use tools within the prompt.

Interestingly, different AI models showed a common preference for certain design components. Kang Su-jin revealed that when she used the description 'kinetic sculptures and mechanical movements,' all three models generated very similar-feeling outputs. This phenomenon occurs because the models' generative scope, interpretable from the prompt, is very similar, suggesting that AI models have similar perceptions of specific visual patterns.

The core of AI model utilization is not simply model ranking but rather selecting a model that aligns with one's goals and meticulously configuring its settings accordingly. Kang Su-jin advised, "Rather than these rankings, I think a model that achieves your goal is a good one... Even if you input the same prompt, it differs by model, and there's a difference in settings, so meticulously configuring settings to suit your task will be the shortcut to effectively utilizing inference models." This provides practical guidance for users to effectively leverage AI.