Dokpamo Project's Benchmark Leader Motif Eliminated, Evaluation Criteria Spark Controversy

In the government's Original Foundation Model (Dokpamo) project, three out of four participating models advanced to the finals. During this process, Motif, which ranked first in the benchmarks, ultimately failed to proceed to the main competition. The project is expected to continue with the remaining three companies moving on to the next stage.

It has been pointed out that even with high benchmark scores, the performance users experience in actual service environments often falls short of expectations. Despite high benchmark scores, many models cause inconvenience in actual use.

SK Telecom is scheduled to be included in the government's GPU support program again starting in July. Prior to that, the company is known to have trained this AI model at its own expense without government support. This highlights SK Telecom's commitment to technology development.

The Motif team built its foundation model with a relatively small team of around 30 people. This suggests that massive capital investment or personnel size does not necessarily lead to high-performance model development, showcasing the efficient capabilities of a startup.

It was pointed out that the top-ranked model differed depending on the evaluation metric. Motif took first place in the global Artificial Analysis benchmark, but the SK Telecom model reportedly ranked first in the government's undisclosed internal benchmark. This difference demonstrates the variability of results based on benchmark objectives and evaluation criteria.

Such differing benchmark results raise the need to evaluate models across various aspects. Global benchmarks might measure general performance, while government benchmarks could assess performance aligned with specific policy objectives. Ultimately, the ranking varied depending on the criteria used for evaluation.

The primary goal of the Dokpamo project was not merely to select excellent models, but to collectively foster Korea's frontier model ecosystem. In the evaluation criteria, the consortium's composition and cooperative structure played a crucial role, alongside the model's performance itself. This implies that pre-determined evaluation rubrics were used to ensure fairness.

The government comprehensively considered not only model development capabilities but also the ripple effect on domestic industries and the potential for technology sharing and dissemination. In other words, it prioritized synergy for the growth and development of the entire ecosystem rather than the technical superiority of a specific company.

Therefore, it can be interpreted that the consortium, which allows various players to cooperate and strengthen the entire ecosystem, became a major evaluation indicator. This approach reflects the government's commitment to strengthening domestic AI capabilities from a long-term perspective.

Consequently, it is emphasized that this selection process was an evaluation that strategically incorporated policy objectives beyond technical performance.

Even as benchmark scores are continuously updated and new evaluation criteria emerge, there's a growing recognition that high benchmark scores somewhat correlate with the actual intelligence of a model. For example, Motif received high marks for the intelligence of the model itself, while Solar showed strengths in serving optimization.

This difference suggests the possibility of separating inference performance from actual task execution ability. That is, a model with high intellectual capability does not necessarily guarantee optimal service performance in all situations.

Ultimately, benchmarks are tools for measuring specific intelligent aspects of a model, and it must be re-recognized that overall performance in a real service environment is determined by more complex factors.

Perceptions of Dokpamo, a government-led AI project, have changed significantly compared to the past. Initially, there was negativity towards the tax investment and government-led approach, with low expectations for success. This was due to concerns over past failures of domestic and international government-led projects.

However, witnessing good models emerging from countries like China recently, the perspective has shifted towards the possibility that a government-led approach could enable technological self-reliance. The Chinese examples demonstrated that large-scale investment and focus can lead to meaningful results.

Currently, there is a growing recognition that fostering an ecosystem and commercializing models is more important than model development itself. In other words, model development is not an absolute moat, and the key lies in what value is created through it.

This shift in perception suggests that government-led AI projects should be evaluated not just for technological development, but within the broader context of their ripple effects across industries and ecosystem creation.

Considering the rapid pace of AI technology development, it was suggested that models from projects like Dokpamo should be able to be released quickly, not just during limited competition periods, but every one to two months. This emphasizes the importance of maintaining competitiveness through continuous resource input and model updates.

The rapid release cycle stems from the need to provide continuously improved and evolving models in line with the changing speed of the AI market. This ensures technological superiority and meets user expectations.

Ultimately, there is a growing recognition that AI model development is not a one-off project but a process that requires continuous learning and improvement.

Sutton's 'Alberta Plan' methodology for continual learning was introduced. The core of this method is to differentiate the learning rate (step size) per parameter so that an AI agent preserves previously learned values during the learning process while continuously updating new information. That is, important information is preserved by setting the learning rate close to zero, and new parts to be learned are flexibly reflected by keeping the learning rate high.

This approach mimics brain plasticity, helping the model acquire new knowledge without losing existing knowledge through continual backpropagation. This provides a foundation for AI models to gradually improve intelligence over a long period and adapt to changing environments.

As a result, the Alberta Plan is considered an important methodology that helps agents learn and develop efficiently.

Regarding how to consume information, a perspective was presented: 'Don't summarize, read the entire original text, and read expansively.' This focuses on understanding the overall context and details of the original text, rather than simply abbreviating information.

Specifically, reading a transcript, generating an HTML diagram based on it, and repeating an SVG visualization loop to improve readability was mentioned. This helps deepen understanding by exploring information through a zoomable interface.

Current AI models are highly adept at 'hill climbing,' optimizing by incrementally reducing measurable metrics. This approach works efficiently when there are clear quantitative goals.

However, this partial optimization process inevitably falls into the limitation of 'local minimums.' That is, if a model focuses only on specific metrics without an overall view, it may remain at a local optimum instead of reaching a better global optimum. This raises the need for the model itself to assess the overall situation and have a broader perspective.

In parametric font experiments, attempts were made to quantify interdependent spaces by pulling the sections of initial, medial, and final consonants, and then training. Modeling similar to automatic differentiation was attempted through comparative learning with the Noto KR font space, but the limitation was confirmed that results suitable for actual use are still difficult to obtain.

It is pointed out that current experiments can produce results that are merely passable, but this is a different dimension from creating actual usable fonts that require subtle design sensibility. This shows that AI still struggles to fully replace human judgment in unstructured and artistic domains.

The core challenge presented is the process of AI models acquiring design sensibility, or judgment (taste), on their own. Current models can rapidly iterate (hill climb) at certain scales, but the judgment process itself entering the loop is still considered a difficult area.

The main problem is that current models struggle to acquire judgment in non-verifiable domains—areas like design or artistic sensibility where quantification or clear standards are difficult to establish. In these areas, simply optimizing metrics has clear limitations.

The ultimate goal is for models to perform abstraction themselves, making creative and aesthetic judgments without human intervention. This is an important step for AI to evolve from a mere tool to a creative agent.

It was emphasized that philosophy and management issues beyond engineering problems are crucial in building AI systems. In particular, setting objectives and evaluation metrics is the core of automation, and it is argued that the evaluation loop should be structured to allow agents to evolve on their own.

Beyond simple repetitive learning, 'constraints' and 'control planes' must accompany the system to enable agents to make autonomous decisions and learn. This is essential to prevent unintended malfunctions and to continuously generate desired levels of output.

Ultimately, system design must go beyond mere technical implementation to clearly define the values and goals AI should achieve, and include mechanisms for evaluating and improving them. Such a philosophical and managerial approach will enable a long-term sustainable AI loop.

If such a design is achieved, AI systems can function not just as simple tools, but as autonomous entities that learn and evolve on their own.

A new discovery was presented: during agent workflow operations, subjecting agents to repetitive evaluations, or a 'feedback loop,' creates unexpected byproducts that generate value. By having agents repeatedly perform tasks that humans might finish in one go, new solutions or insights that humans might not have discovered can be derived during the process.

At this time, it is important to use a 'Control plane' to enforce outputs and ensure quality. Within such a controlled repetitive process, the network of unexpectedly generated byproducts can change the importance (weight) of actual decisions and positively influence the final outcome.