Hugging Face Urges Shift to Skill-Based Vision AI

Merve Noyan argues that developers should transition from directly using Vision Language Models (VLMs) to building end-to-end vision applications that leverage specialized skills.

She notes that excessive reliance on large VLMs often fails to deliver real-time performance, frequently yielding only 30-40 frames per second.

Furthermore, dedicated models like RT-DETR consistently demonstrate superior performance in specific tasks, while restrictive licenses such as AGPL 3.0, used in models like YOLO, necessitate a shift toward Apache 2.0 alternatives for broader adoption.

"I don't want developers to directly use vision language models anymore. And uh I want every single developer to start uh building vision langu vision applications end to end."

Merve Noyan

Noyan introduced 'web vision,' a toolkit inspired by Mazar's work, which enables vision models to function as tools for agents, simplifying complex vision tasks.

Model selection within the toolkit prioritizes Apache 2.0/MIT licenses, strong performance benchmarks, and personal preference to ensure usability and adaptability.

The toolkit integrates a 'vibe training' section designed to automate data labeling and evaluation, providing a streamlined workflow for developers who may have limited initial data.

"this toolkit has my favorite models as tools uh so that you can give it to your agent because your coding agent is a bit of a clueless computer vision engineer basically."

Merve Noyan

The presented pipeline supports local and remote execution via Hugging Face infrastructure, including batch processing and serverless routing through Qwen 2.5 7B labeling and an ensemble of Gemma 2 9B and Llama 3.2 Vision judges.

The system supports both local and remote execution via Hugging Face infrastructure, including efficient batch processing and serverless routing options.

The architecture employs Qwen 2.5 7B for initial labeling, followed by an ensemble of judges, including Gemma 2 9B and Llama 3.2 Vision.

Ensembling these smaller judge models has been shown to be more effective than relying on a single large model for accurate assessments.

Current methods often involve asking VLMs or LLMs to assign scores to data, a process that proves ineffective, particularly when dealing with models of varying sizes serving as judges.

In response to this inefficiency, Noyan pivoted to training RFDTR (RT-DETR) models instead, which showed more promising results.

The workflow also supports training on datasets without pre-existing labels by implementing an annotation-on-the-fly method.

Instead of directly processing raw tokens from Qwen, the system overlays bounding boxes onto images to enhance the accuracy of judge verdicts.

The judge then evaluates these visual cues against label descriptions that are generated by coding agents.

The pipeline merges verdicts based on minimum agreement, rather than strict consensus, to ensure a sufficient dataset size for training.

Noyan also performs manual approval of the label descriptions generated by the agents to maintain quality.

The coding agents employed in the pipeline were developed using Opus 3.5 and GLM 5.2 to optimize performance and efficiency.

Most of the models integrated into the system adhere to Apache 2.0 licenses, with the LFM model being a notable exception.

The entire training pipeline is designed to be highly cost-efficient, with an estimated total expenditure of approximately three to four dollars.

Infrastructure options include serverless platforms like DeepInfra or Together, and Hugging Face jobs are utilized for the judging processes.

The system was rigorously tested on tasks including road sign detection and document parsing, which involved identifying images, tables, and signatures.

The mean average precision exceeded 50, indicating favorable performance despite a minor discrepancy when compared to ground truth data.

Notably, for document parsing, the model demonstrated superior generalization capabilities, successfully detecting signatures that the initial Qwen annotations had missed.

This effectiveness is largely attributed to the robust performance of the RT-DETR backbone used in the model.

LFM models tend to be excessively restrictive, often resulting in high rejection rates, which makes achieving strict consensus among judges difficult.

To ensure an adequate amount of data for generalization, the system adopted a minimum agreement threshold rather than requiring full consensus.

However, human validation of agent-generated prompts remains a critical bottleneck that affects overall data quality, highlighting the ongoing need for human oversight.

Advanced coding agents like OPUS 4.8 often execute inappropriate data augmentations, such as horizontal flips on traffic signs or jitter on traffic lights, which one expert notes is "clueless as a computer vision engineer as well as it misses common sense."

These agents can perform inappropriate data augmentations, such as applying horizontal flips to traffic signs or jitter to traffic lights.

Such errors lead to corrupted datasets, necessitating manual intervention to control and configure augmentation settings precisely.

The repository leverages Hugging Face benchmarks to allow for comprehensive comparison of models based on their size and performance metrics.

Falcon Perception, a model with 600 million parameters and an Apache 2.0 license, is particularly highlighted for its ability to handle complex, multi-referential segmentation tasks that standard SAM models cannot.

The toolkit also incorporates the Sapiance family for robust pose and human detection, alongside Moon Dream 3 and Grounding DINO for advanced zero-shot detection capabilities.

Integration with Roboflow's Supervision and Tracker libraries is also included, providing powerful functionalities for bounding box and instance tracking.

Natural language prompting often proves insufficient for industrial parts that are challenging to describe accurately.

As an alternative, image-guided detection is proposed, where a reference image is used to identify specific objects across multiple files.

Experiments are currently underway with Intersection over Union (IoU) merging to refine the accuracy of bounding box generation by models, rather than relying on simple binary acceptance criteria.

A dedicated vision repository provides extensive resources covering fine-tuning, quantizing, and multi-modal modeling, along with comprehensive transformer task guides.

Hugging Face skills offer pre-configured computer vision and infrastructure tasks, streamlining the training process for developers.

While the prospect of self-improving Vision Language Models (VLMs) is compelling, the immediate focus remains on facilitating task-specific model training for developers.

The primary goal is to empower developers to efficiently train and deploy these models locally or on edge devices, addressing current practical needs first.