Sarvam's 3B Model Sets Document AI Record

With 3 billion parameters, the model processes English and 22 official Indian languages while outperforming competitors 100 times its size on a single GPU.

The model is designed to support English and 22 official Indian languages, demonstrating the ability to outperform models 100 times larger in document AI tasks.

Digital archives contain ample Indian linguistic material, yet the Common Crawl corpus used for frontier model training holds well under 1% representation for these languages.

This underrepresentation is not due to a lack of data but rather a challenge in digitizing existing Indian language content.

Machine learning for Indian scripts requires knowledge extraction because complex, fused Unicode characters and low-resource status render simple text extraction ineffective.

Indian scripts present unique challenges due to their complex, fused Unicode characters, which differ significantly from how machines typically process text, and the fact that most of the 22 languages are classified as low-resource.

Sarvam Vision adopts a state-space architecture, departing from conventional transformers, to address computational inefficiencies inherent in sequence processing.

Transformers face quadratic compute complexity with respect to sequence length because every token must look at every other token.

State-space models (SSMs) overcome this by maintaining a single state and updating it token by token, ensuring a constant memory footprint and linear compute scaling.

The first stage, texton pre-training, involves 13 trillion tokens across English, Indian languages, text, math, and code, aiming to leverage prior knowledge to resolve ambiguous or smudged text effectively.

Subsequent stages include continual pre-training on 300 million image-text pairs (Stage 2), supervised fine-tuning on 100 million OCR samples across 22 Indian languages and English (Stage 3), and finally Reinforcement Learning (RL) in Stage 4 to push performance beyond fine-tuning limits, encompassing tables, equations, and handwriting.

To overcome the lack of ready-to-use labeled data for many of the 22 Indian languages, Sarvam creates synthetic and document data.

The team also implements continuous improvement pipelines based on evaluation performance, establishes proprietary evaluation metrics to verify true state-of-the-art results, and prioritizes end-user utility as a key metric for model performance.

OCR tasks are machine-verifiable, making correctness a machine-readable problem suitable for automatic reward scoring through unit tests.

This approach allows for scalable reinforcement learning pipelines for document intelligence by reinforcing average value targets and exploring agentic vision capabilities for future releases.

Sarvam's model achieved state-of-the-art results on CRB bench with 84.3 and Omnidoc bench with 93.2, demonstrating superior performance in document AI.

Crucially, it holds an unbeaten lead in 22 Indian languages when compared to frontier models like Gemini, GPT, and Opus, and is currently used by enterprises in banking, insurance, and government sectors to process over 35 million pages, with API, on-prem, and agentic platform deployment options available.

Multilingual training across 22 Indian languages boosts performance in English, highlighting a significant cross-lingual benefit for the architecture.

While there are some parallels with coding assistance applications, this area does not constitute the primary focus or periphery of this particular work.

Synthetic data is critical for post-training stages such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL).

For RL, using real-world documents that are complex enough to facilitate the creation of detailed unit tests and various reward mechanisms is preferred, allowing for rewards based on criteria like character error rate, table structure, math equations, or grammar rules, which iteratively improve model performance.

This extensive dataset spans a vast historical period, from the 1800s to the modern day, and includes diverse content types such as prose, poetry, literature, finance, and tables across various document layouts in Indian languages.

In OCR, the primary goal is high-fidelity extraction, meaning the model must accurately reproduce any mistakes present in an image, rather than making arbitrary changes such as transliteration.

No existing frontier model, whether closed-source or open-source, performs well with complex Indian language documents.

This creates a critical need for localized, sovereign AI infrastructure, particularly as India seeks to digitize paperwork for its 1.4 billion people, with the Sarvam model already digitizing 35 million pages since its launch.

Existing frontier models, both closed and open source, struggle to effectively process complex Indian languages, highlighting the critical need for specialized solutions to address document intelligence problems for such a large population.

Achieving State-of-the-Art (SOTA) performance necessitates high-quality data, which is especially vital for AI sovereignty.

Government and insurance sectors prioritize AI sovereignty to ensure robust data privacy and control, enabling organizations to precisely track where data is sent, when it is used, and how it is processed.

Since its launch, the Sarvam model has already facilitated the digitization of 35 million pages in India, aligning with these sovereignty goals by maintaining 40% Indic languages in its training data, with the remainder in English, math, and code.

The training data for the model comprises 40% Indic languages, with the remaining 60% consisting of English, math, and code.

Current deployments across various Indian states focus on processing regional languages and mixed-language documents, and the team has established a pipeline to integrate real-world feedback from these deployments directly into future training cycles for continuous improvement.