DatologyAI Generates 12 Trillion Synthetic Tokens

Current scaling laws for AI models demonstrate that exponentially increasing compute and data resources only yield linearly better performance, presenting a significant challenge.

Existing usable text data on the web is limited, making it insufficient for training frontier-level models and necessitating new approaches to data acquisition.

Spanning web, math, and code domains, the team recently completed a synthetic data run totaling approximately 12 trillion tokens.

This extensive data generation project encompassed diverse domains, including web, math, and code, addressing broad AI training needs.

The company assists both startups and large enterprises by identifying and processing high-quality data points through its proprietary recipe to enhance model training.

DatologyAI's BeyondWeb synthetic data recipe utilizes rephrasing and restructuring of existing high-quality data.

Synthetic data allows for significantly less token usage, showing 2.7x to 5.3x efficiency gains compared to datasets like Neotron Synth and Cosmopedia for equivalent results.

For instance, a 3B model trained with BeyondWeb can achieve the same performance as an 8B model using traditional data sources, demonstrating substantial resource savings.

This method involves identifying top-tier data points within customer datasets and then transforming them through a specialized pipeline, preventing models from merely replicating their original training distributions.

Rephrasing existing content helps to diversify the data while maintaining its inherent quality.

A pool of GPUs runs various models to process these documents after the core synthetic generation pipeline prefixes them with specific prompts.

These operations leverage various models deployed across a pool of GPUs to create the synthetic data outputs.

Ensuring consistent throughput from these GPU operations proves to be a practical challenge.

This fragmented setup resulted in slow workflows, frequent errors, and underutilization of valuable GPU resources, limiting the company's operational velocity.

The strategic goal was to consolidate both training and synthetic data generation onto a single, cohesive Kubernetes-based infrastructure to maximize GPU efficiency and streamline operations.

The current architecture features an orchestration layer within a single Kubernetes cluster, utilizes Spark for data processing, and employs Ray for distributed task execution.

S3 serves as the primary storage layer, ensuring compatibility across diverse cloud environments such as AWS, GCP, and on-premises setups.

The H100 cluster on AWS is now managed through EKS on HyperPod, which streamlines operations by eliminating the need for manual data backfilling between disparate clusters.

An internal job scheduler now orchestrates all training, evaluation, and data generation tasks within one integrated workflow, enhancing efficiency.

The available high-quality web text suitable for pre-training large language models is estimated to be approximately 30 trillion tokens, a finite resource.

DatologyAI addresses this scarcity by aggregating existing high-quality datasets like FineWeb and Dolma as foundational inputs for its synthetic generation processes.

Engineering challenges at this massive scale include managing S3 metadata, handling GPU failures, optimizing cross-cluster scheduling, and fine-tuning inference performance.

DatologyAI implemented batching requests to reach the 1,000-request limit per call, successfully reducing metadata processing time for trillions of tokens from 11 days to 2 hours.

Individual S3 requests frequently led to throttling, causing GPUs to remain idle and severely slowing down overall processing.

Initially, fetching metadata for trillions of tokens was projected to take an impractical 9 to 11 days, highlighting a critical bottleneck.

DatologyAI mitigated this by implementing batching requests to S3, leveraging the API's capability to bundle up to 1,000 requests per call, which dramatically improved performance.

This optimization reduced the metadata processing time from an estimated 11 days to a mere 2 hours, greatly accelerating data readiness.

Large-scale synthetic data generation runs inevitably encounter hardware failures, including "bad GPUs" and machine crashes, disrupting progress.

System interruptions during long computational tasks, such as an 8-hour partition job, can result in substantial losses of work and require re-execution from scratch.

DatologyAI implemented a dual strategy of right-sizing partition sizes and frequent checkpointing to address these instabilities, ensuring that jobs can resume from recent progress points.

Effective checkpointing requires idempotency, allowing retried jobs to safely overwrite any partially completed outputs from previous attempts without data corruption.

Managing both GPU and CPU resource requirements simultaneously across multiple clusters presented a complex bin packing problem for DatologyAI's operations.

Initial challenges arose when jobs had sufficient GPU capacity but lacked the necessary CPU resources in the available pools.

The team addressed this by establishing dedicated resource pools for Spark, which is CPU-heavy, and for Ray heads and workers, allowing for more granular control.

Atomic scheduling now ensures that both CPU and GPU requirements are satisfied concurrently for Ray jobs, thereby enabling seamless and predictable scaling.

Disciplined benchmarking and tuning of vLLM and SGlang parameters proved critical for optimizing inference efficiency in synthetic data generation.

By adjusting specific flags within vLLM, DatologyAI achieved a significant 40% gain in throughput.

The team focuses on batch inference, meticulously managing batch sizes and employing speculative decoding techniques to enhance performance.

Systematic grid searches over hyperparameters consistently lead to improved performance, reflecting the value of iterative optimization.

Recent large-scale jobs generated approximately 7 trillion web tokens and another 5 trillion tokens for math and code, fulfilling the needs of various clients.

The team continues its research to refine synthetic data recipes across multiple domains, including web, multilingual, math, code, and legal data.

Future efforts will involve detailed comparisons of data generation differences specific to each of these diverse data types.