Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

NVIDIA, in collaboration with IIT Madras and its AI4Bharat lab, is set to release Nemotron-pretraining-Indic-v1, a dataset comprising approximately 2.8 trillion tokens across 22 Indian languages. This initiative aims to provide a substantial resource for Indic pretraining, covering the first languages for 97% of Indians. Accompanying the dataset is an open, GPU-accelerated data generation pipeline, which includes prompts, LLM workflows, and quality checks, allowing developers to adapt it for their own data generation needs.

This release is the first to offer a GPU-accelerated pipeline for generating Indic language data, unlike previous efforts that focused solely on model parameters.