🔍 Read the full analysis: Scaling Indic Pretraining: 2.8 Trillion Tokens And An Open Data Generation Pipeline on ThorstenMeyerAI.com
TL;DR
NVIDIA and AI4Bharat at IIT Madras say they plan to release Nemotron-pretraining-Indic-v1, a corpus with about 2.8 trillion retained tokens across 22 Indian languages, alongside the GPU-accelerated pipeline used to produce it. The corpus and pipeline are not yet available; the release date, licensing terms and detailed quality metrics have not been specified.
NVIDIA and AI4Bharat at IIT Madras say they plan to release Nemotron-pretraining-Indic-v1, a corpus containing about 2.8 trillion retained tokens across 22 Indian languages, together with an open, GPU-accelerated data-generation pipeline. The proposed resource is aimed at teams training language models in Indic languages, where the announcement says limited training data remains a constraint; the release date has not been provided.
The reported inventory contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer. Native-script text covers all 22 languages, while romanized text—Indic-language content written in Latin letters—is included for 12. The project describes these as 34 language-script configurations. It cautions that records can include translations and script variants of shared source documents, so the token total is not a count of unique English documents.
The generation workflow begins with English documents from 15 Nemotron-CC-v2 subsets, including web text, question-and-answer material, mathematics and code. Using NVIDIA’s NeMo Data Designer framework, the process translates documents, evaluates translation quality, repairs outputs that score poorly and converts some accepted translations into Latin script. The project says records retain source identifiers and that checks compare outputs with inputs, including checks intended to preserve equations, tables and code.
The teams describe the release as planned rather than available now. They say users will be able to train directly on the corpus or adapt the prompts, workflows and quality checks for other source material. The project also says continued pretraining for Nemotron 3.5 Super VL used 10 billion Indic tokens from a 12-language subset alongside naturally authored Indic text. That reported training use is separate from the corpus inventory: the announcement does not say the model was trained on all 2.8 trillion tokens.
More Training Material for Indic Models
A large, openly released corpus could give model developers a broader starting point for training across languages that have fewer readily available resources than English. The announced source mix includes technical material, code and question-and-answer text, which may be useful for models designed to explain technical topics or respond to questions in Indic languages. However, the announcement provides no independent evaluation showing how much the corpus improves model quality or performance.
Publishing the generation pipeline alongside the text could let developers inspect or adapt the method instead of relying only on a finished dataset. Source identifiers and input-output checks may support traceability, while reusable workflows could help teams work with other document collections. The real value will depend on what code, documentation, licenses and quality results accompany the eventual release.
The corpus is not described as a substitute for naturally authored Indic-language writing. Because the process starts with English documents, translations may carry the source material’s subject matter and framing without representing the full range of local knowledge or cultural context. The project says naturally authored Indic data remains essential.
Indic language NLP training datasets
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Building on AI4Bharat’s Indic Data Work
AI4Bharat is an Indian-language research lab at IIT Madras. The announcement points to Sangraha, its open pretraining corpus for 22 Indian languages, as part of the existing foundation for open Indic-language model research. Nemotron-pretraining-Indic-v1 would add a large body of translated material and a described process for generating more from curated English sources.
The project says its 22 languages are those listed in India’s Constitution and that they are the first languages of 97% of Indians, citing nearly 1.2 billion people in the 2011 Census. That population figure refers to the cited census, not a current population estimate. The announcement also quotes AI4Bharat head Mitesh Khapra: “We do not have enough data for Indian languages.”
The reported token totals are measured using the Gemma-4 tokenizer and include accepted rewrites and script variants. Token counts vary by tokenizer and should not be treated as a count of documents, a measure of unique content or proof that a model has trained on the full inventory.
“We do not have enough data for Indian languages.”
— Mitesh Khapra, head of AI4Bharat
GPU-accelerated data generation tools for AI
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Release Terms and Quality Details Pending
The announcement does not specify when or where the corpus and pipeline will be published, or the terms governing redistribution and adaptation. It also does not provide per-language token breakdowns, detailed quality scores, translation acceptance thresholds or independent assessments of translation fidelity.
It remains unclear how the corpus will affect downstream model performance: the announcement gives no comparative results, and it does not establish how much synthetic translation data should be combined with naturally authored Indic text in future training. Since the corpus may contain translations and script variants of shared sources, the reported token count should not be read as a measure of unique source material.
large multilingual text corpora for machine learning
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Publication and Independent Review
The next milestone is the planned publication of the corpus and generation pipeline. Once available, researchers and developers can review the dataset, workflow, documentation and licensing, and assess the reported quality checks against their intended uses. No release date has been announced.
Further evaluation will be needed to establish how well translations preserve meaning across languages and subject areas, how consistently quality controls work, and whether training with the corpus improves model results. The project’s eventual release materials may also clarify per-language counts, acceptance criteria and how users can adapt the workflow to their own sources.
AI model training resources for Indian languages
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What are NVIDIA and AI4Bharat planning to release?
They plan to release Nemotron-pretraining-Indic-v1, a corpus with about 2.8 trillion retained tokens across 22 Indian languages, along with the GPU-accelerated pipeline used to generate it.
How much of the corpus is romanized?
The reported inventory includes 224.6 billion romanized tokens, alongside 2,601.0 billion native-script tokens. Romanized text is included for 12 of the 22 languages. Counts use the Gemma-4 tokenizer.
Is the corpus available now?
No release date has been specified. The announcement describes the corpus and pipeline as planned releases, not resources that are already available.
Does the token total represent unique documents?
No. The project says records may include translations and script variants of shared source documents. The reported total is a tokenizer-based token count, not a count of unique documents.
Has the project shown that the corpus improves model performance?
The announcement does not provide independent performance evaluations or comparisons. It reports that a 12-language subset contributed 10 billion Indic tokens to continued pretraining for Nemotron 3.5 Super VL, alongside naturally authored Indic text.
Primary source: Hugging Face · via ThorstenMeyerAI.com
