📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva-3B, a European sovereign language model trained from scratch with 2.5 trillion tokens, achieved low performance on academic benchmarks. This challenges assumptions about the scale needed for country-specific AI models.
Italy’s Minerva-3B, a large-scale European sovereign language model trained from scratch on 2.5 trillion tokens, scored just 4.9% on the INVALSI Italian school-exam benchmark, revealing significant challenges in achieving deep language understanding through scale alone.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research and supercomputing infrastructure, trained models with up to 7 billion parameters using approximately 50% Italian data. Despite the substantial investment and open release of weights, data, and code, the model’s performance on complex academic tests was near chance, contradicting expectations that larger, native-language models would excel in country-specific knowledge.
According to the Minerva research team, the evaluation results suggest that dataset size and parameter count are more critical for handling complex language tasks than the proportion of native-language data alone. The low score on the INVALSI benchmark raises questions about the effectiveness of scaling up native-language data without proportionate increases in model size and training resources. The project exemplifies the European approach of building sovereign AI infrastructure and highlights the ongoing debate over the optimal scale and investment required for effective country-specific language models.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
AI language model training datasets
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
large-scale language model development kit
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
Engineering a Small AI Language Model: Training, Evaluation, and Deployment Without Myth
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI model evaluation tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Scale and Investment in European Sovereign LLMs
The results from Minerva challenge the assumption that training large, native-language models from scratch guarantees deep understanding of country-specific knowledge. Despite Italy’s significant investment and infrastructural support, the low academic benchmark score indicates that scale alone may not be sufficient. This finding has broad implications for European AI strategies, suggesting that future projects may need to consider even larger models or different training methodologies to achieve desired levels of language comprehension and knowledge depth. The case underscores the importance of aligning resource investment with realistic expectations about model capabilities, especially for complex language tasks.
European Sovereign LLM Development: Approaches and Challenges
Italy’s Minerva project represents a deliberate choice to train a language model from scratch on a massive dataset, contrasting with other European efforts like Portugal’s AMÁLIA, which focused on continuation training of multilingual models with smaller native-language datasets. Minerva’s approach was supported by Italy’s national AI strategy, leveraging supercomputing resources and open data policies. While Minerva’s models outperform comparable multilingual models on Italian benchmarks, their low performance on complex academic tests reveals limitations in scale-driven approaches. This development highlights ongoing debates within the European AI community about the optimal balance between investment, data, and model size for country-specific language models.
“Our results suggest that dataset composition and model scale need to be balanced carefully to achieve meaningful country-specific knowledge.”
— Minerva research team
Unresolved Questions About Scaling and Model Effectiveness
It remains unclear what specific scaling thresholds are necessary for models trained from scratch to achieve high performance on complex, country-specific benchmarks. The impact of different training methodologies, dataset compositions, and model architectures on these thresholds is still under investigation. Additionally, the generalizability of Minerva’s findings to other languages and domains has not yet been established, and ongoing research aims to clarify these issues.
Next Steps for European Sovereign AI Development
The Minerva team plans to continue iterating on their models, including experiments with larger parameter counts and refined training data strategies. Future benchmarks will assess whether increased scale can improve performance on academic and complex language tasks. European AI policymakers and researchers are likely to reevaluate investment levels and technical approaches based on these findings, potentially shifting focus toward scaling models further or exploring alternative training paradigms.
Key Questions
Why did Minerva perform poorly on the Italian academic benchmark?
The evaluation suggests that while the model was trained on a large dataset, scale alone was insufficient to develop deep country-specific knowledge, indicating the need for larger models or different training strategies.
Does this mean European sovereign LLMs are not viable?
Not necessarily. The results highlight the challenges and suggest that scale must be complemented with other strategies. Ongoing research aims to identify optimal approaches for effective country-specific models.
How does Minerva compare to other multilingual models?
Minerva outperforms comparable multilingual models on Italian benchmarks but underperforms on complex academic tests, indicating that native-language training alone does not guarantee comprehensive understanding.
What are the implications for future AI investment in Europe?
The findings imply that European projects may need to scale models further or adopt new methodologies to achieve desired knowledge depth, influencing future funding and strategic decisions.
Source: ThorstenMeyerAI.com
