Inception and MBZUAI Launch AraGen Leaderboard: A New Benchmark for Arabic LLMs

Inception, a G42 company specializing in AI-native innovations, has partnered with the Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) to launch the AraGen Leaderboard. This groundbreaking framework is designed to revolutionize the evaluation of Arabic Large Language Models (LLMs). Powered by the newly developed 3C3H metric, AraGen sets a new standard for evaluating Arabic Natural Language Processing (NLP) with a robust, transparent, and holistic approach that prioritizes factual accuracy and usability.

Addressing the Needs of 400 Million Arabic Speakers

Serving the diverse linguistic and cultural needs of over 400 million Arabic speakers worldwide, the AraGen Leaderboard fills critical gaps in AI evaluation. It features a meticulously constructed dataset tailored to the complexities of the Arabic language and its cultural context. The framework tackles key challenges such as benchmark leakage, reproducibility issues, and the absence of holistic metrics, ensuring both core knowledge and practical utility are evaluated effectively.

Redefining Benchmarks with Generative Tasks

AraGen introduces generative tasks, marking a significant shift in evaluating Arabic LLMs. Unlike traditional leaderboards, which relied on static and likelihood-based accuracy metrics, AraGen’s dynamic evaluation framework captures real-world performance. This shift emphasizes the importance of practical utility and fosters innovation in Arabic AI, demonstrating the transformative potential of this new benchmark system.

Comprehensive Evaluation Across Six Dimensions

The AraGen Leaderboard evaluates models using six core dimensions: correctness, completeness, conciseness, helpfulness, honesty, and harmlessness. With 279 carefully designed questions spanning tasks such as Arabic grammar, general Q&A, reasoning, and safety, the framework addresses the specific needs of Arabic speakers. Regular quarterly updates ensure the leaderboard remains relevant and adaptable, inviting public submissions to drive continuous improvement and growth in the Arabic AI ecosystem.

“The AraGen Leaderboard redefines Arabic LLM evaluation, setting a new standard for fairness, inclusivity, and innovation,” said Andrew Jackson, CEO of Inception. “By addressing the gaps in previous benchmarks and introducing generative tasks, the platform empowers researchers, developers, and organizations to create culturally aligned AI technologies. AraGen ensures transparency, reproducibility, and trust while advancing the global NLP landscape.”

Empowering Organizations with Transparent Insights

AraGen provides organizations with detailed performance insights, enabling them to select models that best align with their goals. By minimizing the need for extensive internal testing, the framework ensures cost-effectiveness while building trust through its transparent and reproducible methodology. This innovative approach positions AraGen as a key enabler of growth and refinement within the Arabic AI landscape.

"AraGen is a major step towards open, collaborative, and reproducible evaluation of large language models for Arabic, with a focus on their text generation capabilities. This contrasts with popular leaderboards, which rely primarily on multiple-choice questions. Moreover, AraGen is a dynamic board with new questions every three months, which makes it much harder to game compared to existing leaderboards,” said Professor Preslav Nakov, Department Chair of Natural Language Processing and Professor of Natural Language Processing, Mohamed bin Zayed University of Artificial Intelligence (MBZUAI).”

“Our goal was to create a benchmark that introduces generative task evaluation with a strong emphasis on transparency, reproducibility, and a rigorous measurement of models’ performances,” said Ali El Filali, Machine Learning Engineer at Inception and lead author of this work. “By evaluating models across multiple dimensions to assess both factuality and usability, the AraGen Leaderboard provides actionable insights for diverse NLP tasks. This empowers the Arabic AI community to develop safe and high-performing models for real-world needs that are important to our region. Moreover, AraGen sets a global example by demonstrating how AI benchmarks can prioritize equity and inclusion for underrepresented languages. It’s a step toward ensuring no language or culture is left behind in the AI revolution.”

Conclusion: Pioneering the Future of Arabic NLP

The AraGen Leaderboard is a landmark development in Arabic AI, setting new standards for model evaluation and fostering innovation. By addressing existing challenges and prioritizing user-centric metrics, it establishes a solid foundation for advancing Arabic NLP and empowering the region’s AI capabilities.