Context
A central challenge for AI benchmarking in education is that the landscape moves faster than any single institution can track. New models, benchmarks, and scoring methodologies appear weekly. Scores saturate, benchmarks become contaminated, and yesterday’s leaderboard is today’s historical artefact.
This page explores the infrastructural and community requirements for building a sustainable tool that gathers benchmarks, stores historical data, and generates accessible dashboards for the education community.
Key Takeaways
How Other Projects Do It
Several existing projects demonstrate viable patterns for community-driven benchmark collection. Each offers lessons for an education-focused tool.
MLCommons Science Working Group — YAML Benchmark Catalogue
The MLCommons Science Working Group maintains a versioned YAML catalogue of scientific AI benchmarks. Each benchmark entry includes citations, FAIR (Findable, Accessible, Interoperable, Reusable) scores, and ratings. A Python toolchain validates entries, runs URL checks, and generates Markdown, LaTeX, and MkDocs outputs. Contributions are submitted via pull request and reviewed by maintainers before publication.
EveryEvalEver — Crowdsourced Benchmark Results Database
EveryEvalEver, led by researchers at IBM, Hugging Face, and Technical University of Munich, collects evaluation results into a standardised JSON format with four metadata blocks: source provenance, model information, generation configuration, and metric semantics. The database already contains 22,000+ model results across 2,200 benchmarks, translated from 31 different evaluation formats. Auto-translation tools exist for HELM, lm-eval-harness, and Inspect AI. Each submission receives a unique ID and cannot be deleted — conflicting results remain visible in the metadata.
Website · Paper · Database on HuggingFace
HuggingFace Community Evals — PR-Based Model Results
HuggingFace’s Community Evals project adds structured evaluation results to model repositories using the .eval_results/ YAML format. Anyone can submit results via pull request, and results appear on both model pages and benchmark leaderboards. It supports extraction from model cards, the Artificial Analysis API, and HuggingFace Papers. A Claude Code skill exists for automated evaluation management.
Repository · Benchmark Tracker
EleutherAI LM Evaluation Harness — The De Facto Standard
The LM Evaluation Harness is the most widely used framework for standardised LLM evaluation. It supports 60+ academic benchmarks with hundreds of subtasks, local and API-based models, publicly available prompts for reproducibility, and a YAML-based task configuration system. It serves as the backend for HuggingFace’s Open LLM Leaderboard and has been used in hundreds of published papers and by organisations including NVIDIA, Google, and Mosaic ML.
Repository · Task Guide · Integration Guide
Measurement Data Bank (AIMS Foundation) — Psychometric Response Matrices
The Measurement Data Bank at Stanford’s AIMS Foundation curates 146 AI evaluation benchmarks as standardised (subjects × items) response matrices for Item Response Theory (IRT) and psychometric analysis. Each benchmark has a self-contained build.py that downloads raw data, builds the matrix, generates a heatmap, converts to a PyTorch payload, and uploads to HuggingFace Hub. The pipeline is fully reproducible and covers 92 ready benchmarks with per-item response data.
MoltBench — Crowdsourced Benchmarks Built by AI Agents
MoltBench is an executable benchmark built by AI agents, for AI agents. Agents submit tasks, a different AI agent peer-reviews each submission through blind adversarial testing (three adversarial tests per submission), and accepted tasks are merged. The review process is fully deterministic — no LLM-as-a-judge — using file content, exit codes, and JSON structure checks. Tasks trace back to real agent behaviours observed on the MoltBook platform (770,000+ autonomous LLM agent interactions).
AI Observatories — Governance Models for Education
Several observatory initiatives provide institutional frameworks that an education benchmarking tool could complement or contribute to.
EdTech Hub AI Observatory & Action Lab
The AI Observatory & Action Lab (supported by FCDO) scans AI trends in education, tests real-world applications, and provides tailored guidance for decision-makers in low- and middle-income countries. The Observatory produces weekly “signals” — curated observations of AI-in-education developments — while the Action Lab runs pilots and engagements directly with governments. A dedicated workstream on AI Tool Benchmarking assesses foundational AI models against education criteria such as pedagogical standards and support for special educational needs.
Observatory Page · Learning Brief Series · Tools Portal
UNESCO Observatory on AI in Education for Latin America and the Caribbean
The UNESCO Observatory is the first regional platform anchored in the UN system dedicated to AI in education. It brings together 33 Ministries of Education, universities, research centres, and technology partners. Its lines of action include evidence generation, ethical and regulatory framework development, teacher training, and pilot initiatives. The Observatory operates on the principle that “AI cannot govern education; education must govern AI.”
Global AI Observatory (Society & AI)
The Global AI Observatory tracks how governments integrate AI into K-12 and higher education across 56 nations. Each country profile documents official initiative names, issuing bodies, policy types, and direct links to primary source documents. Data is sourced exclusively from official government publications and manually verified.
AI-for-Education.org Benchmarks
The AI-for-Education.org project, in collaboration with Fab AI and others, has developed the Pedagogy Benchmark — the world’s first benchmark testing whether LLMs can pass teacher exams (based on Chilean Ministry of Education questions) — and the Visual Reasoning Benchmark for primary school visual mathematics. Results are published on a public Education Leaderboard.
Pedagogy Benchmark Leaderboard · About the Benchmarks
Proposed Tool Architecture
Drawing on the patterns above, a community-driven education AI benchmarking tool would likely include:
Data Layer
- A YAML or JSON schema for benchmark metadata and results (following EveryEvalEver and MLCommons Science patterns)
- Versioned storage with immutable result IDs and transparent conflict handling
- Standardised response matrices enabling psychometric analysis (following the Measurement Data Bank pattern)
Automation Layer
- GitHub Actions workflows for periodic data collection from live sources (Artificial Analysis, HuggingFace, Epoch AI)
- PR-based contribution model with automated validation checks (following HuggingFace Community Evals and MLCommons Science patterns)
- Auto-translation adapters from major evaluation harnesses (lm-eval-harness, HELM, Inspect AI)
Presentation Layer
- Static dashboards generated from collected data (the current Jekyll + flexdashboard approach)
- Benchmark cards documenting what each benchmark measures, its limitations, and known contamination status (following the EvaluationCards pattern)
- Education-specific overlays mapping benchmark results to pedagogical criteria
Governance Layer
- Open-source repository with documented contribution guidelines
- Transparent maintenance policy including deprecation of contaminated benchmarks
- Institutional partnership with an education-focused organisation for sustained resourcing
Maintenance
Maintenance would need to be continuous due to the rapid progress of AI capabilities and model versions. Scores saturate, benchmarks become contaminated, and new variants appear monthly. The level of effort required would only be justified by a large-scale community process, perhaps through the administration of a central AI observatory dedicated to the education sector.
In the meantime, the content of this website should be viewed as an experiment intended to inform and build literacy and interest. It demonstrates what is technically possible while acknowledging that sustainable operation requires institutional backing.
Evolution
The convergence of several trends makes this an opportune moment for an education AI benchmarking tool:
-
Standardisation efforts are maturing. EveryEvalEver’s JSON schema, MLCommons’ Croissant format, and HuggingFace’s .eval_results/ format provide interoperable foundations.
-
Education-specific benchmarks are emerging. The Pedagogy Benchmark, the AI-for-Education.org leaderboard, and the EdTech Hub’s AI Tool Benchmarking workstream are early signals of demand.
-
AI observatories provide institutional homes. UNESCO, EdTech Hub, and Society & AI are building the governance infrastructure that a benchmarking tool could feed into.
-
The cost of evaluation is rising. Sharing and reusing prior results becomes more valuable as frontier model evaluations grow more expensive. EveryEvalEver’s estimated $370,000 reproduction cost makes the case for open, reusable results databases.
Given the interest many have in this crucial area of Alignment, please reach out if you have interesting contacts, research, or wish to offer support.
This page was generated by an AI assistant on 28 July 2026. It synthesises publicly available sources including the MLCommons Science Working Group benchmark collection, EveryEvalEver, HuggingFace Community Evals, the EleutherAI LM Evaluation Harness, the AIMS Foundation Measurement Data Bank, MoltBench, the EdTech Hub AI Observatory & Action Lab, the UNESCO Observatory on AI in Education, the Global AI Observatory (Society & AI), and the AI-for-Education.org benchmarks. Verbatim claims and cost estimates should be checked against the primary sources.