You will find context specific explanations by clicking on the buttons such as this one.

This is a network diagram of 250 benchmarks. You can click on a particular node and it will give you the benchmark name. At the centre of each group of benchmarks is the type of benchmarking. You can also drag and drop.

Number of benchmarks per theme

You can see from the diagrams that:

• There are popular and less popular themes.
• Themes are at times interconnected.
• There are a lot of benchmarks.

What you should also know:

• The network is evolving rapidly with new benchmarks being developed and older ones discarded.
• There are new abilities emerging that impact education, such as emotive understanding, which might affect the popularity of specific benchmarks.
• Benchmarks have associated datasets which may be used to train new models, which in turn might mean that scoring becomes less reliable over time.

You can currently find the related dataset at Evidentlyai. The dataset is not exhaustive and is based on the data available at the time of analysis (250 benchmarks, Cutoff late 2025).

AI alignment is the effort to ensure that artificial intelligence systems act in ways that match human values, goals, and intentions. In other words, it’s about making sure AI behaves as we want it to

As the field matures and we become accustomed to deploying scorecards, we will gain a more fine-grained understanding of the risks and benefits of using benchmarks and scorecards. Using AI to keep track of AI is a helpful start. Consider using a prompt such as the following one to monitor evolution:

You are an expert AI policy and education technology analyst. Your sole mission is to keep me continuously up-to-date on the risks and benefits of using AI benchmarks to track, evaluate, and compare the performance of AI systems deployed in the education sector (AI tutors, automated grading tools, personalized learning platforms, adaptive content generators, misconception detectors, etc.).

Every time I ask for an update (or on a schedule if you support it), deliver a concise, balanced, evidence-based briefing that covers:

1. Recent developments in how benchmarks are being used or critiqued in educational contexts (new studies, papers, reports, leaderboards, or real-world EdTech deployments from the last 1–3 months).
2. Benefits of benchmark-driven evaluation, such as:
   • Objective, standardized comparisons across models
   • Guidance for schools/districts/EdTech companies when selecting tools
   • Ability to measure progress on education-specific capabilities (pedagogy, student misconception detection, essay scoring, etc.)
   • Acceleration of model improvement for teaching and learning tasks
3. Risks and limitations, including but not limited to:
   • Benchmark saturation or overfitting that does not translate to real classroom outcomes
   • Lack of correlation with actual student learning gains, teacher effectiveness, or long-term retention
   • Cultural, linguistic, or socioeconomic biases (especially in low- and middle-income countries)
   • Narrow focus on test-like performance instead of pedagogical depth, emotional support, or equity
   • Potential for “teaching to the benchmark” instead of genuine educational value
4. Well-known benchmarks you must explicitly reference and analyze in every update (including how they are currently performing, any new critiques, and their relevance to education):
   • MMLU (Massive Multitask Language Understanding) – general knowledge across academic subjects
   • GSM8K (Grade School Math 8K) – mathematical reasoning with word problems
   • Pedagogy Benchmark and SEND Pedagogy Benchmark (AI for Education / Fab AI)
   • Visual Maths Benchmark (early-grade visual math problems)
   • ASAP 2.0 (automated scoring of student persuasive essays – Learning Agency leaderboard)
   • Eedi Misconception Annotation Project (identifying student math misconceptions)
   • Any newly emerging education-specific benchmarks (e.g., PERSUADE dataset extensions or multi-modal classroom benchmarks)
5. Practical implications and actionable recommendations for:
   • Educators and school leaders
   • Policymakers and regulators
   • AI developers building education tools
6. Always cite sources with dates and links (academic papers, leaderboards, reputable reports, or news). Structure the response clearly with headings (Recent Developments, Benefits, Risks, Benchmark Spotlight, Recommendations). Keep the tone neutral, evidence-driven, and forward-looking. If no major new information exists since the last update, explicitly say so and summarize the current state of the field.

Begin every response with: “AI Benchmarks in Education – Update as of [current date]”

The graph is from Epoch AI, which creates a scorecard based on several well-understood, popular, and high-quality benchmarks.

Change is not a constant; it is actually accelerating across several fronts. Inference is getting cheaper and faster. Model release cycles are decreasing in time. Capabilities are emerging, and our collective understanding of how to use and orchestrate AI for meaningful results is improving and scaling.

EPOCH AI releases its findings and datasets regularly. Yor can find the datsets here

Europe is falling behind in AI frontier capabilities. We can use Chinese (open source) and US models, but as a community, we should add measurable risk factors such as digital sovereignty to our composite scorecard.

Artificial General Intelligence(AGI) refers to a type of artificial intelligence that has the ability to understand, learn, and apply knowledge across a wide range of tasks at a level comparable to human intelligence. Unlike narrow AI, which is designed for specific tasks, AGI would be capable of performing any intellectual task that a human can do. AGI raises ethical and safety concerns.

Change is a constant… Implications

1. How do we keep track of the changes that are coming?
2. Play to our sectoral strengths and avoid weaknesses.
3. Adopt emergent properties and apply a safetynet to mitigate emergent risks.

The diagram is from AGI Definition, which points to a paper on benchmarking various aspects in AGI. The diagram illustrates the different dimensions of AGI and how ChatGPT performs across those dimensions. The trend is obvious, capabilities are improving, but unevenly. There are still qualities that need addressing.

This section explains the concept of an observatory to monitor rapid changes associated with the implications of the rapid evolution of AI. Without a central observatory, organisations would need to repeatedly track these changes, which may not have sufficient attention or resources to do so effectively.

Source

Bloom’s taxonomy of needs or an updated version allows for a more nuanced understanding of the different levels of cognitive processes and how they relate to learning outcomes. It can help educators design more effective learning experiences and assessments that target specific cognitive skills and promote deeper learning. The taxonomy also allows us to divide up AI-related activities into pedologically relevant zones of sophistication and benchmark each zone.

In general, the higher the zone, the more sophisticated. Google, for example, trains and then measures its models based on the different. This allows them to track progress and identify areas for improvement.

Other taxonomies are also possible, but Bloom’s is a good starting point because it is widely used and understood in education. It also allows us to categorise benchmarks in a way that is meaningful for educators and students.

Source Npuls LA team

Learning Analytics data is an accelerator the education sector.

Learning Analytics (LA) defines a life cycle around designing educational nudges that improve student outcomes based on online digital traces such as those left by interacting with AI.

Npuls has had a team ofLearning Analytics experts review the literature and create a comprehensive overview of best and worst practices in learning analytics. This overview can be used to inform the design of benchmarks that are relevant for learning analytics, and to identify areas where more research is needed.

Correctly structured data gathering can provide the training data for models and benchmarks. If the Dutch cannot compete on building frontier models, then perhaps we should focus on specialising in domain-specific models. These models need fine-tuning.

Only a small amount of Learning Analytics data is needed to fine-tune models. Making the process practical, cheap, and fast. This is a crucial finding because it means that we can create domain-specific models that are tailored to the needs of educators and students, without needing to invest in large amounts of data or computational resources. We will however need to invest in building our expertise and processes and align our institutional policies to agile data driven practices.

Source

At the different levels of Bloom’s taxonomy, different applications of AI can be used to support learning outcomes. For example, at the lower levels of the taxonomy, AI can be used to provide personalised feedback and support for basic skills such as memorisation and comprehension. At the higher levels of the taxonomy, AI can be used to support more complex cognitive processes such as analysis, synthesis, and evaluation.

As the deployment of AI systems becomes more sophisticated, we will be looking at more complex interactions between generative AI agents at times glued together with scripts and custom text documents providing instructions in written form. Overtime benchmarking will focus on the composite systems and applications, rather than on the individual models. This is because the composite systems are what educators and students will be interacting with, and they will need to be evaluated based on their overall performance and impact on learning outcomes, rather than just on the performance of the individual models that make up the system.

Source Google are using taxonomies to align and improve their models and have quantified the improvements via benchmarking. This leads to a virtuous cycle of improvement, where the benchmarks inform the development of the models, and the improved models lead to better benchmarks. This is an accelerator example of how benchmarking can drive progress in AI development and deployment in education.

You are an AI Research Scientist specializing in Large Language Model Evaluation. Your task is to map contemporary LLM evaluation suites to the hierarchical levels of Bloom’s Taxonomy and provide concise, expert‑level analysis.

Instructions

1. Scope

• Include static benchmarks (e.g., ARC, MMLU, HellaSwag, GSM8K).
• Include dynamic/agentic benchmarks (e.g., AgentBench, SWE‑Bench, WebArena, ToolBench).
• When relevant, distinguish between task format and cognitive demand.

2. Cognitive Mapping

For each benchmark:

• Assign its primary Bloom’s Taxonomy level.
• Note any secondary cognitive skills (e.g., multi‑step reasoning, planning).
• Use Bloom’s hierarchy:

  • Lower‑Order Thinking: remembering, understanding, basic application

  • Higher‑Order Thinking: analysis, evaluation, synthesis/creation

3. Critical Evaluation

Provide a concise, evidence‑based discussion of limitations in using these benchmarks to measure human‑like cognition, including:

• Overfitting and benchmark contamination
• Shortcut exploitation
• Limited ecological validity
• Lack of metacognition measurement
• Differences between human conceptual reasoning and LLM statistical patterning
• Partial improvements from agentic benchmarks and remaining gaps

4. Output Format

Use:

• Hierarchical headings
• Bullet points
• Clear taxonomy labels Structure the output as:

1. Overview of Bloom’s Taxonomy
2. Static Benchmark Mapping
3. Dynamic/Agentic Benchmark Mapping
4. Cross‑Benchmark Comparison
5. Limitations for Human‑Like Cognition
6. Summary Table

5. Constraints

• Provide concise reasoning, not detailed internal thought processes.
• Do not reveal chain‑of‑thought; instead give short, direct explanations.
• Maintain a scientific, research‑grade tone.
• Avoid generic descriptions; focus on cognitive implications.

Here is an example set of capabilites that can be benchmarked. This is not an exhaustive list.

The implication is that the more capabilities are deployed, the more benchmarks we need to keep track of. This is because each capability has its own set of risks and benefits, and we need to ensure that we either monitor effectively and/or buy in models that have been benchmarked appropriately. This is a big ask for individual institutions, which is why we need a central observatory to do this work for us and make the results accessible and actionable for educators.

LLM Leadershipboard display models against various benchmarks or composite sets of benchmarks. A number of the boards enable community updates and feedback processes.

Examples

• The Pedagogy Benchmark
• Vellum AI
• Onyx

Saturation of LLM benchmarks refers to the point where large language models consistently score extremely high on widely used evaluation tests—so high that the benchmarks no longer meaningfully differentiate between models or reveal their weaknesses. When a benchmark becomes saturated, it stops functioning as a reliable measure of progress.

For the educational sector, this matters because schools, universities, and policymakers increasingly rely on benchmark results to judge whether AI tools are trustworthy. The sector needs fresh, robust, and domain‑specific evaluations to ensure AI supports learning rather than distorting it.

Communities of experts can help address this issue by continuously developing new benchmarks that are more complex, realistic, and aligned with educational goals.

Humanities Last Exam (HLE) is a shinning example on how to engage a community of experts similar to the ones working within Npuls and keep benchmarks fresh. We can certainly learn a lot form their approach.

Humanities Last Exam

Quote:” AI capability is evaluated based on benchmarks, yet as their progress accelerates, benchmarks become quickly saturated, losing their utility as a measurement tool. Performing well on formerly frontier benchmarks such as MMLUand GPQA are no longer strong signals of progress as frontier models reach or exceed human level performance on them.

In partnership with the Center for AI Safety, HLE addresses the problem of benchmark saturation by creating Humanity’s Last Exam (HLE): 2,500 of the toughest, subject-diverse, multi-modal questions designed to be the last academic exam of its kind for AI. HLE is designed to test for both depth of reasoning (eg. world-class mathematical problems) and breadth of knowledge across its subject domains, providing a precise measurement of model capability. Current frontier models perform poorly on HLE with low accuracies, and systematically exhibit uncalibrated overconfidence in their answers.

Searchable questions were removed by the following procedure. A question is potentially searchable if a model with search tools answered correctly, but answered incorrectly without search. Each of these potentially searchable questions was then manually audited, removing any that were easily found via web search. We used GPT-4o mini/GPT-4o search and Perplexity Sonar models in this procedure.”

We will need to engage a community to weight abilities and values and to keep the scorecard fresh and relevant. This is a mockup of what a community process could look like. Simply ask and weight based on the feedback. The process can be repeated as often as needed and incrementally.

Community Scorecard Summary

A practical issue for generating scorecards is that not all models are tested against all benchmarks. The missingness issue leaves us with several possible, at times complementary approaches to benchmarks, including:

• Opportunism: We choose benchmarks and leadership boards based on their popularity or relationship to leadership boards.
• GAP analysis: We act on the missing data and run community-selected benchmarks for the models we care about.
• Embrace our individuality: We design a composite scorecard and only deploy models tested against the scorecard.
• Test per situation: Provide a framework to design your own scorecard based on already existing metrics.

https://artificialanalysis.ai provides an API to access data on LLM models, including their evaluations and pricing.

Kaggle is a popular platform for data science competitions and datasets. The dataset used in this analysis contains information about various LLM benchmarks, including their performance metrics and the licenses under which they are released.

Kaggle datasets are generally static and age quickly for LLM benchmarks, which are rapidly evolving. However, they can still provide valuable insights into the landscape of LLM benchmarks at a given point in time. The dataset used here is from April 2026, and it includes a ‘license’ column that indicates the type of license for each benchmark.

This pie chart shows the distribution of licenses among the LLM benchmarks in a Kaggle dataset. The ‘license’ column in the dataset indicates the type of license under which each benchmark is released. The chart helps to visualize how many benchmarks are available under each license type, which can be important for understanding the accessibility and usage rights of the benchmarks.

A lot of LLM family have slightly modified open source licenses, which can be confusing.The pie chart helps to clarify the distribution of licences and highlights the prevalence of of proprietary licence types in the LLM benchmark landscape.

Library databases such as Web of Science allow you to search for terms across large sets of research and, by applying those datasets, analyse the returned data. In this case, using a known library in R bibliometrix.

In this case, we search for the terms

dataset AND Github AND AI AND benchmark*

Within Seconds you can get an overview of your area of interest

And with little effort you can drill down to track the emergence of new benchmarks and the names of specific well-performing LLMs, the health of the community of researchers, which journals and authors to track, and much more.

The screengrab shows how citations are distributed within a specific document. In this case: Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation. Content analysis of highly rated papers or theses is helpful for what is called back propagation (while searching), where, from one good paper, you can find others.

From content analysis, you can also extract relevant entities, such as the location of datasets used for benchmarking. The datasets are reusable in educational contexts, such as fine-tuning domain-specific models or serving as a safety net against unsafe or inefficient practices.