INTRODUCTION

1 a: Coverage

Artificial Analysis (https://artificialanalysis.ai) provides its gathered benchmarks for free through a REST API. Their coverage spans a range of benchmarks (around 10) based on their own AGI benchmark. Artificial Analysis also provides a detailed dashboard that you may consider interacting with in your own research.

The graph is dynamic. Hovering over a point will display a bar with further information.

1 b: Best models

This table shows the best performing models across all benchmarks.

1 c: Finding the balance

You can see that some models perform well on the benchmark at considerably less cost than the best performing model.

1 d: Exploring

Lots of actionable information on one graph.

Click on a point to reveal the model name and benchmark. The click also fixs all the datpoints for a given benchmark in focus. Click again to reset the view.

2 a: Human Judgement

Arena’s allow humans to choose (online) between different models on a given task. The table shows the results of a human judgement benchmarks.

2 b: Ranking

Opensource models are ranked on a leaderboard based on human judgements. Some of the Open Source models are near the top of the leaderboard and might be useful once we take into account the need for Digital Sovereignty or keeping your data within your own data center.

2 b: Ranking Safety

CISCO LLM Security Leadership board

Comprehensive model safety and security rankings, including single-turn score, multi-turn score, and detailed metrics.

See

Other available Security benchmarks:

• CVE=Benchmark Evaluating AI agents on real world web vulnerabilities and exploits collected from National Vulnerability Database. CVE-Bench includes 40 critical-severity Common Vulnerability and Exposures (CVE) with the reference automatic exploits available on requests.
• TOSSS A benchmark that measures the ability of LLMs to choose between secure and vulnerable code snippets.

2 c: Updating and Routing

LLM-Stats provides a live feed of model updates, including newly added models to the ecosystem. We can use this type of information to keep our benchmarks up to date and to identify promising new models for testing.

LLM-Stats also provides a routing service that automatically directs API calls to the best-performing model. We can use such a service to ensure that we always use the near best available model for specific capabilities. For a more complex community scorecard, it would require a discussion with a provider of routing services.

3 a: More information

Here a list of best models for a given benchmark is shown. The dataset is from Epoch.AI. The dataset is updated regularly.

Certain models are robust across a series of benchmarks and therefore specific capabilities.

3 b: LLM input / Output

OpenRouter operates one of the largest AI inference platforms in the world, generating an authoritative empirical dataset relied upon by government agencies, academic researchers, major industry analysts, and global media outlets.

Here we use their dataset on the input and output modality of the models that they measure. For policymakers, by doing so, we can keep track of the expanding range of modalities and their popularity. This is an important consideration as you select benchmarks relevant to the LLMs in your organisation.

Conclusions

Choose AI models based on our values is possible via a set of current benchmarks. Much of the necessary benchmarks are available, but …

It is Nuanced:

• Not all benchmarks are created equal.
• Different domains have different capability needs.
• Some benchmarks are more accessible than others.
• We might run our own benchmarks for missing models.
• It is only an estimate, but the higher (and more popular) the benchmarks, the more reliable the estimate.
• There are many model variations (Size, fine-tuning, etc.), and the benchmarks capture only a subset of them.
• Fine-tuning can improve models for our specific tasks, so we will have more models to test in the future.
• Benchmarks become less reliable over time due to Saturation effects such as models trained on benchmark data.

AI alignment requires a nuanced understanding of the details for us to enact safely. However, the payoff is great.

We do need a community process. Here is a mockup of the dashboard that supports such a process.