Dutch Benchmarks
Context
There are a number of benchmarks tuned specifically to the Dutch language, ranging from grammatical evaluation and reading comprehension to logical reasoning, bias assessment, and information retrieval. The landscape is evolving quickly, with older benchmarks being superseded by newer versions or incorporated into larger frameworks.
Before using any benchmark listed here, consider the following:
Comprehensive Benchmarks
These benchmarks cover multiple task types and provide a broad assessment of Dutch language model capability.
DUMB — Dutch Model Benchmark
DUMB (de Vries et al., EMNLP 2023) is a balanced benchmark of nine Dutch language tasks spanning word-level, word-pair-level, sentence-pair-level, and document-level tasks across low-, medium-, and high-resource settings. It includes four tasks that were previously unavailable in Dutch.
Tasks: Part-of-Speech Tagging (Lassy Small), Named Entity Recognition (SoNaR-1), Word Sense Disambiguation (WiC-NL), Pronoun Disambiguation (DPR), Causal Reasoning (COPA-NL), Natural Language Inference (SICK-NL), Sentiment Analysis (DBRD), Abusive Language Detection (DALC), Question Answering (SQuAD-NL).
Key innovation: DUMB introduces Relative Error Reduction (RER) as a comparative metric — measuring how much a model improves over the BERTje baseline — rather than using raw mean scores. This makes scores comparable even as the set of models changes over time.
Leaderboard · Repository · Paper
MTEB-NL — Massive Text Embedding Benchmark for Dutch
MTEB-NL (Banar et al., 2026) is a comprehensive embedding benchmark for Dutch, extending the international MTEB (Massive Text Embedding Benchmark) with tasks and datasets tailored to Dutch. It was originally developed as a separate repository (mteb-nl-dev) before being incorporated into the main MTEB project.
The benchmark covers 7 task types across 40 datasets: classification, multi-label classification, pair classification, reranking, retrieval, clustering, and semantic textual similarity. It includes both existing Dutch datasets and newly created ones.
Alongside the benchmark, the authors released E5-NL, a series of compact yet efficient Dutch embedding models, and a training dataset compiled from available Dutch retrieval resources.
MTEB-NL Leaderboard · Paper (ACL 2026) · arXiv
BLiMP-NL — Linguistic Minimal Pairs for Dutch
BLiMP-NL is a corpus of 8,400 Dutch sentence pairs intended for grammatical evaluation of language models. Each pair consists of a grammatical sentence and a minimally different ungrammatical sentence. The corpus covers 84 paradigms grouped into 22 syntactic phenomena.
The dataset improves on the original English BLiMP in two key ways: (1) native speakers created and checked all minimal pairs, and (2) human validation used both acceptability ratings (7-point scale, 30+ participants per item) and reading times (self-paced reading). The authors evaluated 14 Dutch transformer models, with GPT-2-based models showing superior overall performance.
Dataset · Paper (MIT Press, 2025)
ScandEval — Scandinavian and Germanic Language Evaluation
ScandEval is a leaderboard and Python package for evaluating LLMs on Scandinavian and Germanic languages, including Dutch. It provides a standardized evaluation framework with multiple benchmark tasks. The Open Dutch LLM Evaluation Leaderboard (formerly at HuggingFace Spaces) is no longer maintained; users are referred to ScandEval for Dutch evaluations instead.
ScandEval includes Dutch tasks such as conll_nl (named entity recognition), dutch_social (sentiment analysis), squad_nl (question answering), mmlu_nl (knowledge), hellaswag_nl (commonsense reasoning), and others.
ScandEval Dutch NLG · Repository
Task-Specific Benchmarks
These benchmarks target specific capabilities or domains.
SQuAD-NL — Dutch Question Answering
SQuAD-NL is the Dutch version of the Stanford Question Answering Dataset. It consists of reading comprehension questions posed on Wikipedia articles, where the answer is a text span from the passage. SQuAD-NL is included as one of the nine tasks in the DUMB benchmark and is also used by ScandEval for Dutch evaluation.
Dutch LLM Evaluation Blog Post · Original SQuAD Paper
SICK-NL — Dutch Natural Language Inference
SICK-NL is a dataset targeting natural language inference in Dutch, obtained by translating the English SICK dataset (Marelli et al., 2014). Having a parallel inference dataset allows comparison of both monolingual and multilingual NLP models for English and Dutch on the same tasks.
BEIR-NL — Dutch Information Retrieval Benchmark
BEIR-NL (Lotfi et al., 2025) is a Dutch-translated version of the BEIR benchmark for zero-shot evaluation of information retrieval models. It covers 14 datasets across multiple retrieval tasks and domains, automatically translated using Gemini-1.5-flash. The authors evaluated BM25, multilingual dense ranking models, and reranking models.
Key findings: BM25 remains a competitive baseline, only outperformed by larger dense models; combining BM25 with reranking matches the best dense models. The study also found that back-translation causes a performance drop, highlighting the limitations of translation-based benchmarks and the need for native Dutch resources.
Paper (BUCC 2025) · HuggingFace
SLR-Bench-Dutch — Logical Reasoning in Dutch
SLR-Bench-Dutch is the Dutch-language version of the Scalable Logical Reasoning Benchmark. It provides 19,000+ inductive reasoning tasks across 20 complexity levels grouped into 4 tiers (basic, easy, medium, hard). Each task includes a natural language prompt in Dutch, an executable validation program for automatic evaluation, and a latent ground-truth rule.
The benchmark supports curriculum learning — models can be trained and evaluated across progressively harder levels — and is fully automatically generated, requiring no human annotation.
Dataset on HuggingFace · Paper (NeurIPS 2025)
DutchCrowS — Dutch Stereotype Benchmark
DutchCrowS is the first benchmark specifically designed to evaluate Dutch stereotypes in LLMs. It adapts the CrowS-Pairs dataset by selecting, translating, and adapting data, then extending it with newly crowdsourced Dutch-specific stereotypes across nine social groups (831 sentence pairs total).
Findings: models explicitly trained on Dutch data (GEITje-7B-Ultra, EuroLLM-9B-Instruct) exhibit higher stereotyping scores than general multilingual models — suggesting that training on a language introduces bias specific to that language.
MSc Thesis (Utrecht University, 2025)
Social Bias Benchmark (MinBZK) — Dutch Hiring Bias
The Dutch Ministry of the Interior and Kingdom Relations (MinBZK) developed a benchmark to assess social bias in LLMs within a hiring decision setting, focusing on gender and country of origin. It systematically generates thousands of template-based prompts and measures bias through acceptance rate differences.
Evaluated models (GPT-4o-mini, Claude 3.5 Haiku, Geitje-7B-Ultra, EuroLLM-9B-Instruct) all exhibited social bias to some extent. The benchmark is publicly available under an EUPL-1.2 license.
Repository · Paper (LREC 2026)
ANS Challenge Set — Dutch Grammatical Challenge
A challenge set based on the Algemene Nederlandse Spraakkunst (ANS), the comprehensive resource of Dutch prescriptive grammar created by linguists. The study collected acceptability judgments from Dutch native speakers and evaluated both transformer-encoder and transformer-decoder Dutch LLMs. Encoder models showed near-perfect accuracy, but sensitivities for specific sentences differed between models and humans, partially due to mismatches between reference grammar and actual Dutch usage.
FinGEITje — Dutch Financial Evaluation
FinGEITje is a Dutch financial evaluation benchmark for assessing LLM performance on financial domain tasks in Dutch. It includes model weights, an evaluation package, and task definitions specific to the Dutch financial sector.
Model Code · Evaluation Package · Paper
Benchmarking Zero-Shot Text Classification for Dutch
De Langhe et al. (2024) studied zero-shot text classification performance for Dutch, benchmarking several models and approaches. This work provides insights into how well models handle Dutch text classification without task-specific training data.
Psychometric Evaluation of Dutch LLMs
A systematic evaluation of 14 Dutch transformer models examined how well their surprisal estimates account for reading times in sentence, paragraph, and book reading corpora. GPT-2-based models (particularly gpt2-small-dutch) demonstrated superior overall performance. The study replicated the inverse scaling trend (smaller models performing better for reading time prediction) and the linear effect of surprisal on reading times for Dutch.
Paper (Behavior Research Methods, 2025) · Data & Code
Dutch Language Models (with Evaluation Context)
Several models have been specifically trained or adapted for Dutch and are evaluated using the benchmarks above.
RobBERT Family
RobBERT (Delobelle et al.) is a Dutch version of RoBERTa, with multiple versions:
- RobBERT v2 — Dutch vocabulary, trained on OSCAR 2019
- RobBERT-2022 — Updated with larger OSCAR 22.01 corpus
- RobBERT-2023 — State-of-the-art: achieves +18.6 RER on DUMB benchmark (large variant, 355M params), outperforming all existing Dutch and multilingual models at time of release. Uses Tik-to-Tok model conversion strategy from English RoBERTa.
- RobBERTje — Distilled variants (40M–74M params) for faster inference
RobBERT Page · Repository · RobBERT-2023 Paper
BERTje
BERTje (de Vries et al., 2019) is the first Dutch pre-trained language model, a Dutch version of BERT-base. It serves as the baseline model for the DUMB benchmark’s RER metric. Although now outperformed by larger models, it remains a widely used reference point.
GEITje Family
GEITje (Rijgersberg, Vanroy) is a family of Dutch LLMs based on Mistral 7B, continued-pretrained on Dutch corpora (GigaCorpusNL, MADLAD-400). Variants include:
- GEITje-7B — Base Dutch model
- GEITje-7B-chat / chat-v2 — Instruction-tuned versions
- GEITje 7B Ultra — First preference-optimized Dutch LLM, using SFT + preference optimization
Repository · GEITje Ultra Paper
Fietje
Fietje (Beersmans et al., 2025) is a family of small language models specifically designed for Dutch, based on Phi-2 (2.7B parameters). Despite its small size, Fietje demonstrated competitive results with larger models at time of release. It is fully open-source: weights, datasets, training, and evaluation code are all publicly accessible. Its instruction-tuned and chat variants (Fietje 2B Chat) showed substantial improvements, outperforming larger 7B models like GEITje and Tweety on multiple benchmarks.
Leaderboards and Evaluation Infrastructure
Dutch LM Evaluation Harness
A fork of EleutherAI’s LM Evaluation Harness with support for Dutch evaluation benchmarks (e.g., SQuAD-NL) and Dutch prompts. Can evaluate models loaded via transformers, vLLM, commercial APIs, and PEFT adapters.
ITHAX NL Benchmark
ITHAX provides a practical AI benchmark for Dutch, testing models on language interpretation, in-context learning, programming, translation, and more. Based on 35+ questions across 9 categories, drawn from real client use cases. Includes models ranging from small to large, open to commercial.
European LLM Leaderboard
A collection of multilingual evaluation results using a fork of LM Evaluation Harness (OpenGPTX), based on version 1 of the HuggingFace Open LLM Leaderboard.
General-Purpose Leaderboards with Dutch Relevance
These platforms aggregate benchmarks and pricing data, including multilingual results:
| Platform | Description |
|---|---|
| BenchLM | 178 benchmarks, real pricing, runtime data. Includes multilingual section. |
| Onyx LLM Leaderboard | General leaderboard with multilingual results. |
| Artificial Analysis | Cross-provider model comparison with pricing, speed, and quality metrics. |
Hardware Context
Summary: Notes for Practitioners
- For broad NLU evaluation: DUMB (9 tasks, encoder-focused) or ScandEval (generative models, Germanic languages).
- For embedding quality: MTEB-NL (40 datasets, 7 task types).
- For grammatical knowledge: BLiMP-NL (8,400 minimal pairs, 22 phenomena) or ANS Challenge Set (prescriptive grammar).
- For information retrieval: BEIR-NL (14 datasets, zero-shot).
- For logical reasoning: SLR-Bench-Dutch (19,000+ tasks, 20 levels).
- For bias and fairness: DutchCrowS (stereotypes, 9 social groups) or MinBZK Social Bias Benchmark (hiring decisions).
- For finance: FinGEITje (Dutch financial domain evaluation).
- For reading time / psycholinguistics: Psychometric evaluation data (14 models, 3 eye-tracking corpora).
- For Dutch models: RobBERT-2023-large (best encoder), Fietje 2B Chat (best small model), GEITje 7B Ultra (best conversational).
This page was generated by an AI assistant on 28 July 2026. It synthesises publicly available sources including DUMB, MTEB-NL, BLiMP-NL, ScandEval, SQuAD-NL, SICK-NL, BEIR-NL, SLR-Bench-Dutch, DutchCrowS, the MinBZK Social Bias Benchmark, the ANS Challenge Set, FinGEITje, RobBERT, BERTje, GEITje, and Fietje publications, the Dutch LM Evaluation Harness, ITHAX, the European LLM Leaderboard, and BenchLM. Specific claims and scores should be verified against the primary sources.