Purpose

This document maps SURF Developer Platform (SDP) capabilities against three regulatory frameworks, identifies gaps, and recommends open-source options.

Technology choices in consultation with SDP and team Scalable Data Analytics (SDA) at SURF. Not all capabilities need to run on SDP — SDA has experience with additional platforms.

NIS2 Art. 21 — Coverage (1/2)

• (a): No data classification engine at application level
• (b): No application-level incident workflow for 24h/72h/1mo chain
• (c): SLA: "at most data from the last 24 hours may be lost". No DR orchestration

NIS2 Art. 21 — Coverage (2/2)

• (f): No immutable or tamper-proof logging
• (h): Kafka topics NOT encrypted at rest. PII storage prohibited on production. PostgreSQL/MinIO encryption at rest undocumented
• (i): Authorization namespace-level only — no fine-grained RBAC/ABAC per data layer

EU AI Act — Coverage

Education AI = high-risk (Annex III, Cat. 3): access/admission, learning outcome evaluation, education level assessment. Deadline: Aug 2, 2026.

Loki: not tamper-proof, not structured for decision reconstruction, retention not configurable to 10 years.

Data Act — Coverage

No semantic interoperability layer. No data sharing agreement management.

Overview — 14 Gaps

* Only if Kafka is used. Batch-only pipelines avoid this gap.

GAP-01: PII Storage Prohibited on Production

This is the prerequisite for the entire platform. Every use case (enrollment forecasting, study success analysis, learning analytics) requires student-level data.

Action: Engage with SDP / SDA team to define PII-compliant storage roadmap before PoC proceeds with real data.

GAP-02: Kafka Topics Not Encrypted at Rest

Only relevant if event streaming via Kafka is used. For batch-only pipelines, this gap does not apply.

Possible option: Application-level envelope encryption — encrypt payloads before writing to Kafka using a KMS. No SDP changes needed.

Possible option: Request SDP / SDA team to enable disk-level encryption for Kafka storage.

Consider whether batch-only patterns (MinIO file ingestion, scheduled DB queries) can avoid Kafka entirely.

GAP-03: No Immutable Audit Logging

Required: Immutable, append-only audit log with structured schema for decision events and cryptographic integrity verification.

Possible option: OpenSearch with write-once indices + S3 (MinIO/SURF Object Store) archival for long-term retention.

Possible option: Sigstore Rekor — cryptographically signed, tamper-evident transparency log.

GAP-04: No Data Lineage / Provenance

Required: Track data from source (DUO, SIS, 1cHO) through Bronze → Silver → Gold, recording every transformation.

Possible option: OpenLineage (open standard, CNCF) + Marquez (reference implementation, PostgreSQL backend). Integrates with Spark, dbt, Airflow, Dagster.

Possible option: Apache Atlas — data governance and metadata management including lineage. Heavier to operate.

GAP-05: No ML Model Registry

Required: Model versions, training metadata, evaluation metrics, deployment status, and AI Act conformity metadata (risk classification, accuracy benchmarks, training data documentation).

Recommended: MLflow Tracking Server + Model Registry (PostgreSQL + MinIO backend). Extend with custom metadata schema for AI Act conformity fields.

MLflow is the dominant open-source option. Focus should be on schema design for AI Act metadata.

GAP-06 to GAP-10 — HIGH Severity

GAP-10 is a force multiplier: Vault solves GAP-01 (KMS for PII storage), GAP-02 (transit encryption for Kafka), and GAP-10 (dynamic secrets) jointly.

GAP-11 to GAP-14 — MEDIUM Severity

• GAP-12: Art. 9 explicitly requires attention to vulnerable groups
• GAP-13: NannyML useful where outcome labels are delayed (graduation/dropout = months/years)
• GAP-14: EDC aligns with DSSC Blueprint but has significant operational complexity

Required Capabilities — Summary

Storage Options

Both MinIO and SURF Object Store are S3-compatible. SURF Object Store may offer better guarantees for long-term retention and scale.

For columnar analytical workloads: DuckDB (embedded, no infra) or Apache Spark (cluster-based).

Architecture — Capabilities on SDP

Compliance Deadlines

Research (May–August 2026)

Understanding how to cover the critical gaps. Not all capabilities need to run on SDP — SDA team has experience with additional platforms.

Experimentation (September–December 2026)

Parallel to PoC — experimenting with implementations to assess operational load.

• Deploy 2-3 applications in a test environment (SDP or other platform)
• Assess operational complexity, integration effort, and maintenance burden
• Evaluate whether capabilities can be shared across institutions or need per-institution deployment

Summary

SDP provides: Solid infrastructure — K8s, GitLab, Harbor, MinIO, PostgreSQL, Kong, Grafana/Loki, SURFconext, SURFsecureID. ISO 27001. Covers NIS2 Art. 21(d)(e)(j) well.

SDP does not provide: 14 gaps identified across NIS2, EU AI Act, and Data Act. Five are CRITICAL — they block compliant operation entirely.

Most significant: GAP-01 — PII storage currently prohibited on SDP production. Must be resolved before PoC can proceed with real student data.

11 open-source capabilities needed. Not all need to run on SDP — SDA team has experience with additional platforms. Compliance tooling wraps around existing analytical functions — core Bronze/Silver/Gold pipeline logic unchanged. Functions remain locally executable without the full compliance stack.

2026 focus: Research how to cover critical gaps + experiment with implementations to assess operational load.