Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

🏠 Homepage (coming soon) · 🤗 Data · 📄 Paper (coming soon)

Official evaluator for Diagram-MMU, a multi-modal benchmark that assesses how well MLLMs parse and understand scientific diagrams.

Diagram-MMU contains 3.7k curated diagrams and 18.3k human-validated evaluation instances (nominally up to 1 D2C-P, 2 D2C-E and 2 DQA per diagram, though not every diagram has all of them — some are dropped during human review) across six domains (charts, planar geometry, 3D shapes, graph structures, chemistry, circuit diagrams). It evaluates MLLMs on three tasks common to scientific-writing workflows:

This repository is the scoring engine: you bring model predictions (or let it run inference end-to-end) and it produces per-task metrics. The dataset is loaded directly from the HuggingFace Hub.

git clone <this-repo-url> Diagram-MMU && cd Diagram-MMU
python3 -m venv .venv && source .venv/bin/activate
pip install -e '.[all]'

Dependency extras (install only what you need):

D2C tasks compile LaTeX, so they need a TeX toolchain (DQA does not). For paper-matching results we recommend TeX Live 2019 with these pinned package versions (placed in a ~/texmf overlay):

Also install ghostscript and poppler-utils. Verify packages are visible:

kpsewhich circuitikz.sty tkz-euclide.sty chemfig.sty simplekv.sty

Newer TeX versions compile most records but may cause small metric drift.

diagram-mmu download AIGrounding/Diagram-MMU --split testmini   # or: --split test

You can also pass the HF repo id directly to any command via --dataset; the dataset is auto-landed to ~/.cache/diagram-mmu/ on first use (override with $DIAGRAM_MMU_CACHE). Landing is idempotent and incremental.

Run your model first to produce one prediction JSONL per task (see Prediction Format), then:

# DQA (rule + LLM-judge fallback; add --no-llm-judge for rule-only)
diagram-mmu score --dataset AIGrounding/Diagram-MMU --task dqa --split testmini \
  --predictions preds/dqa.jsonl --output out/dqa

# D2C-P / D2C-E (needs the TeX toolchain)
diagram-mmu score --dataset AIGrounding/Diagram-MMU --task d2c-p --split testmini \
  --predictions preds/d2c-p.jsonl --output out/d2c-p

The DQA judge reads OPENAI_BASE_URL / OPENAI_API_KEY (point them at a Qwen3-Next-80B-compatible endpoint).

bench runs inference + evaluation + report against a served, OpenAI-compatible model:

diagram-mmu bench --config configs/models/qwen3-vl-8b.yaml \
  --dataset AIGrounding/Diagram-MMU --split testmini --task all --output out_bench

scripts/ wraps the CLI for the common flow (thin orchestration; report.sh just prints the JSON summaries):

DATASET=AIGrounding/Diagram-MMU SPLIT=testmini PRED_DIR=preds bash scripts/run_all.sh

The dataset is hosted on Hugging Face: Diagram-MMU. It has four subsets (diagrams, d2c-p, d2c-e, dqa) and two splits:

On first use the evaluator materializes the Hub data into a local cache with the layout the scorer consumes:

<cache>/
  dataset_info.json
  diagrams.jsonl / diagrams_testmini.jsonl
  images/<diagram_id>.png
  d2c-p/{test,testmini}.jsonl + d2c-p/reference_images/<id>.png
  d2c-e/{test,testmini}.jsonl + d2c-e/reference_images/<id>.png
  dqa/{test,testmini}.jsonl

diagram-mmu validate --dataset AIGrounding/Diagram-MMU checks the landed data.

D2C-P / D2C-E (reported ×100):

• Object F1 — type / text / bbox / color, then F1_avg. Color uses ΔE CIEDE2000; bbox matching uses IoU ≥ 0.3.
• CrystalBLEU — code similarity.
• Image — SSIM, CLIP, LPIPS, FID, computed on 256×256 white-background renders.
• Composites — SC = mean(SSIM, CLIP) (↑), FL = mean(FID, LPIPS) (↓), All = mean(F1_avg, CrystalBLEU, SC).
• D2C-E additionally reports (preserve, edit) splits for F1 and CrystalBLEU.

DQA — binary accuracy. Answers are extracted from the model output and matched by rule; low-confidence cases (rule confidence < 0.85) fall back to an LLM judge (Qwen3-Next-80B-A3B-Instruct; judge temperature 0.7 from the released config). Accuracy is also broken down by output-instruction type, question type, and domain.

One JSONL per task, one prediction per line. Required fields are id (matching a dataset record id) and prediction (the raw model output); unknown fields are ignored.

{"id": "0505_d2cp",  "prediction": "```latex\n\\documentclass[tikz]...\n```"}
{"id": "0505_dqa_0", "prediction": "{\"answer\": \"21\", \"reasoning\": \"...\"}"}

For D2C tasks the LaTeX is extracted from a fenced ```latex block. For DQA a JSON object like {"answer": ..., "reasoning": ...} is expected.

• Pin the dataset revision: --dataset-revision <tag-or-commit>.
• Use TeX Live 2019 + the pinned packages above for paper-matching D2C scores.
• Image-metric weights (FID InceptionV3, LPIPS AlexNet, CLIP ViT-B/32) download on first use; pre-stage the torch / HuggingFace caches for offline runs.
• Keep the DQA judge model and temperature as documented for comparable scores.

D2C scoring compiles LaTeX, i.e. it executes an external toolchain on model-generated code. Run evaluation of third-party predictions in a sandbox or container.

@article{bo2026diagrammmu,
  title   = {Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams},
  author  = {Bo, Weihao and Zhang, Shan and Sun, Yanpeng and Liu, Jie and
             Yao, Yongke and Du, Jinhao and He, Wei and Zou, Kai and
             Li, Zechao and Wang, Jingdong},
  year    = {2026},
  note    = {TODO: add arXiv id / venue}
}

Code is released under the Apache-2.0 License. The dataset is distributed under its own terms on Hugging Face.

The metric implementations follow conventions from prior diagram/chart-to-code and VQA benchmarks (e.g. CharXiv, ChartMimic, DeTikZify). Thanks to those projects and to the contributors of Diagram-MMU.