Benchmarks

EngiAI ships an evaluation suite that measures what the agent system actually does, rather than what a model says it would do. Every problem runs the real agents against real tools — EngiBench solvers, the RAG service, a SLURM cluster — and scores the artefacts that come out.

This page is the map. The deeper material lives next to the code: benchmarks/README.md for metric definitions and benchmarks/evaluations/README.md for the harness.

Running an evaluation

Two steps. Prompts are generated once per problem and style, then evaluated:

# 1. generate the prompt set
cd benchmarks/problems/beams2d
python generate_prompts.py --samples 5 --style workflow-random --seed 42

# 2. run the agents against it
python benchmarks/evaluations/evaluate_agent.py \
    --problem beams2d --samples 5 --prompt-style workflow-random --seed 42

--problem accepts only registered problems — the choices come from benchmarks/shared/problem_registry.py, so an unregistered name is rejected by argparse rather than failing somewhere downstream.

Note

Adding an entry to PROBLEMS in the registry exposes it in every --problem choice list automatically. New CLIs must keep that true by passing choices=list_problems(); a test enforces it.

The four families

Family	Problems	What it measures
Design optimization	beams2d, photonics2d, thermoelastic2d	the design produced, against ground truth
RAG retrieval	rag_beams2d	whether the agent looks parameters up instead of guessing
HPC orchestration	hpc_train_beams2d, hpc_train_photonics2d	driving a SLURM training run end to end
Training-code generation	training_code_beams2d, training_code_photonics2d	writing the training code

The last three families carry no dataset and no objectives, which the problem config permits because objectives are only required when a dataset is present. Each brings its own scoring split rather than the shared design one.

Warning

thermoelastic2d is registered, but the engineering agent’s EngiBench tools support only beams2d and photonics2d (src/tools/problems.py). Evaluations against it fail at create_problem.

Scoring

Weights come from the registry, so they are the values actually applied:

Design optimization — design_quality 0.65, tool_efficiency 0.20, task_completion 0.15. Within design quality: IoU 0.31, pixel accuracy 0.19, objective match 0.15, constraint match 0.12, connectivity 0.12, watertightness 0.11. (thermoelastic2d shifts constraint match to 0 and objective match to 0.27.)

RAG retrieval — a single rag_accuracy term. Parameter accuracy is only credited when the RAG tool was actually called, so a lucky guess scores nothing.

HPC orchestration — workflow_completion 0.80, tool_efficiency 0.20.

Training-code generation — six weighted terms:

Term	Weight
model_quality	0.55
code_quality	0.15
job_completion	0.10
dryrun	0.10
syntax	0.05
slurm_accepted	0.05

Prompt styles

beams2d and photonics2d both accept seven styles. Each isolates a different failure mode, and all but the first two were added to probe behaviours that exact-parameter prompts cannot reach:

Style	What it tests
full	baseline: exact numerical parameters
natural	natural-language descriptions only
workflow-random	following precise randomized export parameters
workflow-derived-params	arithmetic — parameters must be computed from the optimization inputs
workflow-distractor	semantic disambiguation between two plausible values for the same parameter
workflow-conditional	tool-output → reasoning → tool-input chaining, branching on the simulation result
workflow-multi-export	working memory: two exports from one optimization, with different parameters

workflow-distractor is the one worth understanding, because the tool call succeeds either way. Both candidate values are valid for the schema, so the STL file is produced regardless; only the parameter-validation scorer notices that the agent took the preview value instead of the export one.

The other problems have their own styles: rag_beams2d uses rag-eval, and the hpc_train_* problems use hpc-train-cgan, hpc-train-diff, hpc-train-natural-cgan and hpc-train-natural-diff.

Training-code generation

The agent authors the missing parts of a training script, validates it, wraps it in SLURM, submits it, and is scored on whether the resulting model samples well. All nine tools bind to HPCAgent deliberately, so the efficiency scorer sees one unbroken sequence rather than a trace split across two agents.

The canonical order is write_training_script → validate_training_script → generate_training_slurm → validate_slurm_script → dryrun_training_script → submit_slurm_job → monitor_job_until_complete → evaluate_model, with diagnose_job_failure as the recovery path.

Warning

“Dryrun” means two different things here. validate_training_script runs a local one-epoch subprocess with no SSH; dryrun_training_script submits a one-epoch SLURM job. They are separate gates.

Difficulty tiers

--prompt-style {tier}-{terse|verbose}, sixteen combinations. Each tier blanks more of the reference file:

Tier	Blanked	Capability under test
t2	sampling, logging	plumbing
t3	loss, optimizers, inner step	the training loop
t4	model class bodies	architecture
t5	most of the module	whole module from a skeleton
t6	nothing — no template	novel algorithm, fixed interface
t6v2	nothing — no template	novel algorithm and self-declared interface

The -func variants of t4 and t5 also blank the sample() adapter and score functionally: the model is evaluated through the agent’s own sampler, so an implementation that works but diverges structurally still counts.

--algorithm selects the source template, cgan_cnn_2d (default) or diffusion_2d_cond.

How model quality is graded

model_quality is graded rather than binary:

clip(baseline_mmd / agent_mmd, 0, 1)

against a committed per-seed baseline, so matching or beating the reference scores 1.0.

Important

With no baseline for that (algorithm, seed) — or an evaluation that reported no usable MMD — the trial is UNSCORABLE: model_quality_score and training_code_score are both None, and model_quality_unscorable_reason records which. This is deliberately neither 1.0 (which gave an uncalibrated model full marks) nor 0.0 (a missing baseline is a harness gap, not an agent failure). None keeps the trial rescorable once a baseline exists, and the aggregation scripts refuse to average a cell containing one.

Tracing

Set USE_WEAVE=true to trace evaluations with Weave. See Weave integration for what gets logged and how to read it back.