Benchmarking with Weave#

This guide explains how to use the benchmarking system powered by Weights & Biases Weave for evaluating agent performance on engineering optimization problems.

Overview#

The EngiAI includes a comprehensive benchmarking system that evaluates agent performance on structural optimization problems using real datasets. Weave integration is already set up throughout the codebase for automatic tracking and evaluation.

Key Features:

Automated Evaluations: Run agents on standard problem sets with automatic scoring
Multiple Problem Types: Beams, photonics, and thermoelastic optimization
Performance Metrics: Both per-design scores and global distribution metrics
Trace Visualization: View detailed execution traces in the Weave UI

Quick Start#

Configuration#

Add the following to your .env file:

# Weave configuration (for LLM tracing and benchmarking)
USE_WEAVE=true
USE_WEAVE_CHATBOT=false
WEAVE_PROJECT="YOUR_WANDB_ENTITY/engiai-benchmarks"

USE_WEAVE: Enable Weave tracing for evaluations and benchmarks
USE_WEAVE_CHATBOT: Enable/disable tracing for chatbot interactions (default: false)
WEAVE_PROJECT: W&B project name in format entity/project-name

Running Benchmarks#

The main evaluation script is located in benchmarks/evaluations/:

cd benchmarks/evaluations
python evaluate_agent.py \
    --problem beams2d \
    --samples 10 \
    --scorers all

📚 For complete documentation see:

Benchmarks Overview - Available problems, datasets, and metrics
Evaluation Guide - How to run evaluations and configure scorers

Available Problem Types#

The registry (benchmarks/shared/problem_registry.py) defines eight benchmark problems; --problem accepts any of them.

Design-optimization track

Problem	Dataset	Design Space	Metrics
beams2d	`IDEALLab/beams_2d_50_100_v0`	Structural beams, 50×100	Compliance
photonics2d	`IDEALLab/photonics_2d_120_120_v1`	Photonic devices, 120×120	Total overlap
thermoelastic2d	`IDEALLab/thermoelastic_2d_v0`	Thermal structures	Structural compliance, thermal compliance, volume fraction

Note

thermoelastic2d is still registered as a benchmark problem, but the engineering agent’s EngiBench tools currently support only beams2d and photonics2d (src/tools/problems.py), so evaluations against it will fail at create_problem.

Other tracks: rag_beams2d (retrieval), hpc_train_beams2d / hpc_train_photonics2d (SLURM training), training_code_beams2d / training_code_photonics2d (training-code generation).

See benchmarks/README.md for detailed problem descriptions.

Evaluation Workflow#

1. Choose Your Problem#

Select from the available optimization problems:

# Structural beam optimization
python evaluate_agent.py --problem beams2d

# Photonic device design
python evaluate_agent.py --problem photonics2d

# Thermoelastic optimization — registered, but see the note above: the
# engineering agent's tools do not currently support this problem
python evaluate_agent.py --problem thermoelastic2d

2. Configure Evaluation#

Specify the number of test samples:

# Run on 10 samples for quick testing
python evaluate_agent.py --problem beams2d --samples 10

# Run on full dataset (50 samples for beams2d)
python evaluate_agent.py --problem beams2d --samples 50

3. Select Scorer#

Choose which scorers to compute:

all (default): All scorers — output_quality + task_completion + tool_use
output_quality: Per-design quality metrics only
task_completion: Task completion checker only
tool_use: Tool usage efficiency only

# Full evaluation with all scorers
python evaluate_agent.py --problem beams2d --scorers all

# Task-completion only
python evaluate_agent.py --problem beams2d --scorers task_completion

See the Evaluation Guide for scorer details.

4. View Results#

All evaluation runs are automatically tracked in Weave. Access your results at:

https://wandb.ai/your-username/engiai-benchmarks/weave

The Weave UI shows:

Traces: Full execution traces with timing
Calls: Individual LLM calls with inputs/outputs
Metrics: Evaluation scores and performance data
Costs: Token usage and estimated API costs

Understanding Metrics#

Per-Design Metrics (generic scorer)#

Evaluated for each generated design:

Compliance: Structural flexibility (lower is better)
Volume Fraction: Material usage (target-dependent)
Binary Score: How close to binary (0/1) values

Global Metrics (computed offline)#

Evaluated across the full set of generated designs. These are no longer exposed as a Weave scorer — they are computed offline by benchmarks/shared/metrics.py and the post-hoc analysis scripts:

MMD (Maximum Mean Discrepancy): Distribution similarity to a reference set
DPP (Determinantal Point Process): Design diversity
RVC (Ratio of Violated Constraints): Fraction of designs violating constraints (lower is better)
Optimality Gap (IOG/COG/FOG): Distance from optimal solutions

See benchmarks/README.md for detailed metric definitions.

Advanced Usage#

Custom Model Configuration#

Specify the model and temperature:

python evaluate_agent.py \
    --problem beams2d \
    --samples 10 \
    --model "openai:gpt-4.1" \
    --temperature 0.0

Batch Evaluations#

Evaluate multiple configurations:

# Test different problems
for problem in beams2d photonics2d thermoelastic2d; do
    python evaluate_agent.py --problem $problem --samples 10
done

# Test different scorers
for scorer in output_quality task_completion tool_use all; do
    python evaluate_agent.py --problem beams2d --scorers $scorer
done

Programmatic Evaluation#

evaluate_agent.py has no importable Python API — it is argparse-driven, and its only module-level entry point is the async main(). Drive it as a subprocess:

import subprocess

subprocess.run(
    [
        "python",
        "benchmarks/evaluations/evaluate_agent.py",
        "--problem",
        "beams2d",
        "--samples",
        "10",
        "--scorers",
        "all",
    ],
    check=True,
)

Results are written under benchmarks/evaluations/results/ and, when USE_WEAVE=true, tracked in the Weave UI.

Adding New Problem Types#

To add a new optimization problem to the benchmark suite:

Create problem directory: benchmarks/problems/your_problem/
Add to registry: Update benchmarks/shared/problem_registry.py
Implement generator: Create generate_prompts.py for your problem
Add dataset: Create or reference HuggingFace dataset

See the Evaluation Guide for complete instructions.

Best Practices#

Start Small: Test with --samples 10 before running full evaluations
Use Generic Scorer: For per-design analysis and debugging
Use EngiBench Scorer: For comparing overall performance across agents
Track Costs: Monitor token usage in the Weave UI to manage API costs
Version Control: Keep track of agent configurations and model versions
Regular Testing: Run benchmarks regularly to detect performance regressions

Troubleshooting#

Weave Not Initialized#

Problem: No traces appearing in Weave UI

Solution: Ensure USE_WEAVE=true in your .env file and WANDB_API_KEY is set:

export WANDB_API_KEY="your-api-key"
echo "USE_WEAVE=true" >> .env

Import Errors#

Problem: Cannot import benchmark modules

Solution: Run from the correct directory:

cd benchmarks/evaluations
python evaluate_agent.py --problem beams2d

Dataset Loading Fails#

Problem: Cannot load HuggingFace dataset

Solution: Check internet connection and dataset name in problem_registry.py:

# Verify dataset names match HuggingFace
PROBLEMS = {
    "beams2d": ProblemConfig(
        dataset="IDEALLab/beams_2d_50_100_v0",  # Must match exactly
        ...
    )
}

Scorer Not Found#

Problem: ValueError: Unknown scorer: ...

Solution: Use valid scorer names: all, output_quality, task_completion, or tool_use

# Correct
python evaluate_agent.py --problem beams2d --scorers all
python evaluate_agent.py --problem beams2d --scorers task_completion

# Incorrect
python evaluate_agent.py --problem beams2d --scorers custom  # Not supported

Technical Integration Details#

For developers who need to understand how Weave is integrated into the codebase:

Automatic Tracing#

Weave is initialized in config.py and automatically traces all LangChain operations:

from config import config

# Initialize Weave based on .env configuration
config.setup_weave_tracing()  # Called at startup

After initialization, all LangChain components are automatically traced:

Agent executions
LLM calls (OpenAI, Google, etc.)
Tool usage
Chain invocations

Custom Operations#

Use the @weave.op() decorator for custom traced operations:

import weave


@weave.op()
def custom_optimization(design: dict, constraints: dict) -> dict:
    """Custom optimization function with Weave tracing."""
    optimized = run_optimization(design, constraints)
    return optimized

Conditional Tracing#

Check if Weave is enabled before Weave-specific operations:

from src.utils.weave_integration import is_weave_enabled
import weave

if is_weave_enabled():
    dataset = weave.Dataset(name="my_dataset", rows=data)
    weave.publish(dataset)
else:
    # Alternative logging
    print("Weave disabled, using local logging")

See Weave Integration Utils for implementation details.

Benchmarking with Weave

Contents

Benchmarking with Weave#

Overview#

Quick Start#

Configuration#

Running Benchmarks#

Available Problem Types#

Evaluation Workflow#

1. Choose Your Problem#

2. Configure Evaluation#

3. Select Scorer#

4. View Results#

Understanding Metrics#

Per-Design Metrics (generic scorer)#

Global Metrics (computed offline)#

Advanced Usage#

Custom Model Configuration#

Batch Evaluations#

Programmatic Evaluation#

Adding New Problem Types#

Best Practices#

Troubleshooting#

Weave Not Initialized#

Import Errors#

Dataset Loading Fails#

Scorer Not Found#

Further Reading#

Technical Integration Details#

Automatic Tracing#

Custom Operations#

Conditional Tracing#