HPC Integration

EngiAI integrates with high-performance computing (HPC) clusters for running computationally intensive tasks via SLURM.

Prerequisites

• Access to an HPC cluster with SLURM (e.g., ETH Euler)

• SSH key authentication configured

• Proper environment modules on the cluster

Configuration

Shared lab deployment: managed broker

On a shared workstation, do not mount the technical Euler account’s SSH key into the chatbot container. Install the native broker instead:

sudo scripts/install_hpc_broker_macos.sh \
  --euler-user TECHNICAL_EULER_USER \
  --client-user ideal-lab \
  --known-hosts /path/to/verified-euler-known-hosts

The installer creates a hidden _engiai_hpc macOS service account, generates a dedicated Ed25519 key under /var/db/engiai-hpc, and runs the broker as a loopback-only LaunchDaemon. The private key is readable only by the service account. Authorize the printed public key on Euler once, then configure the values printed by the installer in .env. In particular, set SSH_HOST_DIR to an empty directory so no unrelated host SSH keys are copied into Docker. Supply a known_hosts file copied from an administrator machine that has already verified Euler. Do not build this file with an unauthenticated ssh-keyscan during deployment.

The broker exposes only the operations EngiAI needs: controlled file transfer, SLURM validation/submission, status, cancellation, diagnostics, and output download. It rejects paths outside its configured working directories, forbids job-side account/QoS/reservation overrides, applies CPU/GPU/memory/time/array and active-job ceilings, and writes a JSONL audit trail to /var/log/engiai-hpc-audit.jsonl.

The bearer token available to Docker is an authorization capability, not the Euler credential. A user who controls the shared Docker account can potentially reuse that capability to invoke the same bounded broker operations; they still cannot retrieve the Euler private key. Enforce the authoritative spending and access limits on Euler itself with a least-privilege technical account and an appropriate SLURM account/QoS.

The default broker configuration tries direct euler.ethz.ch:22 first and the temporary lab tunnel at 127.0.0.1:2222 second. At ETH it therefore connects directly. During off-campus setup, the reverse tunnel from a VPN-connected administrator Mac can provide the second route without placing VPN credentials on the shared workstation.

Rotating the broker token

The native broker reads its bearer token only when the LaunchDaemon starts. After an administrator replaces both the protected service token and the client copy mounted into Docker, restart the broker before testing the new token:

sudo launchctl kickstart -k system/ch.ethz.ideal.engiai-hpc-broker
docker compose restart chatbot

Restarting the chatbot is required when the client token was replaced rather than edited in place, because Docker may otherwise retain the old bind-mounted file inode. Never paste either token into logs, tickets, or chat messages.

Personal/development deployment: direct SSH

Set up your HPC credentials in .env:

HPC_HOST_ALIAS=euler
HPC_HOSTNAME=euler.ethz.ch
HPC_USERNAME=your_username

And configure your SSH access:

# In ~/.ssh/config
Host euler
    HostName euler.ethz.ch
    User your-username
    IdentityFile ~/.ssh/id_ed25519

Test the connection:

ssh euler

Remote Python Environment Setup

The HPC agent submits SLURM jobs that run training from a clone of EngiOpt (cd $HOME/EngiOpt) inside a Python virtual environment. You need both the clone and the venv on the cluster. This is a one-time setup.

Recommended: setup script

Copy the repo’s scripts/setup_engiopt_remote.sh to the cluster and run it after SSHing in. It loads the modules, clones EngiOpt at the pinned commit into $HOME/EngiOpt, creates the venv at ~/venvs/engiai, and installs the pinned engibench + engiopt:

ssh euler
# from the engineer-assistant checkout, copy the script over, e.g.:
#   scp scripts/setup_engiopt_remote.sh euler:~/
./setup_engiopt_remote.sh

The script is idempotent (re-running re-checks the pin and reuses the venv); pass --force to wipe and recreate both. Defaults can be overridden via environment variables (ENGIOPT_DIR, VENV_DIR, SLURM_PYTHON_MODULE, …) — run with --help for the full list.

Manual setup

If you prefer to do it by hand:

ssh euler
module load stack/2024-06 gcc/12.2.0 python_cuda/3.11.6 cuda/12.1.1
git clone https://github.com/IDEALLab/EngiOpt.git ~/EngiOpt
git -C ~/EngiOpt checkout fb8fb12cfd0945bea0d335f06f8147af700e3ba0
python -m venv ~/venvs/engiai
source ~/venvs/engiai/bin/activate
pip install "engibench[all] @ git+https://github.com/IDEALLab/EngiBench.git@221605876556e23bb97ae3c73d58c74fa7338473"
pip install -e ~/EngiOpt

Note

The commit hashes above must match the versions pinned in pyproject.toml. The script reads the same pins; check pyproject.toml if either is outdated.

The EngiOpt path (~/EngiOpt) and venv path (~/venvs/engiai) are configurable in the Settings UI under SLURM Configuration > EngiOpt Path and Virtual Environment Path.

You must always load the modules before activating the venv, since it depends on the module-provided Python:

module load stack/2024-06 gcc/12.2.0 python_cuda/3.11.6 cuda/12.1.1
source ~/venvs/engiai/bin/activate

Using the HPC Agent

The HPC agent is accessed through natural language via the Streamlit UI or Supervisor Agent. Simply describe what you want to do:

Submitting Jobs

"Submit a training job to Euler with seed 1 and 100 epochs"
"Run the cGAN training on HPC for beams2d"

The agent will:

1. Generate a SLURM script with appropriate resources

2. Submit the job via SSH

3. Report the job ID

Monitoring Jobs

"Check the status of my running jobs"
"Monitor job 12345 until it completes"

Retrieving Results

"Download the output from job 12345"
"Get the training results from HPC"

Available HPC Tools

The HPC agent has access to these tools:

Tool	Description
test_hpc_connection	Verify SSH connectivity to the cluster
submit_slurm_job	Submit a SLURM job script
get_slurm_job_status	Check job status
monitor_job_until_complete	Poll until job finishes
cancel_slurm_job	Cancel a running job
download_job_outputs	Retrieve output files via SFTP
get_active_jobs_summary	List all active jobs

SLURM Configuration

Default SLURM parameters can be set in .env:

SLURM_TIME=00:45:00
SLURM_NTASKS=1
SLURM_CPUS_PER_TASK=4
SLURM_MEM_PER_CPU=7GB
SLURM_GPUS=rtx_4090:1

These can also be configured via the Settings UI.

ML Training Pipeline

The HPC agent supports end-to-end ML training workflows:

1. Generate training script — Creates a SLURM script for EngiOpt model training

2. Submit to cluster — Uploads and submits the job

3. Monitor execution — Tracks job progress until completion

4. Evaluate results — Downloads the trained model and computes metrics

Example:

"Train a cGAN model for beams2d on Euler with seed 1 and 100 epochs,
then evaluate it against the dataset baseline"

SSH in Docker

Direct-SSH development mounts keys from SSH_HOST_DIR (default: ~/.ssh). A shared lab deployment must use the managed broker above and point SSH_HOST_DIR at an empty directory. See the Docker Deployment Guide for direct SSH agent details.

Troubleshooting

Connection Issues

1. Verify SSH access: ssh euler

2. Check SSH key permissions: chmod 600 ~/.ssh/id_ed25519

3. For Docker: ensure SSH agent is running with ssh-add -l

Job Failures

• Check job output: cat slurm-<jobid>.out

• Verify modules are available: module avail

• Check resource requests match cluster limits

Next Steps

• Learn about agents

• Tool reference

• Configuration guide