Frozen MMORE v1 stack (paper reproducibility)#

The RAG-axis results reported in the EngiAI papers were produced with a MMORE deployment that no longer matches what EngiAI runs today. This page explains what changed, and how to reconstruct the original stack if you need to reproduce those numbers.

Everything described here lives in services/mmore/frozen-v1/ in the repository. Start with MANIFEST.txt — it records the exact commits, checksums, index contents, and retrieval hyperparameters.

What changed and why#

MMORE was never vendored into this repository. It lived as a manual clone of swiss-ai/mmore outside the repo, carrying:

one local commit (fa24033), present on no upstream branch, replacing a hardcoded .to("cuda") in the reranker batch loop so the reranker runs on a CPU-only macOS host;
two uncommitted working-tree edits — a milvus-lite version constraint, and a scipy coo_array compatibility fix in the SPLADE sparse embedder;
a pip install "setuptools<72" hotfix embedded in the Docker Compose command:, because the pinned pymilvus still needs the pkg_resources module that setuptools>=72 drops.

None of that was under version control. Meanwhile upstream renamed to EPFLiGHT/mmore, released v2.0.0, and changed ingestion from a synchronous 201 response to an asynchronous 202 + jobId with background indexing. EngiAI now tracks the official upstream v2 container image, which makes the published configuration unreachable from a normal checkout — hence this archive.

All three local patches are obsolete in v2: upstream now selects the reranker device from the loaded model, fixed the SPLADE compatibility issue in a more robust way, and declares its own setuptools constraint. Do not re-apply them.

Reconstructing the stack#

git clone https://github.com/EPFLiGHT/mmore.git mmore-v1   # formerly swiss-ai/mmore
cd mmore-v1
git checkout 6cd51f4
git am    < .../services/mmore/frozen-v1/0001-cuda-device-autodetect.patch
git apply < .../services/mmore/frozen-v1/0002-worktree-milvus-lite-and-splade-scipy.patch

tar xzf ~/Desktop/engiai-archive/mmore-v1-proc_demo.db.tar.gz
shasum -a 256 proc_demo.db   # must match MANIFEST.txt

docker volume create engiai_mmore_cache
docker volume create engiai_mmore_uploads
# the compose file declares four external volumes; these two are for mmore-rag-eval
docker volume create engiai_mmore_eval_uploads
docker volume create engiai_mmore_eval_cache

# then, from the engineer-assistant repo root
MMORE_PATH=$PWD/../mmore-v1 docker compose \
  -f services/mmore/frozen-v1/docker-compose.mmore-v1.yml up -d mmore-rag-service

The external: true volumes really did have to be created by hand — that gap is part of the frozen artifact and is preserved rather than fixed.

Note

src/tools/mmore_client.py on main speaks the v2 async ingest API and will not upload correctly against this v1 server. To exercise the frozen stack end to end, check out a pre-migration commit of the client too. Retrieval (POST /v1/retrieve) is unchanged between v1 and v2 and works either way.

Retrieval configuration#

The mounted retriever_config.yaml is five lines and sets only hybrid_search_weight: 0.5, k: 5, collection_name: my_docs and the database location. Three further values were active at runtime without appearing in any YAML, because they come from the RetrieverConfig dataclass defaults:

Setting	Effective value	Source
Dense embedding	`sentence-transformers/all-MiniLM-L6-v2`, 384-d, cosine, IVF_FLAT (nlist 128)	index config + recorded in the database
Sparse embedding	`naver/splade-cocondenser-selfdistil`, inverted index, inner product	index config + recorded in the database
Reranker	`BAAI/bge-reranker-base`	`RetrieverConfig` default
Web search	disabled	`RetrieverConfig` default

These are the values behind the retrieval-pipeline description in the papers.

What the deployed service does not include#

Worth stating explicitly, because it is easy to assume otherwise from MMORE’s own documentation: the deployed server was mmore.run_live_retrieval, which mounts only the index and retriever routers. It exposes POST /v1/retrieve, which returns ranked chunks. There is no generation step and no language model anywhere in that process.

MMORE does ship a separate generation server (mmore rag, endpoint /rag, configured in examples/rag/config.yaml), but it was never started here — and in the pinned tree that config names OpenMeditron/meditron3-8b, not a Qwen model. Searching the pinned source tree for qwen returns no hits outside the text of unrelated sample PDFs.

So in the evaluated configuration, retrieved chunks were passed back to the agent, and synthesis into the final answer was performed by whichever backend LLM the agent was running (GPT-5-mini, Gemini-3-Flash, Qwen3-4B or Qwen3.5-4B, all at temperature 0).

The index holds seven documents, not two#

This is the detail most likely to trip up a reproduction attempt.

Only engibench_paper and soptx_paper are referenced by the rag_beams2d benchmark. But the agent’s search_documents tool calls MMOREClient.retrieve() without a file_ids filter, so ranking ran over the entire collection — all 267 chunks:

`document_id`	Chunks	Source
`arxiv_2507.08619`	100	fetched and indexed by the ArXiv agent at run time
`soptx_paper`	64	`papers/2505.02438v1.pdf` (SOPTX)
`andreassen_88line`	53	`papers/s00158-010-0594-7.pdf` (Andreassen 88-line)
`engibench_paper`	40	`papers/2508.00831v2.pdf` (EngiBench)
`test_mmore_readme`	8	test artifact
`pdf-test`	1	test artifact
`engibench`	1	test artifact (stray single chunk)

An index rebuilt from only the two target papers is therefore not equivalent to this artifact. The distractor set differs, which can change ranking and move the top-k=5 cut. If you are reproducing published numbers, restore the archived database rather than rebuilding.

The database file#

proc_demo.db is not committed: at 1.7 MB it exceeds the repository’s 500 kB check-added-large-files pre-commit limit, and it is a derived chunk store of third-party PDFs. It is archived outside the repository, with its path, size and sha256 recorded in MANIFEST.txt §4, alongside the sha256 of every source PDF so the inputs can be verified independently.