Paper Import Guide#

This guide explains how to automatically import PDF papers from your local filesystem or mounted network share into your MMORE RAG system for document-based Q&A.

Overview#

The paper import workflow allows you to:

Upload research papers, documentation, and PDFs to the MMORE service
Search and query these documents using natural language
Get answers with context from your document collection
Track which documents have been uploaded

The import script (scripts/import_local_papers.py) provides automatic importing with:

Incremental Updates: Only processes new or modified files
State Tracking: Remembers which files have been imported (in a JSON state file)
Error Handling: Continues processing even if individual files fail
Retry Logic: Automatic retry with exponential backoff for failed uploads (3 attempts)
Recursive Directory Scanning: Processes PDFs in subdirectories
Metadata Tracking: Tags documents with source information in MMORE
Dry-Run Mode: Preview what will be imported without making changes
Large File Support: No size cap — files over 50 MB log a warning that processing may take a while, then upload normally

Workflow#

┌─────────────────┐
│ Local PDFs      │
│ (Your Computer  │
│  or Network     │
│  Share)         │
└────────┬────────┘
         │
         ▼
┌─────────────────┐      ┌──────────────────┐
│ import_local_   │─────>│ MMORE Service    │
│ papers.py       │      │ (localhost:8000) │
│ (Upload Script) │      │                  │
└─────────────────┘      └────────┬─────────┘
         │                        │
         │                        ▼
         │               ┌─────────────────┐
         │               │ Document Store  │
         │               │ (Searchable)    │
         │               └─────────────────┘
         ▼
┌─────────────────┐
│ JSON state file │
│ (Track State)   │
└─────────────────┘

Installation#

1. Install Dependencies#

All dependencies are automatically installed when you set up the project:

# If using conda
conda activate engiai

# If using Docker
# Dependencies are already included in the container

2. Verify MMORE Service#

Ensure the MMORE RAG service is running at http://localhost:8000:

# Test MMORE service is responding (GET / is the health route; there is no /health)
curl http://localhost:8000/

If MMORE is not running, see the Installation Guide for setup instructions.

3. Configure Import Path#

Add the following to your .env file:

# Paper Import Configuration
PAPERS_SOURCE_DIR=/path/to/your/papers
PAPERS_STATE_FILE=data/local_import_state.json
MMORE_RAG_URL=http://localhost:8000  # MMORE service URL

Configuration Parameters:

PAPERS_SOURCE_DIR: Path to directory containing PDFs (can be a mounted network share)
- Examples: /Volumes/ShareName/Papers (macOS), /mnt/share/papers (Linux), C:/Papers (Windows)
PAPERS_STATE_FILE: JSON file tracking import state (default: data/local_import_state.json)
- Tracks file paths, sizes, modification times, and upload status
- Used to detect changes and avoid re-uploading unchanged files
MMORE_RAG_URL: MMORE service URL (default: http://localhost:8000)

Important Notes:

State Tracking: The script tracks uploads in a JSON state file only
- State file: file path, size, mtime, import timestamp and the MMORE file id
- The authoritative record of what is indexed is MMORE itself — query it with MMOREClient.list_files() or python scripts/inspect_mmore.py --list
File Size Limit: Files larger than 50MB will trigger a warning but still upload
Network Shares: Works with any mounted filesystem (NFS, SMB, AFP, etc.)

For Mounted Network Shares:

If you have a network share mounted, use that path directly:

# macOS example
PAPERS_SOURCE_DIR=/Volumes/ShareName/Papers

# Linux example
PAPERS_SOURCE_DIR=/mnt/network_share/papers

# Windows example (in WSL or Git Bash)
PAPERS_SOURCE_DIR=/mnt/c/Users/YourName/Documents/Papers

Usage#

Quick Start#

Once configured in .env, simply run:

# From project root directory
python scripts/import_local_papers.py

# This will:
# 1. Scan PAPERS_SOURCE_DIR for PDFs
# 2. Compare with previously imported files
# 3. Upload only new or modified files to MMORE
# 4. Record the MMORE file id and metadata in the state file
# 5. Save progress to PAPERS_STATE_FILE

Command-Line Usage#

Specify the source directory directly:

python scripts/import_local_papers.py /Volumes/ShareName/Papers

Configuration Priority:

Command-line argument (highest priority)
PAPERS_SOURCE_DIR environment variable from .env
source_dir in the JSON config file (default scripts/local_import_config.json, override with --config; not present in a clean checkout)

If none of the three is set, the script logs “Missing source directory”, prints help and exits 1 — there is no ./papers/ default.

Dry-Run Mode#

Preview what will be imported without uploading:

python scripts/import_local_papers.py --dry-run

# Output example:
# Found 150 PDF files
# 120 already imported (unchanged)
# 25 new files to import
# 5 modified files to re-import
# Total to process: 30 files

Command-Line Options#

Available Options:

source_dir: Path to directory containing PDFs (positional argument)
--state-file: Import state file (default: from PAPERS_STATE_FILE or data/local_import_state.json)
--dry-run: List files without importing
--max-files: Maximum number of files to import
--verbose, -v: Enable verbose logging

Examples#

# Use .env configuration (simplest)
python scripts/import_local_papers.py

# Dry run to preview
python scripts/import_local_papers.py --dry-run

# Specify path directly (overrides .env)
python scripts/import_local_papers.py /Volumes/Share/Papers

# Preview first 5 files
python scripts/import_local_papers.py --dry-run --max-files 5

# Import only 10 files at a time
python scripts/import_local_papers.py --max-files 10

# Verbose logging for debugging
python scripts/import_local_papers.py --verbose

How It Works#

Import Process#

Scan for PDFs: Recursively lists all PDF files in the source directory
Filter New Files: Compares against state file to find new/modified files
Upload to MMORE: Sends PDF file to MMORE service for processing
Retry on Failure: Automatically retries failed uploads with exponential backoff (3 attempts)
Record the file id: Stores the MMORE file_id and file metadata in the JSON state file
Update State: Records imported files with metadata
Save State: Persists state after each successful upload

Indexing is asynchronous#

Step 3 is not a single request. MMORE v2 accepts an upload with 202 Accepted and a jobId, then indexes on a background queue; MMOREClient.upload_file polls GET /v1/jobs/{jobId} until the job reaches a terminal state so that the call still behaves synchronously from the caller’s point of view.

The wait is bounded by MMORE_INDEX_TIMEOUT (seconds, default 900). A large scanned PDF on CPU can genuinely take minutes, so this is the setting to raise before concluding that an import is broken.

Note

A timeout does not mean the document was rejected. The client stops waiting; the server keeps indexing — nothing cancels the job. The importer then retries the upload, MMORE answers 409 because the fileId already exists, and the retry is logged as “… is already indexed”. So a slow file typically shows up as a long pause, a Retrying in Ns… warning and then an “already indexed” line — and ends up correctly indexed.

Genuine failures look different: a job that reaches failed raises MMOREIndexingError, which is treated as non-retryable and stops that file immediately (unsupported media type, corrupt input).

Check what actually made it in with python scripts/inspect_mmore.py --list rather than inferring it from the import log.

State Tracking#

The script maintains a state file (configured via PAPERS_STATE_FILE or default data/local_import_state.json) that tracks:

File path (relative to source directory)
File hash (size + modification time)
Import timestamp
MMORE file ID
Full file path

This enables incremental updates - only new or modified files are re-imported.

File ID Generation#

Each uploaded document receives a unique file ID generated from its relative path:

file_id = "papers_subdir_filename"  # Path separators replaced with underscores

This file ID is used to:

Track documents in MMORE service
Query specific documents
Delete documents when needed

Scheduling Automatic Imports#

Using Cron (Linux/macOS)#

Add to your crontab to run daily at 2 AM:

crontab -e

Add this line:

0 2 * * * cd /path/to/engiai && /path/to/python scripts/import_local_papers.py >> data/local_import.log 2>&1

Using Task Scheduler (Windows)#

Open Task Scheduler
Create New Task
Set trigger (e.g., daily at 2 AM)

Set action to run:

C:\path\to\python.exe C:\path\to\engiai\scripts\import_local_papers.py

Set working directory: C:\path\to\engiai

Using systemd Timer (Linux)#

Create /etc/systemd/system/paper-import.service:

[Unit]
Description=Import Papers to RAG
After=network.target

[Service]
Type=oneshot
User=youruser
WorkingDirectory=/path/to/engiai
EnvironmentFile=/path/to/engiai/.env
ExecStart=/path/to/python scripts/import_local_papers.py
StandardOutput=append:/path/to/engiai/data/local_import.log
StandardError=append:/path/to/engiai/data/local_import.log

[Install]
WantedBy=multi-user.target

Create /etc/systemd/system/paper-import.timer:

[Unit]
Description=Run Paper Import Daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Enable and start:

sudo systemctl enable paper-import.timer
sudo systemctl start paper-import.timer

Using launchd (macOS)#

Create ~/Library/LaunchAgents/com.engiai.paper-import.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.engiai.paper-import</string>
    <key>ProgramArguments</key>
    <array>
        <string>/path/to/python</string>
        <string>/path/to/engiai/scripts/import_local_papers.py</string>
    </array>
    <key>WorkingDirectory</key>
    <string>/path/to/engiai</string>
    <key>EnvironmentVariables</key>
    <dict>
        <key>PATH</key>
        <string>/usr/local/bin:/usr/bin:/bin</string>
    </dict>
    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>2</integer>
        <key>Minute</key>
        <integer>0</integer>
    </dict>
    <key>StandardOutPath</key>
    <string>/path/to/engiai/data/local_import.log</string>
    <key>StandardErrorPath</key>
    <string>/path/to/engiai/data/local_import.log</string>
</dict>
</plist>

Load it:

launchctl load ~/Library/LaunchAgents/com.engiai.paper-import.plist

Querying Imported Documents#

Using the Streamlit UI#

The easiest way to query imported papers is through the chat interface. The RAG agent will automatically search your indexed documents:

"What are the key findings in recent papers about FEA?"
"What does the EngiBench paper say about topology optimization?"

Filtering by Source#

Query specific documents using their file IDs:

from src.tools import MMOREClient

mmore_client = MMOREClient()
results = mmore_client.retrieve(
    query="FEA optimization",
    file_ids=["papers_subfolder_paper1", "papers_subfolder_paper2"],
    max_matches=5,
)

Via Streamlit UI#

The imported papers are automatically available in the Streamlit chat interface:

streamlit run src/ui/streamlit_app.py

Just ask questions and the RAG system will search through all documents in MMORE.

Inspecting the Documents#

Using the Inspector Script#

# Show all uploaded documents
python scripts/inspect_mmore.py

# List unique source files
python scripts/inspect_mmore.py --sources

# List all document IDs
python scripts/inspect_mmore.py --list

Via the MMORE client#

Check the files indexed in MMORE:

from src.tools import MMOREClient

mmore_client = MMOREClient()
file_ids = mmore_client.list_files()

for file_id in file_ids:
    print(file_id)

Troubleshooting#

Configuration Not Found#

Problem: Script says “Missing source directory”

Solutions:

Check that PAPERS_SOURCE_DIR is set in your .env file
Verify .env file is in the project root directory
Try providing path via command line: python scripts/import_local_papers.py /path/to/papers

MMORE Connection Issues#

Problem: “Connection aborted” or “Remote end closed connection without response”

Solutions:

Verify MMORE service is running: docker ps | grep mmore
Check MMORE_RAG_URL is correct in .env (default: http://localhost:8000)
Test MMORE health: curl http://localhost:8000/
For large files (>50 MB), raise MMORE_INDEX_TIMEOUT (default 900 seconds) or upload in smaller batches with --max-files. See Indexing is asynchronous — a timeout does not mean the file was rejected.
Check MMORE service logs: docker logs mmore-rag-service

File Access Issues#

Problem: Cannot read files from directory

Solutions:

Verify the path exists: ls -la $PAPERS_SOURCE_DIR
Check file permissions: ls -l /path/to/papers/*.pdf
For mounted shares, ensure the mount is active: mount | grep Volumes
Try with verbose mode: --verbose

No Files Found#

Problem: Script reports “0 PDF files found”

Solutions:

Check the source directory path is correct
Verify files have .pdf extension (case-insensitive)
Look for hidden files (those starting with . are skipped)
Run with --verbose to see scanning process

File Processing Errors#

Problem: PDFs fail to upload

Solutions:

Check that files are valid PDFs (try opening them)
Verify sufficient disk space for temporary files
Look at detailed logs with --verbose
Check MMORE service has enough memory (increase in docker-compose.yml if needed)
Try processing with --max-files 5 to upload in smaller batches
The script will automatically retry failed uploads 3 times with exponential backoff

State File Corruption#

Problem: State file is corrupted

Solution: Delete the state file to start fresh (will re-import all files):

rm data/local_import_state.json

Mount Disappears#

Problem: Mounted network share disconnects

Solutions:

Use macOS auto-mount on login (Finder → Go → Connect to Server → Add to login items)
Set up automatic reconnection in your scheduling script

Add mount check before import:

#!/bin/bash
if [ ! -d "/Volumes/Soheyl" ]; then
  mount_smbfs //user@server/share /Volumes/Soheyl
fi
python scripts/import_local_papers.py

Logs#

The script maintains logs in two places:

File Log: data/local_import.log - Persistent log of all imports
Console Output: Real-time progress and summary

Log format:

2025-11-06 10:30:15 - INFO - Scanning for PDF files in /Volumes/Papers...
2025-11-06 10:30:16 - INFO - Found 42 PDF files
2025-11-06 10:30:17 - INFO - New file: paper_2025.pdf
2025-11-06 10:30:18 - INFO - Processing paper_2025.pdf...
2025-11-06 10:30:22 - INFO - Successfully imported paper_2025.pdf (15 chunks)

Best Practices#

Use .env: Store configuration in .env for easy management
Test First: Always run with --dry-run before first import
Start Small: Use --max-files to test with a few files first
Monitor Logs: Check data/local_import.log for errors
Backup State: Keep backups of data/local_import_state.json
Schedule Off-Hours: Run imports during low-usage times
Verify Mounts: Ensure network shares are mounted before scheduled runs
Clean Logs: Periodically rotate or clean old log files

Performance Considerations#

File Size: Large PDFs take longer to process
File Count: Processing many files can take time
Embedding Generation: OpenAI API calls are rate-limited
Disk I/O: Reading from network shares is slower than local disk

Optimization Tips:

Use --max-files to process in batches
Schedule during off-peak hours
Consider copying files locally for faster processing
Monitor OpenAI API usage

Advanced Usage#

Multiple Sources#

Import from multiple directories with different .env values:

# Set different source for one-off import
PAPERS_SOURCE_DIR=/Volumes/OtherShare/Papers python scripts/import_local_papers.py

# Or override with command line
python scripts/import_local_papers.py /Volumes/OtherShare/Papers

Separating imports#

# The importer has no collection flag — every document goes into the single
# MMORE collection configured in services/mmore/retriever_config.yaml
# (`collection_name: my_docs`). Separate the runs by state file instead:
python scripts/import_local_papers.py /Volumes/Engineering/Papers \
  --state-file data/engineering_import_state.json

python scripts/import_local_papers.py /Volumes/Research/Papers \
  --state-file data/research_import_state.json

Programmatic Usage#

Use the importer in your own scripts:

from scripts.import_local_papers import LocalPaperImporter

importer = LocalPaperImporter(source_dir="/path/to/papers")

stats = importer.run(dry_run=False, max_files=10)
print(f"Imported {stats['successful']} files")

Support#

For issues or questions:

Check logs in data/local_import.log
Run with --verbose for detailed debugging
Review this guide’s troubleshooting section
Verify .env configuration is correct
Check project README for general setup issues

Quick Reference#

Common Commands#

Command	Description
`python scripts/import_local_papers.py`	Import all new/modified papers from `PAPERS_SOURCE_DIR`
`python scripts/import_local_papers.py --dry-run`	Preview what will be imported
`python scripts/import_local_papers.py --verbose`	Show detailed progress
`python scripts/import_local_papers.py --max-files 10`	Import only first 10 files
`python scripts/inspect_mmore.py`	List all uploaded documents
`python scripts/inspect_mmore.py --sources`	List unique source files

Key Files#

File	Purpose
`.env`	Configuration (PAPERS_SOURCE_DIR, MMORE_RAG_URL)
`data/local_import_state.json`	Tracks file metadata and upload status
`data/local_import.log`	Import operation logs
`scripts/import_local_papers.py`	Main import script
`scripts/inspect_mmore.py`	Document inspection tool

Retry Behavior#

Attempt	Wait Time	Total Time
1	0s	0s
2	5s	5s
3	10s	15s
Failed	-	Give up after 15s

Files larger than 50MB get extra warnings but still upload with retry logic.

Paper Import Guide

Contents

Paper Import Guide#

Overview#

Workflow#

Installation#

1. Install Dependencies#

2. Verify MMORE Service#

3. Configure Import Path#

Usage#

Quick Start#

Command-Line Usage#

Dry-Run Mode#

Command-Line Options#

Examples#

How It Works#

Import Process#

Indexing is asynchronous#

State Tracking#

File ID Generation#

Scheduling Automatic Imports#

Using Cron (Linux/macOS)#

Using Task Scheduler (Windows)#

Using systemd Timer (Linux)#

Using launchd (macOS)#

Querying Imported Documents#

Using the Streamlit UI#

Filtering by Source#

Via Streamlit UI#

Inspecting the Documents#

Using the Inspector Script#

Via the MMORE client#

Troubleshooting#

Configuration Not Found#

MMORE Connection Issues#

File Access Issues#

No Files Found#

File Processing Errors#

State File Corruption#

Mount Disappears#

Logs#

Best Practices#

Performance Considerations#

Advanced Usage#

Multiple Sources#

Separating imports#

Programmatic Usage#

Example: Network Share#

Support#

Quick Reference#

Common Commands#

Key Files#

Retry Behavior#