3. Remote & HPC execution

LatticeMind builds inputs locally and can offload the actual calculation to a remote workstation or a Slurm HPC cluster. It sends a self-contained job bundle — inputs, scripts, pseudopotentials (and, for NanoDCAL, the basis files), a manifest, and a generated job script — runs it over SSH, and fetches the verified results back.

Important

LatticeMind never sends your API key and never runs the AI agent remotely. Only the solver job is executed on the remote host. It also never stores your password, private key, or 2FA code.

3.1. Overview

LatticeMind remote execution flow. — Fig. 3.1.1 Remote execution: LatticeMind bundles the validated job locally, uploads it, submits it (directly or via Slurm), then fetches the verified outputs back into the project.

3.2. Step 1 — Confirm plain SSH works

Before involving LatticeMind, make sure ordinary SSH already connects to your machine or cluster:

ssh my-profile true

LatticeMind drives ssh, scp/rsync, and Slurm commands through your normal SSH configuration. Fix any SSH issues here first.

3.3. Step 2 — Describe the target

Define one or more profiles in ~/.latticemind/remote_profiles.toml.

Direct SSH workstation:

[profiles.office]
host = "my-office-computer"
scheduler = "direct"
remote_root = "/home/myuser/latticemind_runs"
transfer = "scp"
rescu_command = "rescu"
shell_preamble = ""

Slurm cluster:

[profiles.cluster]
host = "my-cluster-login"
scheduler = "slurm"
remote_root = "/scratch/myuser/latticemind_runs"
transfer = "rsync"
rescu_command = "rescu"
shell_preamble = "module load rescu"
account = "my-account"
partition = "short"
time = "02:00:00"
nodes = 1
ntasks = 4
cpus_per_task = 1
mem = "8G"

Table 3.3.1 Remote profile fields
Field	Meaning
`host`	SSH host (as in your `~/.ssh/config`) to connect to.
`scheduler`	`direct` for a plain SSH workstation, `slurm` for an HPC cluster.
`remote_root`	Base directory on the remote host where job bundles are staged.
`transfer`	`scp` or `rsync` for uploading and fetching.
`rescu_command`	The RESCU launch command on the remote host. For NanoDCAL use `nanodcal_command` instead.
`shell_preamble`	Shell lines run before the solver, e.g. `module load rescu`.
`account`, `partition`, `time`, `nodes`, `ntasks`, `cpus_per_task`, `mem`	Slurm resource request for `scheduler = "slurm"`.

3.4. Step 3 — Select and test the profile

From inside LatticeMind:

/remote profiles            # list profiles
/remote profile office      # select one
/remote setup office        # add reusable-SSH settings (helps with 2FA)
/remote warm office         # open one approved SSH session (approve 2FA once)
/remote test office         # check SSH, solver, and scheduler
/compute remote             # or: /compute slurm
/execute on

A convenient shortcut runs the whole selection interactively:

/remote wizard

Tip

If your machine uses 2FA, /remote warm lets you approve it once; LatticeMind reuses that single approved SSH session for upload, submit, status, logs, and fetch.

You can also run remote commands from the shell before opening the app:

latticemind remote setup office
latticemind remote warm office
latticemind remote status office
latticemind remote close office   # close the saved SSH session early

3.5. Step 4 — Launch and track jobs

Before any remote run, LatticeMind shows the exact target, remote directory, upload method, job script, inputs, and Slurm resources for you to review. After submitting, track the job for the active project:

/remote status
/remote logs 80
/remote fetch
/remote fetch latest
/remote workdir
/remote cancel
/remote doctor      # remote/HPC readiness checks

3.6. Serial-only guardrails

Some RESCU post-analysis stages must run serially. LatticeMind parses each input before launch and, for guarded stages such as dfpt-phonon-bs, dfpt-phonon-dos, dfpt-optic, and dfpt-raman, removes MPI launchers and unsafe parallel flags. For Slurm bundles containing only guarded serial stages, it also submits with one node, one task, and one CPU per task. Override the guarded list with LATTICEMIND_RESCU_SERIAL_ONLY_CALC_TYPES only if your RESCU build supports those stages in parallel.

3.7. Runtime profiles (deployment context)

Remote profiles describe how jobs launch. Runtime profiles describe the broader shell/cloud environment and advisory operator instructions LatticeMind should consider when planning run scripts and reports. Set them interactively:

/runtime
/runtime cloud on
/runtime system Linux container with RESCU 3.x under /opt/esapp
/runtime note Slurm jobs must use /scratch/myuser and module load rescu
/runtime prompt Use the shared scratch filesystem for generated job files.
/runtime file /home/myuser/latticemind_operator_prompt.txt
/runtime save

Or via environment variables in cloud deployments:

export LATTICEMIND_CLOUD_MODE=1
export LATTICEMIND_RUNTIME_PROFILE=cluster
export LATTICEMIND_SYSTEM_DESCRIPTION="Linux container with RESCU under /opt/esapp"
export LATTICEMIND_ENVIRONMENT_NOTES="Use /scratch/myuser; module load rescu before jobs"
export LATTICEMIND_OPERATOR_PROMPT_FILE=/home/myuser/latticemind_operator_prompt.txt
export LATTICEMIND_EXECUTION_BACKEND=slurm
export LATTICEMIND_REMOTE_PROFILE=cluster
export LATTICEMIND_REMOTE_SCHEDULER=slurm

Or persist them in ~/.latticemind/config.json (or a file named by LATTICEMIND_CONFIG_FILE):

{
  "runtime_profile": {
    "runtime_profile_name": "cluster",
    "cloud_mode": true,
    "system_description": "Linux container with RESCU installed under /opt/esapp",
    "environment_notes": "Use /scratch/myuser; module load rescu before jobs",
    "operator_instructions": "Use the shared scratch filesystem for generated job files.",
    "execution_backend": "slurm",
    "remote_profile": "cluster",
    "remote_scheduler": "slurm",
    "rescu_command": "/opt/esapp/rescu"
  }
}

Note

Operator instructions are advisory deployment context. They do not enable or disable solver execution. Use /execute on or /execute off for interactive work; agent commands require both --execute and --approve-execution before a solver may launch.