2. Running on Alliance (Compute Canada) clusters
The Digital Research Alliance of Canada (formerly Compute Canada) runs the
national clusters Rorqual, Fir, Nibi, Trillium and Narval. LatticeMind can send
a conversation’s calculations there as Slurm jobs: the conversation and the
model stay on your machine, each approved run becomes one sbatch job, and
the results come back into your project.
Alliance clusters ask for multifactor authentication (MFA) on every new SSH login. LatticeMind never starts a login itself. You open one shared SSH connection per day, answer the MFA prompt once, and LatticeMind sends all its commands through that connection.
/remote setup alliance walks you through everything below, in the terminal
or in the browser (Where calculations run → Set up an Alliance cluster…).
Every step first checks what is already in place, so you can stop at any point
and run it again to continue.
2.1. Before you start
You need:
A CCDB account with an active role under your supervisor’s (PI’s) allocation: https://ccdb.alliancecan.ca. Your cluster username is your CCDB username.
MFA enrolment (Duo app, YubiKey or phone): https://docs.alliancecan.ca/wiki/Multifactor_authentication.
An SSH key registered in CCDB (My Account → SSH Keys): https://ccdb.alliancecan.ca/ssh_authorized_keys. The setup shows the public key to paste and how to create one if you have none. It can take a few minutes for a new key to reach the clusters.
Access to your solver on the cluster. Each solver needs its own licence:
VASP: the Alliance
vaspmodule is only visible to members of a VASP licence group (request it in CCDB with proof of your group’s licence). If you compiled VASP yourself under your licence, give the setup the full path of your build instead, for example~/apps/vasp/6.5.1/bin/vasp_std.RESCU and NanoDCAL: your own installation and licence on the cluster.
On Windows: WSL. The OpenSSH that comes with Windows cannot share one connection between commands, so LatticeMind uses OpenSSH inside WSL. If you do not have WSL yet, run
wsl --install -d Ubuntu-24.04in an administrator PowerShell, restart, and open Ubuntu once to create your Linux user. On Linux and macOS your ownsshis used.
2.2. Setting up
Type, in a LatticeMind conversation:
/remote setup alliance
The setup asks, step by step:
Cluster. Pick Rorqual (
rorqual.alliancecan.ca), Fir, Nibi, Trillium, Narval, or another Slurm cluster (you enter its login hostname). You can also name it directly:/remote setup alliance rorqual yourname.This machine. On Windows it picks your WSL distribution (it asks if you have several).
Username, as in CCDB.
SSH key. It looks in the
~/.sshthat the cluster’s ssh reads (inside WSL on Windows). If your key is only in the WindowsC:\Users\you\.ssh, it offers to copy it into WSL with mode 600: WSL’s ssh refuses keys under/mnt/cbecause Windows reports them as readable by everyone. The copy happens only after you say yes, and an existing file is never replaced. If you have no key, it prints thessh-keygencommand to run. LatticeMind never asks for, or sees, your key passphrase.SSH configuration. It shows the exact lines it would append to
~/.ssh/config(inside WSL on Windows), for example:# BEGIN LatticeMind cluster: rorqual Host rorqual HostName rorqual.alliancecan.ca User yourname IdentityFile ~/.ssh/id_ed25519 IdentitiesOnly yes ControlMaster auto ControlPath ~/.ssh/cm/%r@%h:%p ControlPersist 12h ServerAliveInterval 60 # END LatticeMind cluster: rorqual
Nothing is written until you confirm. The previous file is copied to
config.latticemind-backup-<date>, the block is appended at the end, the file’s line endings are kept, and your other entries are not touched. If you already have your ownHost rorqualentry, it is left as it is; the setup only tells you if it lacks the threeControl...lines, and otherwise uses that entry’s ownControlPath.The daily connection. It prints the one command to run in a terminal, for example on Windows:
wsl -d Ubuntu-24.04 -e ssh -fN rorqual
ssh may ask for your key’s passphrase, then Duo shows a short menu such as
1. Duo Push to XXX-XXX-1234andPasscode or option (1-3):. Type the number of the option you want and approve it on your phone (or type a passcode). The command returns to the prompt and the connection stays open in the background for 12 hours. The first time, ssh asks you to confirm the cluster’s host key; compare it with the fingerprint on the Alliance documentation page for that cluster before typingyes. LatticeMind checks the connection withssh -O check, which only looks at the local connection and never triggers an MFA prompt.Your account (read-only). Through that connection it reads: the login node’s name,
$SCRATCH, thepython3version (the remote supervisor needs 3.7 or later), whethersbatch/squeueexist, your Slurm accounts (sshare), the GPU types (sinfo), whethermodule spiderfinds your solver, and whether the build path you gave is executable.Profile. It proposes a profile from what it found, for example on Rorqual with a GPU build of VASP:
[profiles.rorqual-gpu] host = "rorqual" scheduler = "slurm" remote_root = "/scratch/yourname/latticemind/runs" transfer = "scp" rescu_command = "rescu" vasp_command = "/home/yourname/apps/vasp/6.5.1/bin/vasp_std" shell_preamble = "module --force purge; export CUDA_CACHE_PATH=/home/yourname/.nv/ComputeCache; export CUDA_CACHE_MAXSIZE=4294967296" account = "def-yourpi_gpu" time = "03:00:00" nodes = 1 ntasks = 1 cpus_per_task = 1 mem = "32G" gpus = "nvidia_h100_80gb_hbm3_3g.40gb:1" modules = "StdEnv/2023 nvhpc/25.1 openmpi/5.0.3" launcher = "srun --mpi=pmix" ssh_via_wsl = "Ubuntu-24.04" control_path = "~/.ssh/cm/%r@%h:%p" control_persist = "12h"
You choose the CPU or GPU template, the account (
def-...for the default allocation,rrg-...for a competition allocation; on Rorqual_cpuand_gpuaccounts are separate), the GPU type (a full H100 or a MIG slice such as3g.40gb), the MPI library your build uses (srun --mpi=pmixfor OpenMPI) and the modules your build was compiled with.module --force purgecomes first because the login shell loads an olderStdEnvautomatically. The profile is appended to~/.latticemind/remote_profiles.tomlonly after you confirm, with a backup of the previous file.
The setup offers to use the new profile for the solver you named. In a later
conversation, /remote rorqual-gpu vasp sends its VASP calculations there
(name the solver: without it, the conversation’s default solver is used).
Ask for a calculation as usual; each launch shows the batch script and the
launch script and asks for your approval. In the terminal, /compute slurm
rorqual-gpu time=06:00:00 changes the resources for later runs; a run can
last at most 24 hours. Compute Canada examples: calculations on Rorqual walks through two real
projects on Rorqual.
2.3. Daily use
Open the connection once per day (or after your laptop slept for long):
wsl -d Ubuntu-24.04 -e ssh -fN rorqual
Start LatticeMind in your project folder and work as usual.
/remote status rorqual-gpu(or the Status button in the browser) shows whether the connection is open, LatticeMind’s own jobs in the queue (names startinglm-) with their state, reason and elapsed time, and any run of this project that is waiting for the connection.
If the connection has closed, nothing is lost. A run that was not sent yet says so and can be launched again after you reconnect; a job already in the queue keeps running on the cluster, and LatticeMind collects it once you reconnect and ask about it.
2.4. Costs and allocations
Alliance compute time is free to use but not unlimited: every job is charged to
the Slurm account in the profile, and heavy use lowers that account’s priority
(its fair-share) for everyone in your group. Request only what a run needs:
the time limit and the number of cores or GPUs. A GPU or a whole node counts for
much more than a few cores. Long time limits wait longer in the queue. Check
your group’s usage with sshare -U -u $USER on the cluster or in CCDB.
LatticeMind runs only what you approve, and never resubmits a job by itself.
2.5. Troubleshooting
No open SSH master connectionThe daily connection is closed. Run the
ssh -fNcommand above and try again. Nothing was sent to the cluster.The shared cluster connection ... is busy (too many simultaneous sessions)Many runs use the one connection at the same time, and the cluster limits the sessions open on one connection (usually 10). The connection and your MFA are fine and nothing needs to be done: LatticeMind opens at most 6 sessions at once across all its runs, waits for a free one and retries reads such as status checks and result downloads. A job submission is never repeated; if its answer was lost, the next check finds the job by name. To change the limit, add
max_sessions = N(1 to 64) to the profile. If you use the same connection for other terminals at the same time, a lower value leaves them room.Permission denied (publickey)when connectingThe key is not registered in CCDB yet (or has not reached the cluster), or ssh uses another key. Check the
IdentityFileline and CCDB.UNPROTECTED PRIVATE KEY FILEorbad permissionsOn Windows, the key is read from
/mnt/c. Let the setup copy it into WSL, or runchmod 600 ~/.ssh/<key>inside WSL.- The job fails at once with
module: command not foundor the wrong compiler The profile needs
module --force purgeinshell_preambleand the full module list, starting withStdEnv/2023.python3older than 3.7The remote supervisor needs Python 3.7 or later on the cluster; LatticeMind’s supervisor is kept compatible with 3.7, the login default on Rorqual.
PMIxorMPI_Initerrors from an OpenMPI buildUse
launcher = "srun --mpi=pmix".- VASP finished but produced nothing useful
VASP can exit with status 0 after an error. LatticeMind checks the output files, not only the exit status, and reports an unconverged or failed run as such; read
remote-solver.login the run’s results.- The first GPU run is slow before anything happens
A GPU build compiled for another GPU architecture is compiled again (“JIT”) when it starts. The GPU template keeps that work in
CUDA_CACHE_PATHunder your home, so later runs start quickly.- A job ended with
TIMEOUTorOUT_OF_MEMORY Raise
timeormemwith/compute slurm <profile> time=...and ask for the run again; LatticeMind reports the Slurm state and the end of the logs and does not retry by itself.
2.6. What LatticeMind runs on the cluster
Everything goes through the connection you opened, as your own user:
during setup and
/remote status: read-only commands about your own account (hostname,$SCRATCH,python3 --version,sshare -U -u $USER,sinfo,module spider,test -xon the path you gave,squeue --me);for each run you approve: a new folder under
remote_root, the upload of that run’s input files and LatticeMind’s small supervisor script (remote-agent.py, run with the cluster’spython3), onesbatchof the batch script you saw, queue checks withsqueue --meandsacctfor that job, status reads by the supervisor script, and the download of its outputs;when you ask to stop a run:
scancelof that job only.
It does not read other users’ files, does not look outside its run folders except for the reads listed above, and does not keep your password, passphrase or MFA codes. Your inputs and outputs stay in your own scratch space; the Alliance’s own policies apply to that data (scratch is purged after a period of inactivity, so the results LatticeMind fetched into your project are your copy).