2. Running on Alliance (Compute Canada) clusters

The Digital Research Alliance of Canada (formerly Compute Canada) runs the national clusters Rorqual, Fir, Nibi, Trillium and Narval. LatticeMind can send a conversation’s calculations there as Slurm jobs: the conversation and the model stay on your machine, each approved run becomes one sbatch job, and the results come back into your project.

Alliance clusters ask for multifactor authentication (MFA) on every new SSH login. LatticeMind never starts a login itself. You open one shared SSH connection per day, answer the MFA prompt once, and LatticeMind sends all its commands through that connection.

/remote setup alliance walks you through everything below, in the terminal or in the browser (Where calculations run → Set up an Alliance cluster…). Every step first checks what is already in place, so you can stop at any point and run it again to continue.

2.1. Before you start

You need:

  • A CCDB account with an active role under your supervisor’s (PI’s) allocation: https://ccdb.alliancecan.ca. Your cluster username is your CCDB username.

  • MFA enrolment (Duo app, YubiKey or phone): https://docs.alliancecan.ca/wiki/Multifactor_authentication.

  • An SSH key registered in CCDB (My Account → SSH Keys): https://ccdb.alliancecan.ca/ssh_authorized_keys. The setup shows the public key to paste and how to create one if you have none. It can take a few minutes for a new key to reach the clusters.

  • Access to your solver on the cluster. Each solver needs its own licence:

    • VASP: the Alliance vasp module is only visible to members of a VASP licence group (request it in CCDB with proof of your group’s licence). If you compiled VASP yourself under your licence, give the setup the full path of your build instead, for example ~/apps/vasp/6.5.1/bin/vasp_std.

    • RESCU and NanoDCAL: your own installation and licence on the cluster.

  • On Windows: WSL. The OpenSSH that comes with Windows cannot share one connection between commands, so LatticeMind uses OpenSSH inside WSL. If you do not have WSL yet, run wsl --install -d Ubuntu-24.04 in an administrator PowerShell, restart, and open Ubuntu once to create your Linux user. On Linux and macOS your own ssh is used.

2.2. Setting up

Type, in a LatticeMind conversation:

/remote setup alliance

The setup asks, step by step:

  1. Cluster. Pick Rorqual (rorqual.alliancecan.ca), Fir, Nibi, Trillium, Narval, or another Slurm cluster (you enter its login hostname). You can also name it directly: /remote setup alliance rorqual yourname.

  2. This machine. On Windows it picks your WSL distribution (it asks if you have several).

  3. Username, as in CCDB.

  4. SSH key. It looks in the ~/.ssh that the cluster’s ssh reads (inside WSL on Windows). If your key is only in the Windows C:\Users\you\.ssh, it offers to copy it into WSL with mode 600: WSL’s ssh refuses keys under /mnt/c because Windows reports them as readable by everyone. The copy happens only after you say yes, and an existing file is never replaced. If you have no key, it prints the ssh-keygen command to run. LatticeMind never asks for, or sees, your key passphrase.

  5. SSH configuration. It shows the exact lines it would append to ~/.ssh/config (inside WSL on Windows), for example:

    # BEGIN LatticeMind cluster: rorqual
    Host rorqual
      HostName rorqual.alliancecan.ca
      User yourname
      IdentityFile ~/.ssh/id_ed25519
      IdentitiesOnly yes
      ControlMaster auto
      ControlPath ~/.ssh/cm/%r@%h:%p
      ControlPersist 12h
      ServerAliveInterval 60
    # END LatticeMind cluster: rorqual
    

    Nothing is written until you confirm. The previous file is copied to config.latticemind-backup-<date>, the block is appended at the end, the file’s line endings are kept, and your other entries are not touched. If you already have your own Host rorqual entry, it is left as it is; the setup only tells you if it lacks the three Control... lines, and otherwise uses that entry’s own ControlPath.

  6. The daily connection. It prints the one command to run in a terminal, for example on Windows:

    wsl -d Ubuntu-24.04 -e ssh -fN rorqual
    

    ssh may ask for your key’s passphrase, then Duo shows a short menu such as 1. Duo Push to XXX-XXX-1234 and Passcode or option (1-3):. Type the number of the option you want and approve it on your phone (or type a passcode). The command returns to the prompt and the connection stays open in the background for 12 hours. The first time, ssh asks you to confirm the cluster’s host key; compare it with the fingerprint on the Alliance documentation page for that cluster before typing yes. LatticeMind checks the connection with ssh -O check, which only looks at the local connection and never triggers an MFA prompt.

  7. Your account (read-only). Through that connection it reads: the login node’s name, $SCRATCH, the python3 version (the remote supervisor needs 3.7 or later), whether sbatch/squeue exist, your Slurm accounts (sshare), the GPU types (sinfo), whether module spider finds your solver, and whether the build path you gave is executable.

  8. Profile. It proposes a profile from what it found, for example on Rorqual with a GPU build of VASP:

    [profiles.rorqual-gpu]
    host = "rorqual"
    scheduler = "slurm"
    remote_root = "/scratch/yourname/latticemind/runs"
    transfer = "scp"
    rescu_command = "rescu"
    vasp_command = "/home/yourname/apps/vasp/6.5.1/bin/vasp_std"
    shell_preamble = "module --force purge; export CUDA_CACHE_PATH=/home/yourname/.nv/ComputeCache; export CUDA_CACHE_MAXSIZE=4294967296"
    account = "def-yourpi_gpu"
    time = "03:00:00"
    nodes = 1
    ntasks = 1
    cpus_per_task = 1
    mem = "32G"
    gpus = "nvidia_h100_80gb_hbm3_3g.40gb:1"
    modules = "StdEnv/2023 nvhpc/25.1 openmpi/5.0.3"
    launcher = "srun --mpi=pmix"
    ssh_via_wsl = "Ubuntu-24.04"
    control_path = "~/.ssh/cm/%r@%h:%p"
    control_persist = "12h"
    

    You choose the CPU or GPU template, the account (def-... for the default allocation, rrg-... for a competition allocation; on Rorqual _cpu and _gpu accounts are separate), the GPU type (a full H100 or a MIG slice such as 3g.40gb), the MPI library your build uses (srun --mpi=pmix for OpenMPI) and the modules your build was compiled with. module --force purge comes first because the login shell loads an older StdEnv automatically. The profile is appended to ~/.latticemind/remote_profiles.toml only after you confirm, with a backup of the previous file.

The setup offers to use the new profile for the solver you named. In a later conversation, /remote rorqual-gpu vasp sends its VASP calculations there (name the solver: without it, the conversation’s default solver is used). Ask for a calculation as usual; each launch shows the batch script and the launch script and asks for your approval. In the terminal, /compute slurm rorqual-gpu time=06:00:00 changes the resources for later runs; a run can last at most 24 hours. Compute Canada examples: calculations on Rorqual walks through two real projects on Rorqual.

2.3. Daily use

  1. Open the connection once per day (or after your laptop slept for long):

    wsl -d Ubuntu-24.04 -e ssh -fN rorqual
    
  2. Start LatticeMind in your project folder and work as usual.

  3. /remote status rorqual-gpu (or the Status button in the browser) shows whether the connection is open, LatticeMind’s own jobs in the queue (names starting lm-) with their state, reason and elapsed time, and any run of this project that is waiting for the connection.

If the connection has closed, nothing is lost. A run that was not sent yet says so and can be launched again after you reconnect; a job already in the queue keeps running on the cluster, and LatticeMind collects it once you reconnect and ask about it.

2.4. Costs and allocations

Alliance compute time is free to use but not unlimited: every job is charged to the Slurm account in the profile, and heavy use lowers that account’s priority (its fair-share) for everyone in your group. Request only what a run needs: the time limit and the number of cores or GPUs. A GPU or a whole node counts for much more than a few cores. Long time limits wait longer in the queue. Check your group’s usage with sshare -U -u $USER on the cluster or in CCDB. LatticeMind runs only what you approve, and never resubmits a job by itself.

2.5. Troubleshooting

No open SSH master connection

The daily connection is closed. Run the ssh -fN command above and try again. Nothing was sent to the cluster.

The shared cluster connection ... is busy (too many simultaneous sessions)

Many runs use the one connection at the same time, and the cluster limits the sessions open on one connection (usually 10). The connection and your MFA are fine and nothing needs to be done: LatticeMind opens at most 6 sessions at once across all its runs, waits for a free one and retries reads such as status checks and result downloads. A job submission is never repeated; if its answer was lost, the next check finds the job by name. To change the limit, add max_sessions = N (1 to 64) to the profile. If you use the same connection for other terminals at the same time, a lower value leaves them room.

Permission denied (publickey) when connecting

The key is not registered in CCDB yet (or has not reached the cluster), or ssh uses another key. Check the IdentityFile line and CCDB.

UNPROTECTED PRIVATE KEY FILE or bad permissions

On Windows, the key is read from /mnt/c. Let the setup copy it into WSL, or run chmod 600 ~/.ssh/<key> inside WSL.

The job fails at once with module: command not found or the wrong compiler

The profile needs module --force purge in shell_preamble and the full module list, starting with StdEnv/2023.

python3 older than 3.7

The remote supervisor needs Python 3.7 or later on the cluster; LatticeMind’s supervisor is kept compatible with 3.7, the login default on Rorqual.

PMIx or MPI_Init errors from an OpenMPI build

Use launcher = "srun --mpi=pmix".

VASP finished but produced nothing useful

VASP can exit with status 0 after an error. LatticeMind checks the output files, not only the exit status, and reports an unconverged or failed run as such; read remote-solver.log in the run’s results.

The first GPU run is slow before anything happens

A GPU build compiled for another GPU architecture is compiled again (“JIT”) when it starts. The GPU template keeps that work in CUDA_CACHE_PATH under your home, so later runs start quickly.

A job ended with TIMEOUT or OUT_OF_MEMORY

Raise time or mem with /compute slurm <profile> time=... and ask for the run again; LatticeMind reports the Slurm state and the end of the logs and does not retry by itself.

2.6. What LatticeMind runs on the cluster

Everything goes through the connection you opened, as your own user:

  • during setup and /remote status: read-only commands about your own account (hostname, $SCRATCH, python3 --version, sshare -U -u $USER, sinfo, module spider, test -x on the path you gave, squeue --me);

  • for each run you approve: a new folder under remote_root, the upload of that run’s input files and LatticeMind’s small supervisor script (remote-agent.py, run with the cluster’s python3), one sbatch of the batch script you saw, queue checks with squeue --me and sacct for that job, status reads by the supervisor script, and the download of its outputs;

  • when you ask to stop a run: scancel of that job only.

It does not read other users’ files, does not look outside its run folders except for the reads listed above, and does not keep your password, passphrase or MFA codes. Your inputs and outputs stay in your own scratch space; the Alliance’s own policies apply to that data (scratch is purged after a period of inactivity, so the results LatticeMind fetched into your project are your copy).