2. Advanced installation

2.1. LAN and cluster licenses

If you purchased the LAN or cluster version of RESCU, you simply need to run nano_infogen_linux on a login node to generate dat.inf. Then upload it on the client portal and follow the instructions there to complete the activation.

Note

If you are using RESCU < 2.7.0, you need to point the licenser to the installation node as follows. The information in licenseNumber.txt and license.dat will only unlock RESCU on the node on which it is installed. To avail RESCU on the entire cluster, follow these instructions. Use a login or gateway node to install RESCU. The node’s IP address should be in the ARP cache of all worker nodes (it most likely is already, so you likely need not worry about this). Then create the file ip.dat in the license directory. Add the inet addresses of the gateway node in ip.dat, one by line. For instance,

0.150.188.70
0.16.205.3
0.150.188.73
0.20.205.3

The inet addresses can be listed using the command

ifconfig | grep inet
inet addr:192.168.1.29 Bcast:192.168.1.255  Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe70:e3f5/64 Scope:Link

What you seek here is 192.168.1.29.

2.2. Dependencies

RESCU is mostly written in MATLAB, and hence it does not need to be compiled. Additional functionalities require external programs and libraries which must be compiled however. MATLAB calls these procedures through MEX files, which must be compiled. For one, the atomic orbital calculations require such MEX files. In this instance, external libraries need not be installed and linked, and hence the makeRESCU function deals with it by itself. Otherwise, makeRESCU must be provided with the library paths. The purpose of the present document is to demonstrates how to build and install these external libraries and MEX files.

The following is a list of functionalities that require an installation:

  • Calls certain routines in the CUDA library. This is not required to run RESCU using GPUs, but it may improve performance in certain calculations;

  • A set of functions that call LibXC routines. LibXC is an exchange-correlation functional library including hundreds of correlation, exchange and kinetic energy functionals.

  • A set of functions that call MPI and ScaLAPACK routines. MPI, BLAS, LAPACK and ScaLAPACK libraries are required;

The function makeRESCU compiles and installs a package listed above when provided with one of the following flags: -gpu, -libxc, -smi. For example, type the following command to build the SMI package:

makeRESCU -agreeToLicense y -smi libpaths.m

The last command should compile the SMI interface if the mpicc and mpif90 compilers are found in the PATH environment variable. In addition, the file libpaths.m must contain variables providing the paths of certain external libraries. Specific instructions for each package lie below.

2.3. Pre-existing MPI installation

Many platforms now have several libraries preinstalled. Loading them is made easy with modern package managers via the module command for instance. Then the OpenMPI installations can be listed as

username@computer:~$ module spider openmpi

--------------------------------------------------------------
  openmpi:
--------------------------------------------------------------
    Description:
      An open source Message Passing Interface implementation

     Versions:
        openmpi/3.1.3
        openmpi/4.0.1
        openmpi/4.0.3
        openmpi/4.1.1

OpenMPI is then simply loaded typing

module load openmpi/4.0.3

Similarly, one could seek and load LibXC, BLAS, LAPACK and ScaLAPACK as follows

module load openmpi/4.0.3 libxc/4.3.4 openblas/0.3.9 scalapack/2.1.0

Note that OpenBLAS contains both BLAS and LAPACK. Then, one only need to specify how to link in the file libpaths.m

xc        = '/software/2020/avx512/Compiler/gcc9/libxc/4.3.4';
scalapack = '-lscalapack';
blas      = '-lopenblas';
lapack    = '';

2.4. spack installation

If you need to install OpenMPI and other libraries yourself, the simplest way is probably to use a package manager like spack. This section shows how to install the build tools and dependencies using the spack package manager.

Get spack from the github repository typing

git clone https://github.com/spack/spack.git

and add it to your environment as follows

source spack/share/spack/setup-env.sh

You may add the last command to your .bashrc to make this permanent.

Copy spack.yaml file found in documentation and activate the package extension.

cd $RESCU_DIR
cp documentation/spack.yaml .
spacktivate .

GCC and OpenMPI are usually already installed on computing platforms. You may tell spack to use preinstalled compilers and libraries as follows

spack find compiler
spack external find OpenMPI

Next, concretize the environment and let spack build the dependencies as follows

spack concretize -f
spack install -j8

You may adjust the build parallelism using the -j option (here I use 8 cores). When using rescu, just activate the environment with

spacktivate .

If something goes sideways, try to find a local version (e.g. with perl) and reconcretize

spack external find perl
spack concretize -f

Finally, define the file libpaths.m as at the end of section 1.3 to specify how to link the libraries.

2.5. SMI: ScaLAPACK wrapper

In order to exploit efficiently hundreds of processes, RESCU calls ScaLAPACK in heavy computational operations via smi_proxy.c. If ScaLAPACK is already installed on your computer, you may skip the following subsection. Otherwise, here are some instructions to download, configure, compile, install and ScaLAPACK.

2.5.1. Building the ScaLAPACK library

Download the ScaLAPACK source from Nanoacademic. It has a couple modifications to make it work with MATLAB and handle large matrix redistribution operations.

wget https://storage.googleapis.com/rescumat/scalapack-2.1.0.tar.gz
tar -zxf scalapack-2.1.0.tgz
cd scalapack-2.1.0

Then jumps in the ScaLAPACK directory and copy the SLmake.inc.example file to SLmake.inc.

cd scalapack-2.1.0
cp SLmake.inc.example SLmake.inc

MATLAB requires position independent code libraries, which means compiled with the PIC option. Make sure to add -fPIC to the following flags in SLmake.inc

NOOPT   = -O0 -fPIC
FCFLAGS = -O3 -fPIC
CCFLAGS = -O3 -fPIC
CFLAGS  = -O3 -fPIC

Then make the library by typing make lib and install it by copying libscalapack.a in the appropriate location.

2.5.2. ScaLAPACK interface

Before compiling the ScaLAPACK interface, paths to BLAS, LAPACK and ScaLAPACK libraries must be defined. If no BLAS or LAPACK library is installed, we suggest that you install OpenBLAS, which contains both. As mentioned above, this is done in a MATLAB script. Create an M-file and define three variables called blas, lapack, scalapack containing the paths to the BLAS, LAPACK and ScaLAPACK libraries respectively. For example,

cat > /tmp/installpaths.m <<EOF
scalapack = '/usr/lib/libscalapack.a';
lapack = '/usr/lib/libopenblas.a';
blas = '/usr/lib/libopenblas.a';
EOF

Additional flags may be required depending on the compiler. Specify them using the variable CFLAGS in the M-file (e.g. CFLAGS = ’-ifcore’). Finally, compile smi_proxy.c as follows

makeRESCU -smi /tmp/installpaths.m

2.6. Compiling the LibXC wrapper

A few functionals have been implemented in RESCU (LDA_X, LDA_C_PW92, GGA_XC_PBE, MGGA_X_TB09). More functionals are available via the functional library LibXC. LibXC is part of the Easybuild toolchain, and hence you can skip the installation of LibXC if you executed install.sh. Both the Linux system and Windows system procedures to compile and install LibXC are presented in this section.

2.6.1. Linux

To install LibXC on a Linux system, follow the following instructions. Download LibXC (version 4.0 and above for full functionality) and decompress the tarball.

tar -xzf libxc-4.3.4.tar.gz

Enter the LibXC directory, configure, build and install the library.

cd libxc-4.3.4
./configure --prefix=PATH/TO/LIBXC --disable-shared CFLAGS="-O3 -fPIC"
make && make check && make install

Note that MATLAB requires position independent code libraries, which means compiled with the PIC option. We also suggest building and linking static libraries in MATLAB, and hence the --disable-shared configure option.

The LibXC library is now installed. Create an M-file and define the variable xc as the path to the root of the LibXC library installation (the path specified using the --prefix configure option).

cat > /tmp/installpaths.m <<EOF
xc = '/usr/software/libxc-4.3.4';
EOF

Then invoke makeRESCU to compile the LibXC package.

makeRESCU -libxc /tmp/installpaths.m

The file libxc.mexa64 will be created upon successful compilation.

2.6.2. Windows

To install LibXC on Windows, follow this procedure. To begin with, complete MathWorks’ instructions to install the MinGW-w64 Compiler. To install the compiler, use the Add-Ons menu, select the Features box and search for MinGW. This may not be possible if your version of MATLAB is anterior to 2016, but don’t worry there is a way nevertheless.

Install the MinGW-w64 Compilers through MSYS2. The MSYS2 project is hosted on Github. Download and execute the installer; MSYS2 will install itself automatically. It is advised to use the default paths.

Open an MSYS2 terminal using the command C:\\msys64\\msys2_shell.cmd and update the MSYS2 packages as follows

pacman -Sy pacman
pacman -Syu
pacman -Su

Pacman is an Arch Linux packager manager, similar to yum or apt-get on Linux. You can search a package using the command pacman -Ss and install a package using the command pacman -S. Now, install the following packages

pacman -S --needed base-devel mingw-w64-x86_64-toolchain
pacman -S --noconfirm autoconf autoconf-archive automake make libtool

Download LibXC and decompress the tarball. From an MSYS2 shell, configure and make the library.

autoreconf -f -i -I m4
./configure --prefix '/c/lib/libxc-4.3.4' --disable-shared CFLAGS="-O3 -fPIC"
make && make install

If something goes wrong, you may want to refer to the following Stack Overflow ticket , which contains instructions on how to build LibXC with cmake.

If you use make check, note that it generally fails on Windows because round-off errors give slightly different results on Linux, but the error is negligible in practice. MATLAB will look for libraries with the extension .lib. The libraries in /c/lib/libxc-4.3.4 must thus be renamed accordingly.

cd /c/lib/libxc-4.3.4
cp libxc.a libxc.lib

The LibXC library is now installed.

If you were able to install the MinGW-w64 Compiler MATLAB application, you may execute the following instructions. Create an m-file (*.m) and define the variable xc as the path to the root of the LibXC library (the path specified using the --prefix option). Then invoke makeRESCU to compile the LibXC package.

makeRESCU -libxc /tmp/installpaths.m

The file libxc.mexw64 will be created upon successful compilation.

2.6.2.1. Unsuccessful mex compilation

If makeRESCU fails, you may have to compile libxc.mexw64 from the command line. Open an MSYS2 terminal (64-bit). Add mex and gcc to your path as follows

export PATH=/c/Program\ Files/MATLAB/R2017b/bin/win64:$PATH
export PATH=/c/msys64/mingw64/bin:$PATH

Then change directory to rescu-x.x.x/src and write

mex CC=/c/msys64/mingw64/bin/gcc libxc4.c -output libxc \
-I/c/lib/libxc-4.3.4/include -L/c/lib/libxc-4.3.4/lib -lxc

CC specifies the path MSYS2’s GCC compiler. libxc4.c is the LibXC-4 wrapper, replace by libxc.c if using LibXC-3. -I and -L are the paths to the LibXC header files and libraries.

2.6.2.2. Unsuccessful MinGW-w64 install

If you were not able to install the MinGW-w64 Compiler MATLAB application, you may execute the following instructions. To compile the mex file, create the file mexopts.bat as follows (taken from Dr Bogdan Roman)

set MINGWPATH=c:\msys64\mingw64
set PATH=%MINGWPATH%\bin;%PATH%
set PRELINK_CMDS=echo.>%TEMP%\mexstaticlibs
rem set PRELINK_CMDS1=gcc -print-file-name=libwinpthread.a >> %TEMP%\mexstaticlibs
rem set PRELINK_CMDS2=gcc -print-file-name=libquadmath.a >> %TEMP%\mexstaticlibs
rem set PRELINK_CMDS3=...
rem set POSTLINK_CMDS1=upx -9 "%OUTDIR%%MEX_NAME%%MEX_EXT%"
rem set POSTLINK_CMDS2=...
set COMPILER=g++
set COMPFLAGS=-c -I"%MATLAB%\extern\include" -DMATLAB_MEX_FILE
set OPTIMFLAGS=-O3 -funroll-loops -DNDEBUG
set DEBUGFLAGS=-g
set NAME_OBJECT=-o
set LINKER=g++
set LINKFLAGS=-shared -static-libstdc++ -static-libgcc -L"%MATLAB%\bin\win64" -L"%MATLAB%\extern\lib\win64\microsoft" -lmex -lmx -leng -lmat -lmwlapack -lmwblas
set LINKFLAGSPOST=@%TEMP%\mexstaticlibs
set NAME_OUTPUT=-o "%OUTDIR%%MEX_NAME%%MEX_EXT%"

Copy the file to %USERPROFILE%\AppData\Roaming\Mathworks\MATLAB\R2015a, for instance. Then open a MinGW-w64 terminal and add mex to your path, e.g.

export PATH=$PATH:/c/Program\ Files/MATLAB/R2015a/bin/win64

and compile the mex file typing

mex libxc.c -I/c/lib/libxc-4.3.4/include -L/c/lib/libxc-4.3.4/lib -lxc

Again, the file libxc.mexw64 will be created upon successful compilation.

2.7. Compiling the GPU wrapper

RESCU accelerates certain calculations using GPUs. Doing so requires a CUDA installation and compiling a few MEX files. Here is how to do it. Create an M-file in which the variable cuda is defined as the path to your CUDA installation. For example,

cat > /tmp/installpaths.m <<EOF
cuda = '/usr/local/cuda-9.2';
EOF

Then invoke makeRESCU with the GPU option as follows

makeRESCU -gpu /tmp/installpaths.m

2.8. Compiling and using RESCU as a stand-alone executable

With the help of external libraries like OpenMPI and ScaLAPACK, RESCU can use hundreds, even thousands of cores. However, this may require a prohibitive number of MATLAB licenses. This problem is circumvented by compiling RESCU into a standalone program which do not require any license.

Note that MATLAB and its Compiler toolbox are required. Install the optional packages (e.g. LibXC, SMI) that you intend to use. Call the function compileRESCU in MATLAB’s command window to compile RESCU.

matlab -r "compileRESCU('src','bin')"

The first argument specifies the source directory. It should contain the source code and all relevant MEX files. The second argument is the directory where the RESCU application is installed.

To execute RESCU, the freely available MATLAB Compiler Runtime (MCR) is required. Download and install the MCR that matches the MATLAB version that you used to compile RESCU. Next, follow the MCR manual to set up your environment properly. On Windows and Mac, you likely have nothing to do. On linux, you must modify your environment as follows.

export MCR_CACHE_ROOT=/scratch/johndoe
MCRROOT=/software/MCR/v99
export LD_LIBRARY_PATH=$MCRROOT/runtime/glnxa64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$MCRROOT/bin/glnxa64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$MCRROOT/sys/os/glnxa64:$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=$MCRROOT/extern/bin/glnxa64:$LD_LIBRARY_PATH

The variable MCR_CACHE_ROOT is optional but recommended. It tells the MCR where to store its cache files which may grow quite large. Finally, add the RESCU standalone application location to the PATH. Suppose that RESCU is found in /home/software. On Linux you may add RESCU to your PATH typing

export PATH=/home/software:$PATH

On Windows, go to System\(>\)System Properties\(>\)Environment Variables. Select PATH and click on the Modify button. Then append :/home/software.