Table of Contents
Spack v0.23
Overview - main goals
- Install Spack
- Access Spack anytime (aka sourcing - see below)
- Change
build_stage
directory from/tmp
to/scratch
- Tell Spack what software is available on the cluster
- Connecting two worlds (environment modules with Spack)
- Basic Spack commands
- A warning
- Maintenance
- Which version do I use?
- Update Spack
- Remove Spack
Install Spack
Spack is a tool that makes it easy for you to configure and maintain your own software packages. Here we give a brief overview of the least necessary steps to get you going, but we advise to follow the Getting Started and Tutorial 101 of Spack. All of the following commands can be found at the Basic Usage chapter of Spack.
Run the following command on the command line (if you are reinstalling Spack, remove Spack first):
Download Spack
git clone -c feature.manyFiles=true https://github.com/spack/spack.git
This will create a folder named spack
in your home directory and clone the contents of the Git repository into that newly created spack
folder.
Access Spack anytime
Now source the appropriate script for your shell. Most users on the cluster use the Bash shell.
Sourcing
# Sourcing these files will put the ''spack'' command in your ''PATH'', # set up your ''MODULEPATH'' to use Spack-generated modules, and add other # useful shell integration for certain commands, environments and # modules. For ''bash'' and ''zsh'', it also sets up tab completion. # For bash/zsh/sh . ~/spack/share/spack/setup-env.sh # For tcsh/csh source ~/spack/share/spack/setup-env.csh
Change build_stage directory from /tmp to /scratch
To get started, login to the system and go to your home directory. As the very first step, create a tempdir for Spack builds (you will need it later):
Create your temp directory on /scratch
# This is really important. mkdir -p /scratch/<path to your scratch folder>/spack/tmp
/scratch
for temporary data. If you don't know, what /scratch
is, please read here (or here for FUCHS).
/tmp
. This is shared with all users on the cluster and should not be used for your Spack installation. Spack files in /tmp
will be deleted on a regular base within 30 minutes after creation. In order to change the directory Spack uses for its temporary files you have to set the new directory in the config.yaml
. This can be done by the following commands.
The following steps show you what needs to be changed:
Access config.yaml
# For instance you can use the following command to alter config.yaml nano ~/spack/etc/spack/defaults/config.yaml
~/spack/etc/spack/defaults/config.yaml (original file)
... (stuff above) ... build_stage: - $tempdir/$user/spack-stage - $user_cache_path/stage # - $spack/var/spack/stage ... (stuff below) ...
to
~/spack/etc/spack/defaults/config.yaml (new file)
... (stuff above) ... build_stage: # This path must exist. Please create it if it's not already done. # For instance go to the folder /scratch/<your-project-name>/<your-user-name>/spack/tmp # and type ''pwd'' and copy it to # another terminal where you altering the config.yaml to avoid spelling errors. - /scratch/<your-project-name>/<your-user-name>/spack/tmp ...(stuff below)...
Tell Spack what software is available on the cluster
By default, Spack does not care about software that is already available on the system. You can tell Spack to locate it (please note that not everything can be found using this method, you may need to instruct Spack explicitly on what to find, as illustrated in Example 2).
Example 1:
Find externals and compilers
# # This will create .spack/packages.yaml [<user>@login01 ~]$ spack external find && spack external find slurm ==> The following specs have been detected on this system and added to /home/<user>/.spack/packages.yaml autoconf@2.69 bison@3.7.4 curl@7.76.1 flex@2.6.4 gmake@4.3 m4@1.4.19 pkgconf@1.7.3 automake@1.16.2 cmake@3.20.2 diffutils@3.7 gawk@5.1.0 groff@1.22.4 openssh@8.7p1 tar@1.34 binutils@2.35.2 coreutils@8.32 findutils@4.8.0 git@2.39.3 libtool@2.4.6 openssl@3.0.7 texinfo@6.7 ==> The following specs have been detected on this system and added to /home/<user>/.spack/packages.yaml slurm@23.02.6 [<user>@login01 ~]$ spack compiler find ==> Added 1 new compiler to /home/<user>/.spack/linux/compilers.yaml gcc@11.3.1 ==> Compilers are defined in the following files: /home/<user>/.spack/linux/compilers.yaml
Example 2:
Add an already available Open MPI and build NetCDF against it
[<user>@login01 ~]$ module load mpi/openmpi/5.0.0 [<user>@login01 ~]$ spack external find openmpi ==> The following specs have been detected on this system and added to /home/<user>/.spack/packages.yaml -- no arch / gcc@11.3.1 ----------------------------------------- openmpi@5.0.0 [<user>@login01 ~]$ spack install netcdf-c
Spack will use the external packages when building new software instead of building all dependencies from scratch. On the other hand, it is not always desirable to use an external package. Pre-installed software (e.g. Perl) might lack needed features and using it as a dependency will fail. Then it's better to let Spack build its own version.
Connecting two worlds
It is important to distinguish between the cluster modules (see Environment modules), your Spack packages, your private software and your private modules. You can install, list and load Spack packages using the spack
command. Another way to use Spack packages are modules. If you don't need them or prefer not to use them, you can also skip this section. The cluster modules can be displayed with the module avail
command, which looks like this:
Available cluster modules
[<user>@login02 ~]$ module avail --------------------- /cluster/modulefiles --------------------- intel/oneapi/2023.2.0 nvidia/hpc_sdk/nvhpc-byo-compiler/23.9 intel/oneapi/latest nvidia/hpc_sdk/nvhpc-hpcx-cuda12/23.9 mpi/openmpi/4.1.6 nvidia/hpc_sdk/nvhpc-hpcx/23.9 mpi/openmpi/5.0.0 nvidia/hpc_sdk/nvhpc-nompi/23.9 nvidia/cuda/12.3.0 nvidia/hpc_sdk/nvhpc/23.9 ---------------- /usr/share/Modules/modulefiles ---------------- dot module-git module-info modules null use.own -------------------- /usr/share/modulefiles -------------------- mpi/openmpi-x86_64 Key: modulepath
Your Spack packages can be displayed with spack find
. Now imagine you could combine module avail
with your Spack packages. The following example illustrate how to:
Connecting two worlds
# In this example Python version 3.11.6 will be installed spack info python # Shows all available versions spack install python@3.11.6 # Preferred version at the time of writing this ...after a while... ==> python: Successfully installed python-3.11.6-2zpso6kw573oyfw5cdqt5lq5sbsazv73 Stage: 2.49s. Configure: 44.94s. Build: 21.50s. Install: 16.89s. Post-install: 22.83s. Total: 1m 49.07s [+] /home/<user>/spack/opt/spack/linux-almalinux9-skylake_avx512/gcc-11.3.1/python-3.11.6-2zpso6kw573oyfw5cdqt5lq5sbsazv73 # Here you can learn some interesting details. # Spack knows that we have AlmaLinux on a Skylake compute node with AVX-512 instruction set, # also Spack uses our default compiler gcc-11.3.1. # It follows the desired python-3.11.6 version with a unique identifier # Why is this pointed out here? Please see section "A warning". # Show me what I already installed with Spack: [<user>@login02 ~]$ spack find -- linux-almalinux9-skylake_avx512 / gcc@11.3.1 ----------------- berkeley-db@18.1.40 expat@2.5.0 libbsd@0.11.7 libxcrypt@4.4.35 perl@5.38.0 sqlite@3.43.2 zlib-ng@2.1.4 bzip2@1.0.8 gdbm@1.23 libffi@3.4.4 libxml2@2.10.3 pkgconf@1.7.3 tar@1.34 ca-certificates-mozilla@2023-05-30 gettext@0.22.3 libiconv@1.17 ncurses@6.4 python@3.11.6 util-linux-uuid@2.38.1 diffutils@3.7 gmake@4.3 libmd@1.0.4 openssl@3.1.3 readline@8.2 xz@5.4.1 ==> 25 installed packages # Refresh my modules to integrate them into the 'module avail' command. [<user>@login02 ~]$ spack module tcl refresh ==> You are about to regenerate tcl module files for: -- linux-almalinux9-skylake_avx512 / gcc@11.3.1 ----------------- gndqrbp berkeley-db@18.1.40 d35sdln gdbm@1.23 uffpwpo libiconv@1.17 36lhb6f openssl@3.1.3 gsfue6h sqlite@3.43.2 plgyf6q bzip2@1.0.8 oghfkbk gettext@0.22.3 53e2urm libmd@1.0.4 vthh2zs perl@5.38.0 7f6hcs7 tar@1.34 yuvg6t3 ca-certificates-mozilla@2023-05-30 krzlwmr gmake@4.3 hza4ixw libxcrypt@4.4.35 e4dbf5o pkgconf@1.7.3 l5vb4dw util-linux-uuid@2.38.1 o4iuzop diffutils@3.7 siw2hgu libbsd@0.11.7 xzkdhcp libxml2@2.10.3 2zpso6k python@3.11.6 o2okcsd xz@5.4.1 q5lazqy expat@2.5.0 ni4xv7l libffi@3.4.4 7cld3mz ncurses@6.4 46kn2s6 readline@8.2 nmtn65b zlib-ng@2.1.4 ==> Do you want to proceed? [y/n] y # Important, refresh your spack environment or logout and login again . spack/share/spack/setup-env.sh # Finally type module avail [<user>@login02 ~]$ module avail ------------------------------------------ /home/<user>/spack/share/spack/modules/linux-almalinux9-skylake_avx512 ------------------------------------------ python/3.11.6-gcc-11.3.1-2zpso6k ---------------------------------------------------------------------- /cluster/modulefiles ----------------------------------------------------------------------- intel/oneapi/2023.2.0 mpi/openmpi/4.1.6 nvidia/cuda/12.3.0 nvidia/hpc_sdk/nvhpc-hpcx-cuda12/23.9 nvidia/hpc_sdk/nvhpc-nompi/23.9 intel/oneapi/latest mpi/openmpi/5.0.0 nvidia/hpc_sdk/nvhpc-byo-compiler/23.9 nvidia/hpc_sdk/nvhpc-hpcx/23.9 nvidia/hpc_sdk/nvhpc/23.9 ----------------------------------------------------------------- /usr/share/Modules/modulefiles ------------------------------------------------------------------ dot module-git module-info modules null use.own --------------------------------------------------------------------- /usr/share/modulefiles ---------------------------------------------------------------------- mpi/openmpi-x86_64 Key: modulepath
spack module tcl refresh . spack/share/spack/setup-env.sh
As mentioned, this is only an example that can be set up quickly. If you wish further adjustments of your own modules, for example different name schemes, please look here: Spack modules
Basic Spack commands
With Spack it is easy to install the software needed for you project. But before you try install it, you should make sure Spack has the packages you need and with the correct version.
To list all the available packages use
spack list
To check if Spack has a package you need you can either search through the list or use
spack list <my-package-name>
If you need to know if Spack has a specific version of your package, use
spack versions <package-name>
For further information on a package, e.g. dependencies, use
spack info <package-name>
Finally you can install your package with the latest version with
spack install <package-name>
If you need an older version you can use the “@” symbol appended to the package name and specify a version listed by “spack versions <package-name>”
spack install <package-name>@version
If you also need it with a specific compiler, append
spack install <package-name>@version %<compiler-name>
For further details visit Specs and Dependencies
Using packages
You can use your installed packages in different ways. In many cases it is enough to know where they are installed so you could run a binary. To get the installation location of a package you can use the following command.
spack location --install-dir <your package>
One way to use a package is to load it:
spack load <your package>
Although a simple load is very useful in some cases it comes with some drawbacks, if there are more then one version of the package you want to load the load will fail, unless you specify exactly which of the packages you want to use.
Managing software packages
When you install multiple software packages, Spack uses all the already installed packages to install the new one.
To see all your installed software packages with their version use
(This will show all the software packages with their installed dependency packages)
spack find
If you only want to see your directly installed packages (installed with spack install <package-name>
) use
spack find --explicit
List loaded packages:
spack find --loaded
Uninstalling software
Uninstalling software packages is as easy as installing them.
First you should find the package you want to uninstall.
Then uninstall the package name with the version by using
spack uninstall <package-name>@version
If any other packages depend on it, Spack will print an error and list all the other packages that still depend on your package.
You can check the dependencies of a package yourself by
spack find --deps gcc <package-name>
Adding new compilers
Spack has the ability to build packages with multiple compilers and compiler versions. To see which compilers Spack has found on your system run
spack compilers
Any of these compilers can be used to build Spack packages. If you want to use a compiler which is not in this list you have to add it to your PATH first in order for Spack to find it. You can add a new Compiler to your PATH by loading the module of the compiler. To see which modules are available run
module avail
Then you can load the module of the compiler you want to add to Spack:
module load
When you have loaded a compiler module, run
spack compiler find
to add the compiler to Spack.
A warning
Spack packages work on the machines on which they were built, but may not function on a different machine type1). If you need software packages for different nodes, e.g. GPU units, you need to compile them there. To accomplish this task we recommend to use SLURM's salloc
command to work interactively on the desired node type. Please see here for further assistance: The salloc Command (or here for FUCHS).
Maintenance
Which version do I use?
Spack version - Update required?
# Spack does not update itself. After a while you might see only old versions with 'spack info <package>'. # Use 'spack -V' to check your version. # Example: [user@login01 ~]$ spack -V 0.23.0.dev0
Update Spack
In some cases you do not find with spack info <package>
what you are looking for. Then you might want to update Spack.
Here are some steps how to do so.
Update Spack
spack -V # Your own version of Spack 1. cd ~/spack # Change into your Spack directory 2. git pull # If an error appears see next box 3. cd .. # Go one folder back 4. . spack/share/spack/setup-env.sh # Update your environment
Remove Spack
If something went wrong and you want to start from the scratch or just remove Spack, please delete both following directories.
Remove Spack
# Think twice before doing this! You need to remove two directories. rm -rf /your-home-directory/spack rm -rf /your-home-directory/.spack