====== Spack v0.23 ====== Before you begin, please check if you really need to build your own software packages, since a lot of software is already installed on the system. For example, do you require the GCC, Open MPI, HDF5, Python, CMake or R? They are readily available. Especially when considering building your own MPI, [[https://spack.readthedocs.io/en/latest/getting_started.html#mpi|think twice]]. Your are here for the first time? Jump to overview. You need to remove Spack? Jump to the bottom, follow the steps and then jump back to install Spack. ===== Overview - main goals ===== - Install Spack - Access Spack anytime (aka sourcing - see below) - Change ''build_stage'' directory from ''/tmp'' to ''/scratch'' - Tell Spack what software is available on the cluster - Connecting two worlds (environment modules with Spack) - Basic Spack commands - A warning - Maintenance - Which version do I use? - Update Spack - Remove Spack ===== Install Spack ===== [[https://spack.io/|Spack]] is a tool that makes it easy for you to configure and maintain your own software packages. Here we give a brief overview of the least necessary steps to get you going, but we advise to follow the [[https://spack.readthedocs.io/en/latest/getting_started.html#getting-started|Getting Started]] and [[https://spack.readthedocs.io/en/latest/tutorial.html#tutorial-spack-101|Tutorial 101]] of Spack. All of the following commands can be found at the [[https://spack.readthedocs.io/en/latest/basic_usage.html#basic-usage|Basic Usage]] chapter of Spack. Run the following command on the command line (if you are reinstalling Spack, [[#remove Spack]] first): git clone -c feature.manyFiles=true https://github.com/spack/spack.git This will create a folder named ''spack'' in your home directory and clone the contents of the Git repository into that newly created ''spack'' folder. ===== Access Spack anytime ===== Now source the appropriate script for your shell. Most users on the cluster use the [[https://de.wikipedia.org/wiki/Bash_(Shell)|Bash]] shell. # Sourcing these files will put the ''spack'' command in your ''PATH'', # set up your ''MODULEPATH'' to use Spack-generated modules, and add other # useful shell integration for certain commands, environments and # modules. For ''bash'' and ''zsh'', it also sets up tab completion. # For bash/zsh/sh . ~/spack/share/spack/setup-env.sh # For tcsh/csh source ~/spack/share/spack/setup-env.csh ===== Change build_stage directory from /tmp to /scratch ===== To get started, [[goethe#login]] to the system and go to your home directory. As the very first step, **create a tempdir** for Spack builds (you will need it later): # This is really important. mkdir -p /scratch//spack/tmp We recommend installing Spack into your home directory. Use ''/scratch'' for temporary data. If you don't know, what ''/scratch'' is, please read [[public:usage:goethe#storage|here]] (or [[public:usage:fuchs#storage|here]] for FUCHS). By default Spack stores the temporary files under ''/tmp''. This is shared with all users on the cluster and should not be used for your Spack installation. **Spack files in ''/tmp'' will be deleted on a regular base within 30 minutes after creation**. In order to change the directory Spack uses for its temporary files you have to set the new directory in the ''config.yaml''. This can be done by the following commands. The following steps show you what needs to be changed: # For instance you can use the following command to alter config.yaml nano ~/spack/etc/spack/defaults/config.yaml ... (stuff above) ... build_stage: - $tempdir/$user/spack-stage - $user_cache_path/stage # - $spack/var/spack/stage ... (stuff below) ... to ... (stuff above) ... build_stage: # This path must exist. Please create it if it's not already done. # For instance go to the folder /scratch///spack/tmp # and type ''pwd'' and copy it to # another terminal where you altering the config.yaml to avoid spelling errors. - /scratch///spack/tmp ...(stuff below)... ===== Tell Spack what software is available on the cluster ===== By default, Spack does not care about software that is already available on the system. You can tell Spack to locate it (please note that not everything can be found using this method, you may need to instruct Spack explicitly on what to find, as illustrated in Example 2). **Example 1**: # # This will create .spack/packages.yaml [@login01 ~]$ spack external find && spack external find slurm ==> The following specs have been detected on this system and added to /home//.spack/packages.yaml autoconf@2.69 bison@3.7.4 curl@7.76.1 flex@2.6.4 gmake@4.3 m4@1.4.19 pkgconf@1.7.3 automake@1.16.2 cmake@3.20.2 diffutils@3.7 gawk@5.1.0 groff@1.22.4 openssh@8.7p1 tar@1.34 binutils@2.35.2 coreutils@8.32 findutils@4.8.0 git@2.39.3 libtool@2.4.6 openssl@3.0.7 texinfo@6.7 ==> The following specs have been detected on this system and added to /home//.spack/packages.yaml slurm@23.02.6 [@login01 ~]$ spack compiler find ==> Added 1 new compiler to /home//.spack/linux/compilers.yaml gcc@11.3.1 ==> Compilers are defined in the following files: /home//.spack/linux/compilers.yaml **Example 2**: [@login01 ~]$ module load mpi/openmpi/5.0.0 [@login01 ~]$ spack external find openmpi ==> The following specs have been detected on this system and added to /home//.spack/packages.yaml -- no arch / gcc@11.3.1 ----------------------------------------- openmpi@5.0.0 [@login01 ~]$ spack install netcdf-c Spack will use the external packages when building new software instead of building all dependencies from scratch. On the other hand, it is not always desirable to use an external package. Pre-installed software (e.g. Perl) might lack needed features and using it as a dependency will fail. Then it's better to let Spack build its own version. ===== Connecting two worlds ===== This section can be skipped if your are in a hurry, it just provides some convenience. It is important to distinguish between the cluster modules (see [[public:usage:goethe#Environment modules]]), your Spack packages, your private software and your private modules. You can install, list and load Spack packages using the ''spack'' command. Another way to use Spack packages are modules. If you don't need them or prefer not to use them, **you can also skip this section**. The cluster modules can be displayed with the ''module avail'' command, which looks like this: [@login02 ~]$ module avail --------------------- /cluster/modulefiles --------------------- intel/oneapi/2023.2.0 nvidia/hpc_sdk/nvhpc-byo-compiler/23.9 intel/oneapi/latest nvidia/hpc_sdk/nvhpc-hpcx-cuda12/23.9 mpi/openmpi/4.1.6 nvidia/hpc_sdk/nvhpc-hpcx/23.9 mpi/openmpi/5.0.0 nvidia/hpc_sdk/nvhpc-nompi/23.9 nvidia/cuda/12.3.0 nvidia/hpc_sdk/nvhpc/23.9 ---------------- /usr/share/Modules/modulefiles ---------------- dot module-git module-info modules null use.own -------------------- /usr/share/modulefiles -------------------- mpi/openmpi-x86_64 Key: modulepath Your Spack packages can be displayed with ''spack find''. Now imagine you could combine ''module avail'' with your Spack packages. The following example illustrate how to: # In this example Python version 3.11.6 will be installed spack info python # Shows all available versions spack install python@3.11.6 # Preferred version at the time of writing this ...after a while... ==> python: Successfully installed python-3.11.6-2zpso6kw573oyfw5cdqt5lq5sbsazv73 Stage: 2.49s. Configure: 44.94s. Build: 21.50s. Install: 16.89s. Post-install: 22.83s. Total: 1m 49.07s [+] /home//spack/opt/spack/linux-almalinux9-skylake_avx512/gcc-11.3.1/python-3.11.6-2zpso6kw573oyfw5cdqt5lq5sbsazv73 # Here you can learn some interesting details. # Spack knows that we have AlmaLinux on a Skylake compute node with AVX-512 instruction set, # also Spack uses our default compiler gcc-11.3.1. # It follows the desired python-3.11.6 version with a unique identifier # Why is this pointed out here? Please see section "A warning". # Show me what I already installed with Spack: [@login02 ~]$ spack find -- linux-almalinux9-skylake_avx512 / gcc@11.3.1 ----------------- berkeley-db@18.1.40 expat@2.5.0 libbsd@0.11.7 libxcrypt@4.4.35 perl@5.38.0 sqlite@3.43.2 zlib-ng@2.1.4 bzip2@1.0.8 gdbm@1.23 libffi@3.4.4 libxml2@2.10.3 pkgconf@1.7.3 tar@1.34 ca-certificates-mozilla@2023-05-30 gettext@0.22.3 libiconv@1.17 ncurses@6.4 python@3.11.6 util-linux-uuid@2.38.1 diffutils@3.7 gmake@4.3 libmd@1.0.4 openssl@3.1.3 readline@8.2 xz@5.4.1 ==> 25 installed packages # Refresh my modules to integrate them into the 'module avail' command. [@login02 ~]$ spack module tcl refresh ==> You are about to regenerate tcl module files for: -- linux-almalinux9-skylake_avx512 / gcc@11.3.1 ----------------- gndqrbp berkeley-db@18.1.40 d35sdln gdbm@1.23 uffpwpo libiconv@1.17 36lhb6f openssl@3.1.3 gsfue6h sqlite@3.43.2 plgyf6q bzip2@1.0.8 oghfkbk gettext@0.22.3 53e2urm libmd@1.0.4 vthh2zs perl@5.38.0 7f6hcs7 tar@1.34 yuvg6t3 ca-certificates-mozilla@2023-05-30 krzlwmr gmake@4.3 hza4ixw libxcrypt@4.4.35 e4dbf5o pkgconf@1.7.3 l5vb4dw util-linux-uuid@2.38.1 o4iuzop diffutils@3.7 siw2hgu libbsd@0.11.7 xzkdhcp libxml2@2.10.3 2zpso6k python@3.11.6 o2okcsd xz@5.4.1 q5lazqy expat@2.5.0 ni4xv7l libffi@3.4.4 7cld3mz ncurses@6.4 46kn2s6 readline@8.2 nmtn65b zlib-ng@2.1.4 ==> Do you want to proceed? [y/n] y # Important, refresh your spack environment or logout and login again . spack/share/spack/setup-env.sh # Finally type module avail [@login02 ~]$ module avail ------------------------------------------ /home//spack/share/spack/modules/linux-almalinux9-skylake_avx512 ------------------------------------------ python/3.11.6-gcc-11.3.1-2zpso6k ---------------------------------------------------------------------- /cluster/modulefiles ----------------------------------------------------------------------- intel/oneapi/2023.2.0 mpi/openmpi/4.1.6 nvidia/cuda/12.3.0 nvidia/hpc_sdk/nvhpc-hpcx-cuda12/23.9 nvidia/hpc_sdk/nvhpc-nompi/23.9 intel/oneapi/latest mpi/openmpi/5.0.0 nvidia/hpc_sdk/nvhpc-byo-compiler/23.9 nvidia/hpc_sdk/nvhpc-hpcx/23.9 nvidia/hpc_sdk/nvhpc/23.9 ----------------------------------------------------------------- /usr/share/Modules/modulefiles ------------------------------------------------------------------ dot module-git module-info modules null use.own --------------------------------------------------------------------- /usr/share/modulefiles ---------------------------------------------------------------------- mpi/openmpi-x86_64 Key: modulepath Every time you install a new Spack module it is necessary to redo these two steps: spack module tcl refresh . spack/share/spack/setup-env.sh As mentioned, this is only an example that can be set up quickly. If you wish further adjustments of your own modules, for example different name schemes, please look here: [[https://spack.readthedocs.io/en/latest/module_file_support.html|Spack modules]] ===== Basic Spack commands ===== With Spack it is easy to install the software needed for you project. But before you try install it, you should make sure Spack has the packages you need and with the correct version. To list all the available packages use spack list To check if Spack has a package you need you can either search through the list or use spack list If you need to know if Spack has a specific version of your package, use spack versions For further information on a package, e.g. dependencies, use spack info Finally you can install your package with the latest version with spack install If you need an older version you can use the "@" symbol appended to the package name and specify a version listed by "spack versions " spack install @version If you also need it with a specific compiler, append spack install @version % For further details visit [[https://spack.readthedocs.io/en/latest/basic_usage.html#specs-dependencies|Specs and Dependencies]] ===== Using packages ===== You can use your installed packages in different ways. In many cases it is enough to know where they are installed so you could run a binary. To get the installation location of a package you can use the following command. spack location --install-dir One way to use a package is to load it: spack load Although a simple load is very useful in some cases it comes with some drawbacks, if there are more then one version of the package you want to load the load will fail, unless you specify exactly which of the packages you want to use. ===== Managing software packages ===== When you install multiple software packages, Spack uses all the already installed packages to install the new one. To see all your installed software packages with their version use (**This will show all the software packages with their installed dependency packages**) spack find If you only want to see your directly installed packages (installed with ''spack install '') use spack find --explicit List loaded packages: spack find --loaded Unload packages: spack unload ===== Uninstalling software ===== Uninstalling software packages is as easy as installing them. First you should [[#Managing Software Packages|find the package]] you want to uninstall. Then uninstall the **package name with the version** by using spack uninstall @version If any other packages depend on it, Spack will print an error and list all the other packages that still depend on your package. You can check the dependencies of a package yourself by spack find --deps gcc ===== Adding new compilers ===== Spack has the ability to build packages with multiple compilers and compiler versions. To see which compilers Spack has found on your system run spack compilers Any of these compilers can be used to build Spack packages. If you want to use a compiler which is not in this list you have to add it to your PATH first in order for Spack to find it. You can add a new Compiler to your PATH by loading the module of the compiler. To see which modules are available run module avail Then you can load the module of the compiler you want to add to Spack: module load When you have loaded a compiler module, run spack compiler find to add the compiler to Spack. ++++Adding the Intel compiler (deprecated)| ===== Adding the Intel compiler ===== On the GOETHE HLR you can use the latest Intel compiler, you can load it with module load comp/intel/2020.0 To add the compiler to spack, just type spack compiler find Which should add the intel compiler to your spack compilers. Finally you have to edit the "compilers.yaml" file located at "~/.spack/linux/" by adding the two rpaths to the added Intel compiler. This example shows the configuration for "intel@19.1.0.166" which was previously added to spack. If you intend to use a different version you need to change the paths, too. ... extra_rpaths: - /cluster/intel/2020.0/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64 - /cluster/intel/2020.0/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/ ... So your "compilers.yaml" should have the entry - compiler: spec: intel@19.1.0.166 paths: cc: /cluster/intel/2020.0/compilers_and_libraries_2020.0.166/linux/bin/intel64/icc cxx: /cluster/intel/2020.0/compilers_and_libraries_2020.0.166/linux/bin/intel64/icpc f77: /cluster/intel/2020.0/compilers_and_libraries_2020.0.166/linux/bin/intel64/ifort fc: /cluster/intel/2020.0/compilers_and_libraries_2020.0.166/linux/bin/intel64/ifort flags: {} operating_system: scientific7 target: x86_64 modules: [] environment: {} extra_rpaths: - /cluster/intel/2020.0/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64 - /cluster/intel/2020.0/compilers_and_libraries_2020.0.166/linux/compiler/lib/intel64_lin/ ++++ ===== A warning ===== Spack packages work on the machines on which they were built, but may not function on a different machine type((except a cross-compiler setting was used)). If you need software packages for different nodes, e.g. GPU units, you need to compile them there. To accomplish this task we recommend to use SLURM's ''salloc'' command to work interactively on the desired node type. Please see here for further assistance: [[public:usage:goethe#the_salloc_command|The salloc Command]] (or [[public:usage:fuchs#the_salloc_command|here]] for FUCHS). ===== Maintenance ===== ==== Which version do I use? ==== # Spack does not update itself. After a while you might see only old versions with 'spack info '. # Use 'spack -V' to check your version. # Example: [user@login01 ~]$ spack -V 0.23.0.dev0 ==== Update Spack ==== In some cases you do not find with ''spack info '' what you are looking for. Then you might want to update Spack. Here are some steps how to do so. spack -V # Your own version of Spack 1. cd ~/spack # Change into your Spack directory 2. git pull # If an error appears see next box 3. cd .. # Go one folder back 4. . spack/share/spack/setup-env.sh # Update your environment ++++Some hints to errors due the update prozess| Following error appeared error: Your local changes to the following files would be overwritten by merge: etc/spack/defaults/config.yaml Please, commit your changes or stash them before you can merge. Aborting # This error appears because you have changed the tmp-directory to # /scratch///spack/tmp, # as we advised (see section install), within the config.yaml. during the ''git pull'' process. Here you can find a whole discussion about it [[https://stackoverflow.com/questions/15745045/how-do-i-resolve-git-saying-commit-your-changes-or-stash-them-before-you-can-me|Link]] and how to solve it. 1. cd ~/spack # Previous step like above 2. git pull # Error appears 3. git stash save "" # New step to deal with the error 4. git pull # Redo it again 5. cd ~/spack/etc/spack/defaults/ 6. Edit your 'config.yaml' # See at the top how to alter Now you should be able so see the latest packages which Spack provides. ++++ ==== Remove Spack ==== If something went wrong and you want to start from the scratch or just remove Spack, please delete both following directories. # Think twice before doing this! You need to remove two directories. rm -rf /your-home-directory/spack rm -rf /your-home-directory/.spack