Modules as a convenient way of choosing build chain on macOS
Modules (Environment Modules or Lmod: A New Environment Module System) are the convenient way of setting up your – typically CLI based – environment.
You can efficiently switch back and forth between different tool chains. You can run different tools inside different terminal windows (you probably know the pain of setting up Java inside macOS – when you have more than one JDK installed).
Modules can be easily installed on macOS. All you have to do is following.
1. Building Modules
First of all, you have to get the source code, and install it.
> mkdir -p ~/opt/usr/local > mkdir -p ~/opt/src > cd ~/opt/src > curl -L -O https://netcologne.dl.sourceforge.net/project/modules/Modules/modules-4.4.1/modules-4.4.1.tar.gz > tar zxf modules-4.4.1.tar.gz > cd modules-4.4.1 > ./configure --prefix=$HOME/opt/usr/local/modules/modules-4.4.1 > make > make install
Modules are there. You can now test them
> module avail
---- .../opt/usr/local/modules/modules-4.4.1/modulefiles ----
dot module-git module-info modules null use.own
> module display dot
-------------------------------------------------------------
.../opt/usr/local/modules/modules-4.4.1/modulefiles/dot:
module-whatis {adds `.' to your PATH environment variable}
append-path PATH .
-------------------------------------------------------------
> env | grep PATH
PATH=...:/opt/X11/bin
> module load dot
> env | grep PATH
PATH=...:/opt/X11/bin:.
As you can see, this one is supper simple. However, they can serve you better.
2. Handling complex build chains
If you have tried to install gcc-8.3.0 and gcc-9.2.0 at the same time, you probably know the pain of switching back and forth. With modules, it’s very convenient.
Let’s say you have following layout of toolchains
$HOME/opt/usr/local
|-- OpenCoarrays
| |-- OpenCoarrays-2.8.0-mpich-3.3.1-gcc-8.3.0
| |-- OpenCoarrays-2.8.0-mpich-3.3.1-gcc-9.2.0
| |-- OpenCoarrays-2.8.0-openmpi-4.0.2-gcc-8.3.0
| `-- OpenCoarrays-2.8.0-openmpi-4.0.2-gcc-9.2.0
|-- cmake
| `-- cmake-3.16.1
|-- etc
| `-- modulefiles
|-- gcc
| |-- gcc-8.3.0
| `-- gcc-9.2.0
|-- gcc_system_root
| |-- Library -> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library
| |-- System -> /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System
| `-- usr
|-- modules
| `-- modules-4.4.1
|-- mpich
| |-- mpich-3.3.1-gcc-8.3.0
| `-- mpich-3.3.1-gcc-9.2.0
`-- openmpi
|-- openmpi-4.0.2-gcc-8.3.0
`-- openmpi-4.0.2-gcc-9.2.0
You will end up with this kind of layout once you follow my steps related to installation of:
– gcc-8.3.0,
– gcc-9.2.0.
Anyways, let’s now try to simplify the process of switching back and forth between these toolchains.
3. Setting up modules
First of all, we need to prepare modules. We can do that quite easily by creating new directory inside $HOME/opt/usr/local/etc/modulefiles.
We have to create new directory: gnuchain. After this is done, your layout should look like this
$HOME/opt/usr/local
`-- etc
`-- modulefiles
`-- gnuchain
inside gnuchain directory we need file .module.
#%Module #### -*- mode: Tcl;-*-
##
## author: mko
##
proc get_gcc_version {} {
upvar 1 ModulesCurrentModulefile ModulesCurrentModulefile
set array [split $ModulesCurrentModulefile -]
set idx [lindex $array end]
return lindex $array $idx
}
proc get_version {} {
upvar 1 ModulesCurrentModulefile ModulesCurrentModulefile
return [file tail $ModulesCurrentModulefile]
}
proc get_mpi {} {
if {[string first "openmpi" [module-info version]] != -1} {
return "openmpi"
} else {
return "mpich"
}
}
module-whatis "gnuchain"
conflict gnuchain
set GCC_VERSION [get_gcc_version]
set FULL_VERSION [get_version]
set MPI_VERSION [get_mpi]
prepend-path PATH $env(HOME)/opt/usr/local/gcc/gcc-$GCC_VERSION/bin
prepend-path PATH $env(HOME)/opt/usr/local/$MPI_VERSION/$FULL_VERSION/bin
prepend-path PATH $env(HOME)/opt/usr/local/OpenCoarrays/OpenCoarrays-2.8.0-$FULL_VERSION/bin
this file is responsible for setting up environment such way we get access to all the tools that match our choice.
Once we have it, we need few more files. First of all we need .version. It will point to default version of our module.
#%Module################################ # Default version set ModulesVersion mpich-3.3.1-gcc-8.3.0
we also need few symbolic links. Each of symbolic links points to version we want to provide via the module system. In this case we need: mpich-3.3.1-gcc-8.3.0, mpich-3.3.1-gcc-9.2.0, openmpi-4.0.2-gcc-8.3.0, openmpi-4.0.2-gcc-9.2.0. These are symbolic links that are created following way:
> ln -s .module mpich-3.3.1-gcc-8.3.0 > ln -s .module mpich-3.3.1-gcc-9.2.0 > ln -s .module openmpi-4.0.2-gcc-8.3.0 > ln -s .module openmpi-4.0.2-gcc-9.2.0
Eventually, your gnuchain directory should look like this
gnuchain |-- .module |-- .version |-- mpich-3.3.1-gcc-8.3.0 |-- mpich-3.3.1-gcc-9.2.0 |-- openmpi-4.0.2-gcc-8.3.0 `-- openmpi-4.0.2-gcc-9.2.0
4. Making sure that modules are properly set
Once we have everything in place, we can add modules to ~/.zshrc.
... ... source $HOME/opt/usr/local/modules/modules-4.4.1/init/zsh module use $HOME/opt/usr/local/etc/modulefiles ... ...
5. Working with modules
Now, let’s test the modules. First of all, let’s take a look at what we have
> module avail ---- .../opt/usr/local/etc/modulefiles ---- gnuchain/mpich-3.3.1-gcc-8.3.0(default) gnuchain/mpich-3.3.1-gcc-9.2.0 gnuchain/openmpi-4.0.2-gcc-8.3.0 gnuchain/openmpi-4.0.2-gcc-9.2.0
We can load one of the modules
> module load gnuchain/mpich-3.3.1-gcc-8.3.0
> mpirun --version
HYDRA build details:
Version: 3.3.1
...
...
Now, let’s switch to other one
> module switch gnuchain/openmpi-4.0.2-gcc-8.3.0 > mpirun --version mpirun (Open MPI) 4.0.2 ... ...
and, eventually, let us try unloading module completely
> module unload gnuchain > mpirun -version zsh: command not found: mpirun
That’s it! :)