Python Software Installation: mamba or conda with CHPC-Installed Miniforge

We recommend using the CHPC-installed Miniforge, which is available as an environment module, as a base for mamba and conda environments. This eliminates the need of installing the Miniforge by the user, and provides conda/mamba package manager which is then used to create environments in the user space.

Please note: While the environment management software described in this page has permissive licenses and can be used for free, channels (sources of packages) may have other license terms and may require commercial licenses. It is your responsibility to use an appropriate channel.

Miniforge module

To find what version of Miniforge3 we have installed, use the  module spider miniforge3 command. Pick the version that fits your needs (preferably the latest), and stick to it with the environments that you will be using. This will reduce the chances of dependency conflicts which result in broken environments.

module spider miniforge3
-----------------------------------------------------------------------------------------------------------------------------------------------------------------  
miniforge3:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
     Versions:
            miniforge3/24.9.0

Load the selected module, e.g. module load miniforge3/24.9.0

Do not put the module load miniforge3 command to get loaded automatically in  ~/.custom.csh (for tcsh shell) or ~/.custom.sh (for bash shell), since custom Python installations can break the remote connections using FastX.

Mamba Package Manager Basics

When installing software with mamba with centrally maintaned Miniforge, it has to be installed into a separate virtual environment. If you plan to use multiple packages that each require separate software dependencies, we  recommended to install each package into a separate virtual environment to minimize dependency conflicts.

To find common use cases outside of the scope of this documentation, take a look at the Conda cheat sheet. More detailed documentation is in the Mamba User Guide or Conda User Guide. Mamba is a drop in replacement for Conda, so Conda and Mamba syntax is the same.

Miniforge Virtual Environments

Virtual environments provide a way to install multiple packages with common dependencies. We can list existing environments with:

mamba env list

Assuming one uses a bash shell, we can, for example, install Python 3.11 in a separate environment:

mamba create -n py311 python=3.11

The mamba create command, in conjunction with the -n flag, creates a new environment named py311, which includes Python 3.11.

We can then activate the environment as:

conda activate py311

NOTE:  mamba activate/deactivate does not work, use conda activate/deactivate instead. Overall, use conda for activation/deactivation, and conda for everything else, especially installation, since the mamba dependency search is much faster.

All conda package commands can be used within the activated environment, meaning additional conda packages can be installed in this environment using the mamba install command.

To exit from the environment, run:

conda deactivate

To delete an environment, such as the py311 environment we created above, you would run:

conda env remove --name py311

Installing environments to alternative locations

By default, the environments are installed in user home directory, $HOME/.conda/envs. Since most home directories have 50 GB quota, this can fill up the home directory space quickly. To specify an alternative location for the environments, use the envs_dirs configuration modifier:

conda config --add envs_dirs /uufs/chpc.utah.edu/common/home/MY-group2/miniforge3_envs

chmod -R g+w /uufs/chpc.utah.edu/common/home/MY-group2/miniforge3_envs

Installing Additional Packages with Mamba

Additional packages can be installed in the mamba virtual environment. After following the instructions for setting up the virtual environment, you can list the currently installed Python modules as follows:

mamba list

To install a new package, run

mamba install [packagename]

For example, to install the latest version of the SciPy module, run

mamba install scipy

The installer will think for a little while and then install SciPy and a number of other packages on which SciPy depends, such as NumPy and the accelerated OpenBLAS library.

To install a specific version of scipy, run:

mamba install scipy=1.14.1

To uninstall a package run

mamba uninstall [packagename]

NOTE: Recent numpy/scipy ships with the OpenBLAS accelerated math library, but if you notice that the Intel Math Kernel Library (MKL) is installed instead, since the MKL library by defaults utilizes all the processor cores on the system, if you are planning to run many independent parallel Python calculations, set the environment variable OMP_NUM_THREADS=1 (setenv OMP_NUM_THREADS 1 for tcsh or export OMP_NUM_THREADS=1 for bash). The OpenBLAS has OMP_NUM_THREADS=1 as default. Thus, conversely, if using OpenBLAS, set the OMP_NUM_THREADS to the number of CPUs to use, e.g. export OMP_NUM_THREADS=$SLURM_NTASKS.

Conda Packages in other Channels

If the mamba installcommand cannot find the requested package, chances are that it will be in a non-default conda repository (also called a conda channel). Independent distributors can create their own package channels that house their software. The best approach to find a package that is not in an official conda channel is to do a websearch for it.

For example, to look for a package named Fasta, which is used for biological sequence alignment, we web search for "anaconda fasta". Top search hit suggests the following installation line:

mamba install -c biobuilds fasta

The "-c" option specifies the channel name, in our case biobuilds.

To add a channel to the default channel list, we can:

mamba config add channels biobuilds

However, this puts a channel at the top of the list, with the highest priority. We can add a new channel to the bottom of the list instead in the following way:

mamba config --append channels biobuilds

If you are using Anaconda or Miniconda, it is better to add the free conda-forge channel and remove the commercial defaults channel with the commands below. Although as of October 2024, Anaconda, Inc, which manages the defaults channel says that it is free for academic institutions, there have been some controversies around that.

conda config --add channels conda-forge
conda config --set channel_priority strict
conda config --remove channels defaults

To verify that the defaults channel is removed:

conda config --show channels
channels:
 - bioconda
  - conda-forge
- r

Installing Python Modules with Pip

When a Python package does not exist as a conda package, one can use Python's pip installer. With Miniforge, we recommend using pip only as a last resort because you lose the flexibility of the conda packaging environment, as conda works towards automatic conflict resolution between packages and version upgrade/downgrade).

To install a module using pip, we suggest you run:

pip install bibtexparser

Another install option includes:

python -m pip install bibtexparser

For additional methods of installing Python packages, see our older documentation.

Using Conda/Mamba Virtual Environments in Open OnDemand

Just as we can install software packages within mamba virtual environments, we can also install Jupyter notebook into a virtual environment to start up Jupyter notebook session. It is not recommended to run the Jupyter notebook command from the terminal, as it will require us to create an SSH tunnel to the machine where we run the terminal to access Jupyter in our client's web browser. The Open OnDemand Jupyter app simplifies this problem greatly by launching Jupyter directly in the client's browser.

We can begin this process by first following the mamba installation instructions above, then by creating a virtual environment and installing the software into it. In this example, we will install the software scipy into a Mamba virtual environment that we plan to use in a Jupyter notebook session.

module load miniforge3/24.9.0
mamba create -n my_venv jupyterlab scipy
conda activate my_venv
pip install jupyter notebook
conda deactivate

The module commands above load the newly created miniforge module into our environment. In the 'mamba create' command, we name our virtual environment 'my_venv' and install the jupyterlab and scipy software. We then activate the virtual environment with conda, use pip to install jupyter notebook tools into the environment, and finish by deactivating the environment.

To use the virtual environment in Open OnDemand's Jupyter app, we choose the "Custom (Environment setup below)" option for the "Jupyter Python version", and in the "Environment Setup for Custom Python" text box, put:

module load miniforge3/24.9.0
conda activate my_venv

Once your Jupyter session begins, you should now be able to successfully connect to a Jupyter notebook and access the commands within your virtual environment.

Miniforge Installation Examples

Interactive Machine Learning Environment with Tensorflow, Keras and Jupyter Lab

Tensorflow is a widely used deep learning framework. Keras is an add-on to the framework that attempts to make it more user friendly. Jupyter Lab allows to run Jupyter notebooks, e.g. in our Open Ondemand web portal.

As Tensorflow performs best on GPUs, we will be installing the GPU version. Look at the Tensorflow installation requirements to note the CUDA and CUDNN versions that the latest Tensorflow requires. As of this writing (September 2024), Tensorflow 2.5 requires CUDA 12.1 and CUDNN 8.9.

• Load the CUDA and CUDNN modules, and the newly installed Miniconda module (named tf25.lua)
  

  module load cuda/12.1.0 module load cudnn/8.9.7.29-12-gpu module load miniforge3/24.9.0

• Install Jupyter Lab in a virtual environment and activate the virtual environment.
  

  mamba create -n my_venv jupyterlab conda activate my_venv

• Install Tensorflow. Note that we are using pip, not conda, as Google provides its tensorflow builds in pip repositories and the conda repositories don't have all of the versions. This pip installed Tensorflow also includes Keras. Test that Tensorflow can access the GPU(s).
  

  pip install jupyter notebook pip install tensorflow python -c "from tensorflow.python.client import device_lib; print(device_lib.list_local_devices())" conda deactivate

• In the OpenOnDemand Jupyter Lab app launch window, put the following in the Environment Setup:
  

  module load cuda/11.2 module load cudnn/8.1.1 module load miniconda3/24.9.0 conda activate my_venv

  Note that in order to use GPU with Tensorflow, you need to request a GPU in the 'Advanced options' checkbox.