Charliecloud

Charliecloud setup

Charliecloud is installed by CHPC and available by loading its module, module load charliecloud. It provides a completely user space alternative to Docker, as it can pull Docker images from repositories like Dockerhub, and can build containers based on Dockerfiles.

Note that by default Charliecloud stores the container images in/var/tmp, which has very limited size. Therefore we recommend to set the environment variable CH_IMAGE_STORAGE pointing to a file system with sufficient storage size (e.g. group space).

Running Docker containers with Charliecloud

Docker containers stored on repositories can be pulled as follows:

module load charliecloud
ch-image pull almalinux:8
ch-convert almalinux:8 almalinux.sqfs
ch-run almalinux.sqfs -- /bin/bash

The first command pulls the container image into the CH_IMAGE_STORAGE  directory. The second command creates a single file archive of this container. Third command opens a shell in the container.

Unlike Apptainer/Singularity, by default Charliecloud does not bind-mount the home directories and the system dependent directories like /scratch or /uufs. To have these directories available in the container, first create the /scratch and /uufs in the container image (it can be done by a simple mkdir on the container image path, in $CH_IMAGE_STORAGE/img/container_name, run on the host). Then run with the --bind and --home options:

ch-run --home --bind=/uufs --bind=/scratch almalinux:8 -- /bin/bash

Note that the --home makes the home directory available in /home/$USER, though the --bind=/uufs also results in the home directory being available in the CHPC standard location, /uufs/chpc.utah.edu/common/home/$USER.

Creating Docker Containers with Charliecloud

We can use the ch-build command to build container called hello based on Dockerfile in certain directory as:

ch-image build -t hello -f Dockerfile .

To query what containers are available in the local cache:

ch-image list

For details on the Charliecloud commands, see its documentation. For basic functionality, see the Tutorial. Examples can be found in $CHARLIECLOUD_ROOT/share/doc/charliecloud/examples/.

Charliecloud and MPI

Charliecloud supports parallel MPI programs execution, however, one needs to be careful to build the MPIs in the container in such a way that they include process managers that are compatible with our SLURM launcher. As of early 2024, we have noticed that PMIX is not. Following Charliecloud's examples, we need to remove the PMIX from either the Dockerfile.mpich (--with-pmix=/usr/local/lib) or Dockerfile.openmpi (--with-pmix=/usr/local \).

Then we build the containers, e.g. for OpenMPI (the Charliecloud examples as of this writing use OpenMPI 4.1.4). Note that the CH_IMAGE_STORAGE cache is in user home directory, which is not ideal due to the 50 GB storage quota.:

ml charliecloud
setenv CH_IMAGE_STORAGE /uufs/chpc.utah.edu/common/home/$USER/charliecloud/cache
ch-image build -f Dockerfile.almalinux_8ch .
ch-image build -f Dockerfile.libfabric .
ch-image build -f Dockerfile.openmpi .
mkdir /uufs/chpc.utah.edu/common/home/$USER/charliecloud/cache/openmpi/scratch

The last command creates a bind mount for the /scratch directory. The SLURM jobs define TMPDIR in /scratch/local, which OpenMPI's mpirun automatically picks. If we would not do this, the mpirun would complain about non existent /scratch directory.

Now we can start an interactive job, shell into the container to build the MPI executable, exit out of the container, and run MPI job using the binary built by the container.

salloc -N 2 -n 2 -A notchpeak-shared-short -p notchpeak-shared-short -t 2:00:00
ml charliecloud
ch-run --home openmpi_nopmi -- /bin/bash
cd /home/$USER/tests
mpicc latbw.c -o latbw_openmpi
exit 
ml gcc/8.5.0 openmpi/4.1.4
mpirun -np 2 ch-run -b /scratch --home openmpi -- /home/$USER/tests/latbw_openmpi

We should see the shortest latencies ~1.5 us which indicates that the MPI is using InfiniBand.

On Redwood, while it's still on the CentOS 7 OS, use the following modules: gcc/8.3.0 charliecloud .