Programming Guide
The clusters at CHPC all mount the /uufs/chpc.utah.edu/sys application tree, so we have combined the documentation for applications and libraries into this one guide. Unique attributes of the various clusters will be pointed out throughout this programming guide.
Contents
Available Compilers
C/C++
CHPC offers several compilers for use on the clusters. The GNU Compiler Suite includes the ANSI C, C++ compilers. Its installed version on the RedHat EL 6 OS is 4.4.7. Newer versions of the GNU compilers are also maintained (currently we have /uufs/chpc.utah.edu/sys/pkg/gcc/4.7.2_rh6).
In addition to the GNU compilers, we offer two commercial compiler suites from Intel and the Portland Group (PGI).
The Intel compilers generally provide superior performance. They include C, C++ and Fortran.
The Portland Group Compiler Suite is another good compiler distribution which has compilers for the aforementioned languages.
GNU compilers
The GNU compilers are located in the system default area, i.e. compilers in /usr/bin, libraries in /usr/lib or /usr/lib64, header files in /usr/include, etc. The C compiler is called gcc; the C++ compiler g++. Below is a very basic example of how to generate an executable (exe) from a C source file (source.c) or a C++ source file (source.cc) :
gcc -o exe source.c
g++ -o exe source.cc
In order to use the version of the GNU compilers (presuming that you have the tcsh shell as your default shell) in /uufs/chpc.utah.edu/sys/pkg/gcc (currently 4.7.2):
setenv PATH "/uufs/chpc.utah.edu/sys/pkg/gcc/4.7.2_rh6/bin:$PATH"
gcc -o exe source.c
g++ -o exe source.cc
Intel compilers
The whole suite of the Intel compilers, debugger,etc are located in the directory:
/uufs/chpc.utah.edu/sys/pkg/intel/ics/bin
Before users can invoke any of these Intel compilers/debugger they have to source the file compilervars.sh (bash) or compilervars.csh (tcsh) to set the necessary environmental variables (note - this is done in the chpc provided .tchsrc and .bashrc files):
source /uufs/chpc.utah.edu/sys/pkg/intel/ics/bin/compilervars.chs intel64
The C compiler is invoked by the icc command; the C++ compiler by the icpc command. If you would like to see the list of available flags, please use the man pages by typing man icc (C), man icpc (C++).
To find the compiler's version, use the flag -v, i.e. type icc -v (C), icpc -v (C++).
Generally the flag -fast gives superior performance, though be careful since the optimization flags are fairly aggressive. If the code has numerical problems, try to compile with -O3 instead. Also, CPU architecture can be targeted with -xSSE4.2 for Ember and -xAVX for Kingspeak, by default it the target is the platform on which is the build done.
Consult the icc/icpc man page for more details.
For more information on the C/C++ compilers, visit Intel Parallel Studio XE 2015 page.
PGI compilers
The latest version of the Portland Group compilers for use on kingspeak are located in the directory:
/uufs/chpc.utah.edu/sys/installdir/pgi/std
In order to use the compiler, users have to source shell the script pgi.sh (bash)/pgi.csh (tcsh) that sets paths and some other environmental variables:
source /uufs/chpc.utah.edu/sys/installdir/pgi/std/etc/pgi.csh
The C compiler is invoked as pgcc; the C++ compiler as pgCC.
To find the compiler's version, use the flag -V, i.e. pgcc -V. For a list of all the available flags, use the man pages (e.g. man pgcc).
In order to obtain good performance we generally recommend the flag -fastsse.
For more information on the PGI C and C++ compilers and PGI Tools (Debugger and Profiler), please have a look at:
The Portland Group advises on its website the compiler options for a few well-known codes (ATLAS, FFTW, GotoBLAS,..)
Fortran
CHPC provides several Fortran compilers. The GNU Compiler Suite includes the Fortran compiler. The installed version on the RedHat EL 6 OS is 4.4.7 and newer versions can be found with the gcc install mentioned above.
In addition to the GNU Fortran compilers, we offer two commercial compiler suites which contain Fortran compilers: the Intel and the Portland Group Compiler (PGI) suites.
The Intel Fortran compiler generally provides superior performance.
The Portland Group Compiler Suite is another good compiler distribution.
GNU compilers
The GNU compiler gfortran is located in the default area, i.e. the compiler binary in /usr/bin, libraries in /usr/lib or /usr/lib64. The gfortran compiler supports all the options supported by the gcc compiler. In the next line we give a very basic example of how to compile an F90 source file (source.f90).
gfortran -o exe_f90 source.f90
In order to use the version of the GNU compilers in /uufs/chpc.utah.edu/sys/pkg/gcc:
setenv PATH "/uufs/chpc.utah.edu/sys/pkg/gcc/4.7.2_rh6/bin:$PATH"
setenv LD_LIBRARY_PATH "/uufs/chpc.utah.edu/sys/pkg/gcc/4.7.2_rh6/lib64:$LD_LIBRARY_PATH"
gfortran source.f90 -o exe f90
Intel compilers
The whole suite of the Intel compilers and debuggers are located in the directory:
/uufs/chpc.utah.edu/sys/pkg/intel/ics/bin
Before users can invoke any of these Intel compilers/debugger they have to source the file compilervars.sh (bash) or compilervars.csh (tcsh):
source /uufs/chpc.utah.edu/sys/pkg/intel/ics/bin/compilervars.csh intel64
The Fortran compiler is invoked by the ifort command. For a list of the available flags, please use the man pages by typing man ifort.
To find the compiler's version, use the flag -v, i.e. ifort -v.
We generally recommend the flag ' -xSSE4.2 -O3 ' for superior performance. Very aggressive optimization can be achieved using '-xSSE4.2 -fast'. Beware that the latter option should be handled with caution.
Consult the ifort man page for more details.
For more information on the Fortran compiler, visit the Intel Fortran Compilers page.
PGI compilers
The latest version of Portland Group compilers are located in the directory:
/uufs/chpc.utah.edu/sys/installdir/pgi/std
In order to use the compiler, users have to source shell script pgi.sh (bash)/pgi.csh (tcsh) found in the etc directory under the path given above, to defines paths and some other environment variables:
source /uufs/chpc.utah.edu/sys/installdir/pgi/std/etc/pgi.sh
The Fortran compilers are invoked as pgf77 (F77), pgf90 (F90), and pgf95 (F95).
To find the compiler version, use flag -V, e.g. pgf90 -V. For list of available flags, use the man pages (e.g. man pgf90).
In order to obtain good performance we generally recommend the flag -fastsse.
For more information on the Fortran compiler(s), PGI Fortran Reference and other PGI tools, please have a look at:
The Portland Group advises on its website the compiler options for a few well-known codes (ATLAS, FFTW, GotoBLAS,..)
Parallel application development
MPI - Message Passing Interface
As the CHPC clusters are distributed memory parallel systems, message passing is the most common way to communicate between the processes in the parallel program. The Message Passing Interface (MPI) is the prevalent communication system. There are many different MPI distributions and networks they run on. We support a number of MPI distributions which use is described in the MPI libraries page.
Debugging and Profiling
There are numerous debuggers available either as open source or commercial, and use of them is often a matter of personal preference.
The GNU debugger, gdb, is a text based debugger that is present on most Linux machines and allows for quick checking of the program or loading a core file of a crashed program to get an idea where the program crashed.
Then there are numerous graphical front ends to gdb, such as Data Display Debugger (ddd), or Eclipse CDT, which provide more user friendly interface to the text only gdb. We don't provide these tools mainly because of their limited support of parallel debugging.
The Portland Group Compiler Suite includes a graphical debugger called pgdbg which we have as a part of the Compiler Suite license. Provided the PGI environment
is set up, it can be launched by calling pgdbg.
Totalview is one of two commercial debuggers that support both serial and parallel debugging, including GPUs and Intel Xeon Phi. For details on how to use Totalview, please refer to the CHPC Totalview documentation and have a look at the CHPC Totalview presentation.
For serial profiling, there is GNU gprof and the Portland Group pgprof. Intel provides the VTune Amplifier XE to test the code performance for users developing serial and multi-threaded applications. Intel also provides the Intel Inspector which has been developed to find and fix memory errors and thread errors in serial or parallel code. For more information on the Intel performance tools, visit our Intel Parallel Studio XE page.
Libraries
Math libraries
There are numerous math libraries available, the most commonly used ones are described on the Math Libraries page.