Running MPI Jobs on Lewis
Compilation
The following commands are preferred for compiling MPI programs and linking them with the libraries that support the InifiniBand high-speed interconnect:
mpicc.i (to compile C programs)The commands listed above use the Intel compilers, which are preferred for compiling programs on lewis. The Intel compilers will generally produce more efficient, faster runninng code that is optimized for the Intel processors that are used in the lewis cluster. However, if you must use the GNU compilers for C/C++ or FORTRAN, you can use the following commands to compile and link your MPI programs with the InfiniBand libraries:
mpiCC.i (to compile C++ programs)
mpif77.i (to compile FORTRAN 77 programs)
mpif90.i (to compile FORTRAN 90 programs)
mpicc -cc=gcc
mpiCC -cc=g++
mpif77 -fc=g77
Submission
Sample job script file to submit compiled MPI program to LSF, with LSF integration for InfiniBand:# Set job parametersThe "-a mvapich" option is critical. Also, the mpirun.lsf command will automatically launch the processes on the correct nodes as allocated by LSF.
#BSUB -a mvapich
#BSUB -J jobname
#BSUB -oo jobname.o%J
#BSUB -eo jobname.e%J
# Set number of CPUs
#BSUB -n 16
# Start MPI job
mpirun.lsf ./mpi_program
If you have an MPI program that is already pre-compiled to communicate via a TCP/IP network, instead of InfiniBand, then substitute "-a mpichp4" for "-a mvapich":
# Set job parametersFor details about the other macros in these scripts see Submitting Jobs via LSF on Lewis.
#BSUB -a mpichp4
#BSUB -J jobname
#BSUB -oo jobname.o%J
#BSUB -eo jobname.e%J
# Set number of CPUs
#BSUB -n 16
# Start MPI job
mpirun.lsf ./mpi_program