Compiling and Running MPI Programs on WestGrid

This document is intended to be for my own reference for most part, but I will try to go through it step-by-step just so anybody new to using ssh client and WestGrid like myself would know what to expect as well. I would recommend that new users read the official QuickStart guide here, if you have not yet done so, before proceeding.

In order to use WestGrid, we’ll need a SSH (Secure Shell) client which act as a terminal for us to access the server and a file transfer client (we can’t use the same terminal for both functions, I’m afraid). I use PuTTY as SSH client and PSFTP for file transfer. These two comes together in an installation package. Other SSH clients and file transfer clients work as well, but please check the list given in the QuickStart guide for compatibility.

Transferring Files

In order to compile a program, we need to transfer the source code to the server first. The logging in is pretty simple as the terminal will guide you through it, but first you must choose the server to log into. In this case, I choose the Checkers on WestGrid. To decide which server to use depends on the task we want to do. I choose Checkers because I will be doing parallel processing using MPI and Checkers supports this.


To choose the server in PSFTP, simply type


It will ask for the username and password in succession. Then, BOOM, you’re in!

There are a number of commands that we can use and we can access the list by typing help. Here are some useful ones to remember:

lcd [directory] This is for accessing local directory like using cd in local command line.
put [file] This is for transferring a file from local machine to the server.
get [file] This is for downloading a file from the server to the local machine.
ls This is for listing the files in your directory on the server
bye This is for leaving the server and PSFTP

I found it easier to first go to the folder where the source file is using lcd then uploading the file by using put. We can check if the file is there by using ls. Once the file is up on the server, we switch to PuTTY to compile the source code. We can log in through PuTTY and PSFTP at the same time, but the login has to be done separately.

Compiling the Source Code

Once the source files are on the server, we can use the compilers on the server. First, we log in using PuTTY. No extra setting is needed aside from specifying the server we want to log in. Once PuTTY starts to make the connection, it will ask for username and password again.


The exact command depend entirely on what language the program was written in. The full description of the programming including compiling on Checkers can be found here. Since I coded my test in C with MPI, I use the command

mpicc -O3 main.c -lm -o main

which I take from here. This command use gcc as the underlying compiler (if you are not familiar with that, it’s a common c compiler in Unix and Linux), therefore the options following mpicc are the same as what you would put in for gcc. -O3 (capital O) is an optimization option. You can find more detail here. -lm link the code to standard math library, and -o [file] simply specify the output.

Submitting and Running the Program

To run the program on WestGrid, we need to send a job request to the system. This is because the computational resources are shared with other jobs and, especially in the case of the MPI, the number of processors and memories had to be put aside for large jobs. The resources on WestGrid is managed by TORQUE which takes the job request in form of batch job scripts. This has to be written on a separate file and uploaded onto the server into the same directory as the output of the compiler.


The .pbs file is written in shell script which always start with

#! /bin/bash

This command specify the path in which the code is to be interpret. It will try to find Bash execution in /bin.

The following lines specify the options for running the job which is written after #PBS. #PBS -S /bin/bash does pretty much the same thing as the line before which is specifying the shell to run the script in. Typically, WestGrid has default limit to how much processor, memory, and time we can use for a job. To specify our own limit we use the commands

#PBS -l procs=10 to specify number of processors for the job
#PBS -l mem=2000mb to specify the amount of memory needed
#PBS -l walltime=72:00:00 to specify the job runtime.

Different resources which can be set under -l can be done at once by separating them with commas or specify when submitting a job by putting the option after qsub command.

qsub -l procs=10,pmem=2gb,walltime=72:00:00 main.pbs

If the resources have already been specify, we can simply go qsub main.pbs. For details and options that WestGrid support, read here.

In the later part of the script # will be used as comment symbol.

cd $PBS_O_WORKDIR find the directory where the job is (essentially where the .pbs file is). $PBS_O_WORKDIR let TORQUE find the path automatically rather than typing the absolute path ourselves. The dollar sign indicates that it is a variable.

echo is used to display information of the job. However, the information is not displayed on the terminal and it is actually a better idea to store your output as a file on your server directory instead.

The command to run the program in my case is mpiexec ./main. Number of processors can also be specified here if it hasn’t been previously by using option -n [# of procs] before ./main. More option for command mpiexec can be found here.

Transferring Output File

Once WestGrid finish running the job, we should have the output file ready in the server directory. To move this file to local machine, we once again switched to PSFTP and use command get [file]. This will move the file to your local directory which is presumably the source code directory from earlier. Once we’re satisfy that that we have the correct result, we can log out from PSFTP using the command bye and from PuTTY by simply shutting down the window.


Tags: , , , ,

About vekin

I'm mostly a scientist, but occasionally a writer and an artist, and most definitely a dreamer.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: