Using the Cluster

From NCEAS Knowledge Base

This cluster can be utilized in two ways. The first is with a large number of batch jobs (one per machine), the second using MPI libraries to create a program that runs across all the machines. Both ways should be executed via the sun grid engine.

There are many ways to execute a job on the cluster, but the easiest way follows this pattern.

1. Write a script to run your program
2. Upload program/data to cluster
3. submit script to the queue

The following example will detail submiting a simple job to the queue:

1. Write script

Contents are of the text file "example.sh"

 #!/bin/bash
 #
 #$ -cwd
 #$ -j y
 #$ -S /bin/bash
 #
 date
 

The lines starting with # give the queue system information about the type of job to be run, "date" can be replaced with the name of the program you wish to run, for example, "R CMD BATCH myrjob.r". Use any text editor to create this file, NotePad on Windows works fine. Save as a .sh file.

2. Upload script,program, and data to the cluster

This step is simply copying the example.sh file to the cluster. Again, there are many ways to accomplish this task, for this example we'll be using WinSCP.

A. Connect to the cluster

B. Drag your script (and program/data files, if needed) onto the server, select "copy"

C. That's all, go to step 3

3.Submit script to the queue

A. Connect to the cluster

B. Submit the script into the queue

The submit command here is "qsub example.sh". To view the queue, run "qstat -f". The "date" command executes instantly, so you probably wont see it appear in the queue before it finishes. Any output that normally appears on the screen will be output to a text file named, in this instance, example.sh.o43.