This documentation describes how to use PBS Pro to submit jobs to run on nodes in the Casper cluster. Unless GPUs are required, run jobs that require the use of more than one compute node on Cheyenne.
Procedures for starting both interactive jobs and batch jobs on Casper are described below. Also:
Begin by logging in on Casper (casper.ucar.edu) or Cheyenne (cheyenne.ucar.edu).
Wall-clockThe wall-clock limit on the Casper cluster is 24 hours except as noted below. Specify the hours your job needs as in the examples below. Use either the hours:minutes:seconds format or minutes:seconds. |
Interactive jobsStarting a remote command shell with execcasperRun the execcasper command to start an interactive job. Invoking it without an argument will start an interactive shell on the first available HTC node. The default wall-clock time is 6 hours. To use another type of node, include a select statement specifying the resources you need. The execcasper command accepts all PBS flags and resource specifications as detailed by man qsub. Some common requests include:
Additionally, execcasper provides some convenience flags to simplify requesting job resources. If you specify a select statement as described above, it will take precedence over these flags.
The following execcasper invocations, the first with a select statement and the second with a memory flag, are equivalent.
If you do not include a resource specification by using either a select statement or convenience flags, you will be assigned 1 CPU with 10 GB of memory and no GPUs. If no project is assigned with either the -A option or the DAV_PROJECT environment variable, any valid project listed for your username will be chosen at random. Starting a virtual desktop with vncmgrIf your work with complex programs such as MATLAB and VAPOR requires the use of virtual network computing (VNC) server and client software, use vncmgr instead of execcasper. Using vncmgr simplifies configuring and running a VNC session in a Casper batch job. How to do that is documented here. Batch jobsPrepare a batch script by following one of the examples below. Most Casper batch jobs use the "casper" submission queue. The exception is for GPU development jobs, which are submitted to the "gpudev" submission queue. Be aware that the system does not import your login environment by default, so make sure your script loads the software modules that you will need to run the job. Caution: Avoid using the PBS -V option to propagate your environment settings to the batch job; it can cause odd behaviors and job failures when used in submissions to Casper from Cheyenne. If you need to forward certain environment variables to your job, use the lower-case -v option to specify them. (See man qsub for details.) When your job script is ready, use qsub to submit it from the Casper login nodes. Examples:
See Managing and monitoring jobs for other commonly used PBS commands. Specifying resource requests with select statementsUsers can request certain resources from PBS via a select statement. This syntax allows you to request any number of resource chunks, which will include one or more CPUs, and optionally MPI tasks, Open MP threads, GPUs, custom amounts of node memory, and nodes with a certain CPU type. The general format of a select statement is:
Additionally, the types of GPUs assigned to the job can be constrained by an additional flag, as in this example:
Minimizing your use of resource specifications reduces the length of time your job waits in the queue. In general, ask for the smallest amount of each resource required to perform your task in a timely manner. Also, study the node table on the main Casper page to avoid requesting resource combinations that are not possible. For example, a job submission requesting both gp100s and Cascade Lake processors will sit in the queue indefinitely because the system does not offer that configuration. Two types of GPUs are available:
GPU development jobsA submission queue called "gpudev" is available between 8 a.m. and 5:30 p.m. Mountain time Monday to Friday to support application development and debugging efforts on general purpose and ML/AI GPU applications. This queue provides rapid access to up to 4 V100 GPUs, avoiding the sometimes lengthy queue wait times in the "gpgpu" execution queue. Job submissions to this queue are limited to 30 minutes walltime instead of the 24-hour wallclock limit for all other submissions. All jobs submitted to the queue must request one or more V100 GPUs (up to four) in their resource directives. Node memory can be specified explicitly as usual, but by default jobs will be assigned N/4 of the total memory on a node, where N is the number of V100 GPUs requested. Concurrent resource limitsJob limits are in place to ensure short dispatch times and a fair distribution of system resources. The specific limits that apply to your submission depend on the resources requested by your job. Based on your request, your submission will be classified as shown in the table.
NVMe node-local storageCasper nodes each have 2 TB of local NVMe solid-state disk (SSD) storage. Some is used to augment memory to reduce the likelihood of jobs failing because of excessive memory use. NVMe storage can also be used while a job is running. (Recommended only for I/O-intensive jobs.) Data stored in /local_scratch/pbs.$PBS_JOBID are deleted when the job ends. To use this disk space while your job is running, include the following in your batch script after customizing as needed.
Script examplesThe examples below show how to create a script for running a high-throughput computing (HTC) job. Such jobs typically use only a few CPU cores and likely do not require the use of an MPI library or GPU. See this page for more script examples: Casper job script examples For bash usersInsert your own project code where indicated and customize other settings as needed for your own job.
For tcsh usersInsert your own project code where indicated and customize other settings as needed for your own job. You may see this harmless warning at the start of your PBS batch job output logs on Casper:
It simply means terminal interaction via stdin is not possible because the script is being run in a batch environment. If you need to provide input via stdin, use execcasper to launch an interactive session instead of running a batch job.
Compiling your codeCISL recommends using the default Intel, GNU or PGI compilers for parallel programs, and compiling on a Skylake node unless the code will run exclusively on Cascade Lake nodes.
Serial programs can use any compiler. |