Running Jobs

Running Jobs at Computational Facilities (pdf download)

This tutorial discusses, 1) how to submit jobs at JHU-MARCC with a sample job submission file, 2) how to run interactive jobs at JHU-MARCC, 3) useful commands related to job submission, 4) useful Linux commands, and 5) how to submit jobs at XSEDE with a sample job submission file.

1. How to Submit Jobs at JHU-MARCC

MARCC policy states that all users must submit jobs to the scheduler for processing. Interactive use of login nodes for job processing is not allowed.

MARCC uses,

• SLURM resource manager to manage resource scheduling and job submission.
• Partitions (different job queues) to divide types of jobs which will allow sequential/shared computing, parallel, GPU jobs, and large memory jobs.

For the complete list of partitions available for users please visit:

https://www.marcc.jhu.edu/getting-started/running-jobs/

If you need further info or assistant, please contact MARCC support at,

marcc-help@marcc.jhu.edu

or

Reach us via the PARADIM computation support forum,

http://forums.paradim.org/forums/forum/theory-forum-cau/
 

Sample job submission scripts

A sample script to run a Quantum Espresso job in parallel partition using 24 cores with 5000MB memory in a single node would look like this,

#!/bin/bash -l

#SBATCH --job-name= myjob-1

#SBATCH --time=00:30:00

#SBATCH --partition=parallel

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=24

#SBATCH --mem-per-cpu=5000MB

mpirun -np 24 pw.x < silicon.in > silicon.out

Jobs are usually submitted via a script file. The sbatch command is used.

$ sbatch my-script

 

2. How to Run Interactive Jobs at JHU-MARCC

Users who need to interact with their codes while these are running can request an interactive session. This will submit a request to the queuing system that will allow interactive access to the node.

If you would like an interactive session, you can use the following command

$ interact -p parallel -n 24 -c 1 -t 60 -m 5G

Here we are requesting 24 CPUs, since -n 24 is the number of tasks, and -c 1 is the number of cores per task. We are asking for a session of 60 min (-t 60) and with a total memory of 5 GB (-m 5G).

This command opens a session where we will be able to execute pw.x directly from the command line without a job submission script.

$ mpirun -n 24 pw.x < silicon.in > silicon.out

3. Useful Commands for job Submission

$ sbatch my-script                          submit a job script          

$ squeue                                            list all jobs

$ sqme                                               list all jobs belong to the current user     

$ squeue -u [userid]                       list jobs by user

$ squeue [job-id]                             check job status

$ scancel [job-id]                             delete a job

$ scontrol hold                                 hold a job                          

$ scontrol release                           release a held job

$ sacct                                               show finished jobs

*Users can still use torque commands like qsub, qdel, qstat, etc…

 

4. Useful Linux Commands

• mkdir – make directories
  Usage: mkdir [OPTION] DIRECTORY…
  eg. mkdir paradim
   
• cd – change directories
  Usage: cd [DIRECTORY]
  eg. cd paradim

• ls – list directory contents
  Usage: ls [OPTION]… [FILE]…
  eg. ls, ls -l, ls paradim
   
• pwd – print name of current working directory
  Usage: pwd
   
• vim – Vi Improved, a programmer’s text editor
  Usage: vim [OPTION] [file]…
  eg. vim myscript.txt
   
• cp – copy files and directories
  Usage: cp [OPTION]… SOURCE DEST
  eg. cp myscript.txt myscript_duplicate.txt, cp -r directory1 directory2
   
• mv – rename/move files
  Usage: mv [OPTION]… SOURCE DESTINATION
  eg. mv myscript.txt directory
  eg. mv myoldscript.txt mynewscript.txt
   
• rm ­ remove files or directories
  Usage: rm [OPTION]… FILE…
  eg. rm myoldscript.txt, rm -rf directory
   
• find – search for files in a directory hierarchy
  Usage: find [OPTION] [path] [pattern]
  eg. find myscript.txt, find -name myscript.txt
   
• history – prints recently used commands
  Usage: history
   
• ps – report a snapshot of the current processes
  Usage: ps [OPTION]
  eg. ps, ps -el
   
• kill – to kill a process
  Usage: kill [OPTION] pid
  eg. kill -9 2275

5. How to Submit Jobs at XSEDE

Allocation for The Extreme Science and Engineering Discovery Environment (XSEDE) is available for collaborative research only. Please contact members of PARADIM theory staff available at /people to get access to XSEDE allocations.

If you already have an allocation with XSEDE, following is a sample job scrip to submit jobs.

#!/bin/bash

#SBATCH -J myMPI                # job name

#SBATCH -o myMPI.o%j            # output and error file name (%j expands to jobID)

#SBATCH -n 32                   # total number of mpi tasks requested

#SBATCH -p development          # queue (partition) -- normal, development, etc.

#SBATCH -t 01:30:00             # run time (hh:mm:ss) - 1.5 hours

#SBATCH --mail-user=username@tacc.utexas.edu

#SBATCH --mail-type=begin       # email me when the job starts

#SBATCH --mail-type=end         # email me when the job finishes

ibrun ./pw.x                   # run the MPI executable named pw.x