How to use your python script in a batch script with a specific environment

To run your python script in a batch script,

First, save your python code (which is probably a Jupiter notebook now) as a .py (eg. name.py)

Example for compute partition with 256 GB memory

#!/bin/bash

#SBATCH --job-name=name     
#SBATCH --partition=compute
#SBATCH --time=8:00:00
#SBATCH --mem=256GB
#SBATCH --mail-type=FAIL       
#SBATCH --account=bk1377      
#SBATCH --output=name.o%j    
#SBATCH --error=name.e%j 

/work/bm1344/conda-envs/py_312/bin/python /path/to/your/python/file/name.py

second example for shared partition:

#!/bin/bash

#SBATCH --job-name=name     
#SBATCH --partition=shared
#SBATCH --time=7-00:00:00 
#SBATCH --mem=100GB
#SBATCH --mail-type=FAIL       
#SBATCH --account=bk1377      
#SBATCH --output=name.o%j    
#SBATCH --error=name.e%j 

/work/bm1344/conda-envs/py_312/bin/python /path/to/your/python/file/name.py

In shared partition one can set maximum time up-to 7 days. Similarly, another partition option is interactive with maximum 12 hour limit.

Save it (name.sh) and submit the created batch script as sbatch name.sh

If you get any permission denied error, try chmod +x name.sh before doing sbatch name.sh

to check the queue squeue job_ID to cancel scancel job_ID

To get the python environment (eg: /work/bm1344/conda-envs/py_312/bin/python), run following code snippet in your Jupiter notebook shell:

import sys
sys.executable

More descriptive examples and details (such as memory limits, time limits) about batch jobs can be found at;

Partitions and Limits

Example batch scripts