Restore ability to call from a terminal command

geos-chem-schedule

Reads a GEOS-Chem (http://geos-chem.org/) run directory's input file (input.geos) and uses this to split up a job multiple jobs. These jobs can be submitted as part of this process or manually submitted at a later point.

This script intends to make it easier to split up GEOS-Chem jobs into smaller parts that can better fit into queues available on a high-performance computing (HPC) facility. Currently, this script is compatible with PBS and SLURM.

Install

To install download this repository with git clone. Recommended location would be $HOME/src/geos-chem-schedule. Once downloaded, you can just add a symbolic link for the geos-chem-schedule.py to a run directory and run by calling directly in that directory (e.g. Python geos-chem-schedule.py). Alternatively, you can install to be run through bash and this is done by navigating to the geos-chem-schedule folder containing geos-chem-schedule.py and running the following steps:

Download the script and run the setup with the following commands.

mkdir -p $HOME/src
cd $HOME/src
git clone https://github.com/wacl-york/geos-chem-schedule.git
python geos-chem-schedule/geos-chem-schedule.py --setup

OR

If you have AC_tools downloaded (https://github.com/tsherwen/AC_tools), Go to your AC_tools dir and do the following commands.

git submodule update --recursive --remote
python Scripts/geos-chem-schedule/geos-chem-schedule.py --setup

Copy and paste the command provided into the terminal, which will allow you to use the command "geos-chem-schedule.py" from any folder

Edit your settings.json file for options like default memory requirements, default run queue, default job name, and add your email address.

Use

Via Python

Create a symbolic link in the GEOS-Chem run directory and then call this and follow the steps on screen

ln -s <route to geos-chem-schedule>/geos-chem-schedule/geos-chem-schedule.py .
Python geos-chem-schedule.py

Via bash

Go to your GEOS-Chem run directory and confirm your input.geos file is correct. type the following command if you have followed up the setup:

geos-chem-schedule

Follow the UI instructions on screen.

The final option allows you to run the script immediately, or if you want to run the command later, it creates a file run_geos.sh. This file can be executed by typing:

bash run_geos.sh

The script has a UI to chose job name, queue name, priority, if you want to start the jobs outside of work hours, and if you want to have the script submit the job to the queue.

The script can also take command line arguments. Type geos-chem-schedule.py --help for more info. This can be useful if you have lots of simulations you want to send off via a script.

For example:

geos-chem-schedule.py --job-name=bob --step=month --queue-name=run --queue-priority=100 --out-of-hours=yes --submit=yes

This will give the the job a name of 'bob' in the 'run' queue, split up the jobs into months, and run the job with priority of 100 (only availible for PBS jobs currently). The jobs will only start if out-of-hours and if the job starts in working hours it will resubmit itself with a command to wait until 1800. The job will be submitted at the end of the script.

WARNINGS:

If using bpch output for GEOS-Chem instead of the default NetCDF (v11+), then note this script forces bpch output to be produced (setting=3) for the end of simulation date and replaces all other days with a 0. If you want every day to run with a 3 then use --step=daily.

History

2020-07-31

Updated to allow use of script with SLURM schedular General re-writing of functions for clarity and addition of documentation Various fixes applied (inc. enable email functionality for SLURM jobs) Repository name updated to geos-chem-schedule to reflect functionality

2016-11-14

Allow options for --step=week,day,month Allow setup via geos-chem-schedule.py --setup Code changes to make it more readable

2015-04-01

Updated the naming scheme of the log to YYYYMMDD.geos.log Changed from an error if the job name is over 9 characters to truncating the name to only 9 characters.

2015-02-18

Added options to submit arguments from the command line instead of the UI Added an option to only start jobs out of work hours (0800-1800 Monday - Friday) Changed the naming scheme of the logs.

2015-01-14

Option to send the run script to the queue straight away. Option to name the job (up to 9 characters). Option to name the queue you which to run on with error checking. Option to chose priority of the jobs. Now only sends one job to the queue at a time, reducing mess in qstat. Now calls then next month upon completion of the current month.

	def get_arguments(inputs, debug=False):
	"""
	Get the arguments supplied from command line

	Parameters
	-------
	inputs (GC_Job class): Class containing various inputs like a dictionary
	debug (bool): Print debugging output to the screen

	Returns
	-------
	(GC_Job class)
	"""
	# If there are no arguments then run the GUI
	if len(sys.argv) > 1:
	for arg in sys.argv:
	if "geos-chem-schedule" in arg:
	continue
	if arg.startswith("--setup"):
	setup_script()
	elif arg.startswith("--job-name="):
	inputs.job_name = (arg[11:].strip())[:9]
	elif arg.startswith("--step="):
	inputs.step = arg[7:].strip()
	elif arg.startswith("--queue-name="):
	inputs.queue_name = arg[13:].strip()
	elif arg.startswith("--queue-priority="):
	inputs.queue_priority = arg[17:].strip()
	elif arg.startswith("--submit="):
	inputs.run_script_string = arg[9:].strip()
	elif arg.startswith("--out-of-hours="):
	inputs.out_of_hours_string = arg[15:].strip()
	elif arg.startswith("--wall-time="):
	inputs.wall_time = arg[12:].strip()
	elif arg.startswith("--cpus-need="):
	inputs.cpus_need = arg[12:].strip()
	elif arg.startswith("--submit_jobs_together="):
	inputs.cpus_need = arg[23:].strip()
	elif arg.startswith("--memory-need="):
	inputs.memory_need = arg[14:].strip()
	elif arg.startswith("--help"):
	print("""
	geos-chem-schedule.py

	For UI run without arguments
	Arguments are:
	--job-name=
	--step=
	--queue-name=
	--queue-priority=
	--submit=
	--out-of-hours=
	--wall-time=
	--submit_jobs_together=
	--memory-need=
	--cpus-need=
	e.g. to set the queue name to 'bob' write --queue-name=bob
	""")
	else:
	print("""Invalid argument {arg}
	Try --help for more info.""".format(arg=arg)
	)
	sys.exit(2)
	else:
	inputs = get_variables_from_cli(inputs)
	return inputs

	def setup_script():
	"""
	Creates a symbolic link to allow running "geos-chem-schedule" from any directory
	"""
	print("\n",
	"geos-chem-schedule setup complete. Change your default settings in settings.json\n",
	"To run the script from anywhere with the geos-chem-schedule command,",
	"copy the following code into your terminal. \n")

	script_location = os.path.realpath(__file__)
	# make sure the script is excecutable
	print("chmod 755 {script}".format(script=script_location))
	# Make sure there is a ~/bin file
	print("mkdir -p $HOME/bin")
	# Create a symlink from the file to the bin
	print("ln -s {script} $HOME/bin/geos-chem-schedule".format(script=script_location))
	# Make sure the ~/bin is in the bashrc
	# with open('$HOME/.bashrc','a') as bashrc:
	# bashrc.write('## Written by geos-chem-schedule')
	# bashrc.write('export PATH=$PATH:$HOME/bin')
	print('echo "## Written by geos-chem-schedule " >> $HOME/.bashrc')
	print('echo "export PATH=\$PATH:\$HOME/bin" >> $HOME/.bashrc')
	# Source the bashrc
	print("source $HOME/.bashrc")
	print("\n")
	sys.exit()

wacl-york / geos-chem-schedule Goto Github PK

geos-chem-schedule's Introduction

geos-chem-schedule

Install

Use

Via Python

Via bash

WARNINGS:

History

geos-chem-schedule's People

Contributors

Stargazers

Watchers

Forkers

geos-chem-schedule's Issues

Recommend Projects

Recommend Topics

Recommend Org