Execute Jupyter Notebooks not interactively

Published: Mon 23 September 2019
In: misc
Tags: jupyter notebook condor
Written by: Andrea Zonca

Over the years, I have explored how to scale up easily computation through Jupyter Notebooks by executing them not-interactively, possibily parametrized and remotely. This is mostly for reference.

nbsubmit is a Python package which has Python API to send a local notebook for execution on a remote SLURM cluster, for example Comet, see an example. This project is not maintained right now.
Back in 2017 I tested submitting notebooks to Open Science Grid, see the batch-notebooks-condor repository
Back in 2016 I created scripts to template a Jupyter Notebook and launch SLURM jobs, see slurm.shared.template and runipyloop.sh