I have deployed an experimental Jupyterhub service (ask me privately if you would like access) installed on a SDSC Cloud virtual machine that spawns single user Jupyter notebooks on Comet computing nodes using batchspawner and then proxies the Notebook back to the user using SSH-tunneling.

Functionality

This kind of setup is functionally equivalent to launching a job yourself on Comet, launch jupyter notebook and then SSH-Tunneling the port to your local machine, but way more convenient. You jus open your browser to the Jupyterhub instance, authenticate with your XSEDE credentials, choose queue and job length and wait for the Notebook job to be ready (generally it is a matter of minutes).

Rationale

Jupyter Notebooks have a lot of use-cases on HPC, it can be used for:

  • In-situ visualization
  • Interactive data analysis when local resources are not enough, either in terms of RAM or disk space
  • Monitoring other running jobs
  • Launch IPython Parallel jobs and distribute computation to them in parallel
  • Interact with a running Spark cluster (we support Spark on Comet)

More on this on my Run Jupyterhub on a Supercomputer old blog post.

Setup details

The Jupyter team created a repository for sample HPC deployments, I added all configuration files of my deployment there, with all details about the setup:

Please send feedback opening an issue in that repository and tagging @zonca.