Posts

Mastodon

Hiring a Computational Scientist at the San Diego Supercomputer Center

hpc
Come work in my…
Apr 24, 2023

Cosmology webinar Hyper Supreme Cam

cosmology
Webinar about Cosmology results, in particular tension in \(S_8\) by the Hyper Supreme Cam collaboration:
Apr 23, 2023

Get a Letsencrypt certificate for Dreamhost with certbot

Dreamhost has discontinued offering free Letsecrypt certificated in their shared hosting.
Apr 1, 2023

Update Openstack credentials in Kubernetes

kubernetes
jetstream
If you…
Mar 23, 2023

Work on a Latex Document in Github Codespaces

git
github
I was exploring using Github Codespaces instead of Overleaf to work on a Latex document.
Mar 16, 2023

Gzipping or not Gzipping HEALPix maps

healpy
nersc
HEALPix maps are generally stored in FITS format, they can be Gzipped to save disk space. Reading gzipped maps is natively supported in healpy, however, it takes longer…
Mar 6, 2023

Share data from NERSC publicly via web

nersc
NERSC allows to make data which is on their Community File System publicly available via a web interface.
Feb 24, 2023

Showyourwork! Globus demo

python
Showyourwork! is a great project to make scientific articles reproducible.
Feb 21, 2023

Deploy a NFS server to share data between JupyterHub users on Jetstream

kubernetes
jetstream
jupyterhub
linux
This tutorial is a minor update of https://www.zonca.dev/posts/2020-07-10-nfs-server-kubernetes-jetstream.
Feb 6, 2023

Work on a Latex Document with Github and Overleaf

git
github
In the past I have written about coordinating a large document using multiple overleaf documents and Github repositories. In this post I will just layout the simpler case where we have a Latex document on Overleaf and we also want users to be able to work on it from Github and we want…
Feb 2, 2023

Deploy Kubernetes on Jetstream 2 with GPU support

jetstream2
jupyterhub
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website.
Jan 23, 2023

Compare WebSky Radio Galaxies maps from PySM to Planck

healpy
pysm
I need to check that the amplitude of the maps of Radio Galaxies produced by WebSky are reasonable if compared to experimental…
Dec 5, 2022

Share data between JupyterHub users with Manila shares

jetstream2
kubernetes
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website.
Dec 5, 2022

Deploy BinderHub on top of Kubernetes on Jetstream 2

jetstream2
This work has been supported by Indiana University and is cross-posted on the Jetstream 2 official documentation website.
Nov 15, 2022

Setup a conda environment at NERSC

python
nersc
NERSC recommends to use the “Global Common” to store Conda environments, because it is optimized to store lots of small…
Nov 8, 2022

Build and host Singularity containers on Github

singularity
I present a demo repository configured to build Singularity containers using Github actions and then host them using the Github container registry:
Nov 3, 2022

Generate Click command line options dynamically from class arguments

python
In this application I need to use command line options to create class objects. In order not to repeat all class arguments in the Click configuration, I created a simple…
Oct 26, 2022

Closures in Numba

python
Closures can be used to dinamically create different versions of a function based on some parameter, a similar functionality can be achieved with functools.partial.
Oct 24, 2022

Install the JupyROOT Python kernel in JupyterHub

jupyterhub
After installing ROOT’s conda package, for example with micromamba:
Oct 11, 2022

Singularity on Expanse tutorial

singularity
hpc
jetstream2
As one of my last tasks in XSEDE, I updated and improved the tutorial about running Singularity containers in the HPC system Espanse at my institution the San…
Oct 5, 2022

Migrate from fastpages to quarto preserving git history

Most of my blog posts are in Markdown…however, there are cases where plotting and code are very important, and nothing beats a Jupyter Notebook for that.
Sep 27, 2022

Deploy MariaDB on Jetstream 2 on top of Kubernetes

kubernetes
jetstream2
In this tutorial we will install a MariaDB instance backed by a persistent volume on Jetstream 2. It will be in the jhub namespace, so that it can be…
Jun 6, 2022

Remove unique cell id from Jupyter Notebooks

git
jupyter
I know! Jupyter is littering your git diff with randomly generated cell ids and nbstripout doesn’t remove them, (I’m sure they are useful for some reason).
May 17, 2022

Access running Github action with SSH

github
Sometimes Github actions are failing and it is difficult to reproduce the error locally, in particular if you have a different OS.
May 2, 2022

Monitor Restic backups on Kubernetes

jetstream2
kubernetes
For one of my production JupyterHub deployments on Kubernetes, I have setup an automated system to perform nightly backup of the user data, see the…
Apr 27, 2022

Jetstream2 SU calculator

jetstream2
Jupyter Notebook to compute daily, monthly, yearly consumption of SU based on the number and type of Virtual Machines:
Apr 15, 2022

Custos authentication for JupyterHub

kubernetes
jetstream2
jupyterhub
Custos is a security middleware used to authenticate users to…
Apr 13, 2022

Science Gateway with Dask and Zarr

kubernetes
jetstream2
jupyterhub
dask
This material was presented on April 2022 at the MiniGateways 2022 conference organized by the wonderful Science Gateways Community Institute (SGCI).
Apr 5, 2022

Use the distributed file format Zarr on Jetstream 2 object storage

dask
jetstream2
Zarr is a file format designed for cloud computing, see documentation.
Apr 4, 2022

Deploy Dask Gateway with JupyterHub on Kubernetes

kubernetes
jetstream2
jupyterhub
dask
Updated 28 April 2022: switched to Dask Gateway 2022.4.0
Apr 4, 2022

Deploy JupyterHub on Jetstream 2 on top of Kubernetes

kubernetes
jupyterhub
jetstream2
This tutorial is a followup to: Deploy Kubernetes on Jetstream 2 with Kubespray 2.18.0, so I’ll assume Kubernetes is already deployed with a default storageclass.
Mar 31, 2022

Deploy Kubernetes on Jetstream 2 with Kubespray 2.18.0

kubernetes
jetstream2
This is the first tutorial targeted at Jetstream 2. The system is in early access and will be soon made available, see https://jetstream-cloud.org/.
Mar 30, 2022

Workflow for Jupyter Notebooks under version control

jupyter
git
I’ll present here my strategy for keeping Jupyter Notebooks under version control.
Mar 22, 2022

Deploy Dask Gateway with JupyterHub on Kubernetes

kubernetes
jetstream
jupyterhub
dask
Tutorial OBSOLETE Please check the updated version of this tutorial.
Jan 27, 2022

Make for repeated commands

linux
I often find myself repeat the same commands in a project, for example in Openstack/Kubernetes, but also when I need to interface with Docker (build/push image).
Jan 7, 2022

Fund healpy via Github Sponsors

healpy
Github Sponsor page
Nov 8, 2021

Kubernetes certifications CKA and CKAD

kubernetes
I recently pursued 2 Kubernetes certifications by Linux Foundation:
Nov 4, 2021

Compare HEALPix and Gauss-Legendre pixelizations for sky emission modelling

healpy
cosmology
We compare the transform from spherical harmonics space to pixel space and back using HEALPix and Gauss-Legendre pixelization. The context is the evaluation of sky emission…
Nov 2, 2021

Deploy Hadoop on Kubernetes on Jetstream

kubernetes
jetstream
We are deploying the good old Hadoop on top of Kubernetes on Jetstream. Don’t ask why.
Oct 12, 2021

Plot HEALPix pixel boundaries

healpy
Some time ago I created a Jupyter Notebook based on plotly to create a 3D visualization of the boundaries of HEALPix pixels on the sphere.
Oct 6, 2021

Monitor traffic on Github repositories

github
Github only shows traffic data for the last 2 weeks. Through the API is possible to gather those data every 2 weeks and save them for later collection and reduction.
Oct 1, 2021

Manage Globus groups with the Python SDK

python
hpc
Globus is the best tool to…
Sep 24, 2021

New paper about PySM 3

openscience
python
cosmology
Update November 2021
Aug 4, 2021

Upload a JOSS paper to Arxiv

github
openscience
If you are in Astrophysics, you probably want to have your JOSS paper published to the Arxiv.
Aug 1, 2021

Configure dreamhost email

Configuration to read Dreamhost email from Gmail:
Jun 27, 2021

healpy 1.15.0 released

healpy
python
Just released healpy 1.15.0, this is a large release with many important changes, please test and open issues at https://github.com/healpy/healpy/issues
Jun 22, 2021

Investigate rotation of masks in healpy

healpy
cosmology
Investigating a question about healpy on Stackoverflow
Jun 18, 2021

Learn Italian in San Diego

italian
My wife Maura D’Andrea recently opened a new Italian language school in San Diego.
Jun 17, 2021

Migrate from Google Docs to Overleaf and Github

github
Once a document on Google Docs becomes too big and complicated to make…
May 20, 2021

Correct a power spectrum by the instrument beam with healpy

healpy
cosmology
If you are analyzing a map from an instrument with…
Apr 27, 2021

Backup Kubernetes volumes to OpenStorageNetwork object store

jetstream
kubernetes
In my specific scenario, I have users running JupyterHub on top of Kubernetes on the Jetstream XSEDE Cloud resouce. Each user has a persistent…
Apr 19, 2021

Science Gateways Tech blog - Kubernetes and JupyterHub on Jetstream

kubernetes
jetstream
This is cross-posted from the Science Gateways Community institute tech blog, thanks to Marlon Pierce and Katherine Lawrence for their feeback.
Mar 28, 2021

Stream video from object store on Jetstream

jetstream
In this tutorial we will serve video…
Mar 25, 2021

Tutorial on how to rotate maps in healpy

In this notebook we will explore different ways of rotating maps in healpy.
Mar 11, 2021

Compute the Planck CMB temperature power spectrum with healpy anafast

healpy
cosmology
In this notebook we will load the Planck CMB-only temperature map and try to reproduce the Planck CMB power spectrum in temperature just using healpy.
Feb 28, 2021

Coordinate a large Latex document with multiple Overleaf projects and Github

git
github
In case we need to build a large document (~hundreds of pages) and we have a large number of collaborators (more than…
Jan 28, 2021

Autoscaling script for JupyterHub on top of Kubespray

kubernetes
jetstream
I think the most reliable way of deploying Kubernetes on Openstack is Kubespray, see my latest tutorial about it focused on Jetstream.
Jan 20, 2021

Deploy Kubernetes on Jetstream with Kubespray 2.15.0

kubernetes
jetstream
This is an update to previous tutorials, focused on…
Jan 20, 2021

CFITSIO file writing with checkpointing

astrophysics
I wrote a prototype implementation of writing FITS files in C checkpointing using the FITS flush function so that if the code segfaults, the file is not corrupted, see:
Nov 30, 2020

Paper review workflow with Overleaf, git and Google Docs

openscience
git
In this blog post I propose a workflow to use the collaborative features of Google Docs to review a…
Nov 18, 2020

Install HEALPix and PolSpice in a conda environment

healpy
hpc
nersc
Some notes on how to install HEALPix and PolSpice inside a conda environment, with some details about doing it at NERSC, but most of the tutorial is independent of that.
Oct 14, 2020

Example of the healpy query disc function

python
healpy
cosmology
lonlat=True switches ang2vec from requiring colatitude \(\theta\) and longitude \(\phi\) in radians to longitude and latitude in degrees (notice that also the order changes)
Oct 2, 2020

Read and process Planck CMB power spectra with healpy

healpy
cosmology
In…
Sep 30, 2020

Edit video recordings with ffmpeg

A few scripts to speedup, concatenate and possibly other operations using ffmpeg on linux (also works on Chromebooks) from the command line:
Sep 25, 2020

How to share Jupyter Notebooks

jupyter
github
I tend to preserve and post online all the work I do for later…
Sep 15, 2020

Simple WCS with astropy modeling and gwcs

astrophysics
Some notes on how to convert a FITS WCS to a WCS from gwcs (Generalized WCS) to be used within the JWST pipeline
Sep 10, 2020

Gateways 2020 paper about Kubernetes and JupyterHub on Jetstream

jupyterhub
jetstream
kubernetes
My paper…
Sep 4, 2020

Unit conversion with broadband detectors looking at the CMB

healpy
cosmology
We measure the sky with out broadband instrument, we assume we only measure the CMB solar dipole, initially the units are arbitrary, for example Volts of our instrument.
Sep 4, 2020

Unit conversion with broadband detectors looking at foregrounds

cosmology
We measure the sky with out broadband instrument, we assume we only measure the CMB solar dipole, initially the units are arbitrary, for example Volts of our instrument.
Sep 4, 2020

Migrate from Travis-CI and Readthedocs to Github actions

python
github
For a number of years I have been concerned…
Sep 1, 2020

Redirect readthedocs documentation to another website

python
This is useful if you switch to hosting your own documentation, for example using Sphinx Multiversion on Github pages, tutorial coming soon.
Sep 1, 2020

Renew letsencrypt certificate with NGINX

linux
If you have a default domain already configured with NGINX, the easiest way to renew manually a domain is to temporarily disable your custom domain and then use certbot-auto renew the…
Aug 24, 2020

Investigate broken implementation of bandpass unit conversion in PySM 3

cosmology
Inv…
Aug 24, 2020

Deploy Dask Gateway with JupyterHub on Kubernetes

kubernetes
cloudcomputing
jetstream
jupyterhub
dask
This tutorial is obsolete, please follow https://zonca.dev/2022/01/dask-gateway-jupyterhub.html
Aug 11, 2020

Dask array rounding

python
dask

Dask array rounding float32
Aug 4, 2020

Deploy a NFS server to share data between JupyterHub users on Jetstream

kubernetes
jetstream
jupyterhub
linux
Important: This tutorial is obsolete, please refer to the new version from February 2023
Jul 10, 2020

Handle white noise with healpy 3 not-uniform and partial sky coverage

cosmology
python
healpy

Simulate white noise maps and use hitmaps
Jun 21, 2020

Handle white noise with healpy 2 partial sky coverage

cosmology
python
healpy

Simulate white noise maps and use hitmaps
Jun 20, 2020

Handle white noise with healpy 1 Full sky coverage

cosmology
python
healpy

Simulate white noise maps and use hitmaps
Jun 19, 2020

Deploy Kubernetes on Jetstream with Kubespray

kubernetes
jetstream
This tutorial is obsolete, check the updated version of the tutorial
Jun 15, 2020

My own git cheatsheet

git
I noticed I always google the same procedures…
Jun 1, 2020

Import and export list of blocked users on Twitter

python
Back in 2015, Twitter provided the feature to export and import lists of blocked users, unfortunately they discontinued this service.
Jun 1, 2020

Minimal example of readthedocs configuration for conda

python
Prompted by the announced better support of conda in readthedocs, more memory! I setup a Python package with conda to automatically build the documentation on readthedocs.
May 27, 2020

Deploy Kubernetes and JupyterHub on Jetstream with Magnum

kubernetes
jetstream
jupyterhub
This tutorial deploys Kubernetes on Jetstream with Magnum and then JupyterHub on top of that using zero-to-jupyterhub.
May 21, 2020

White noise NET in Radio-astronomy and Cosmology

cosmology
python
healpy

Create a white noise map and compare with power spectrum expected from the NET
Apr 24, 2020

ECSS Symposium introduction to cloud computing with Jetstream and deployment of Kubernetes

jupyterhub
jetstream
kubernetes
ECSS Symposium, April 2020, Web presentation to the XSEDE Extended Collaborative Support Services.
Apr 20, 2020

Use the Jetstream object store

jetstream
I plan to collect here notes about using the Openstack object store on Jetstream.
Apr 1, 2020

Raise and check a flag array with numpy

python

Handle flag array bits with numpy bitwise operations
Mar 16, 2020

Setup HTTPS on Kubernetes with Letsencrypt

kubernetes
jetstream
jupyterhub
Updated in August 2022: Add patching script to run on master node
Mar 13, 2020

Kill Jupyter Notebook servers

python
jupyter
Just today I learned how to properly stop previously running Jupyter Notebook servers, here for future reference:
Mar 12, 2020

Migrate from Pelican to Fastpages

python
I have been using the Pelican static website generator for a few years, hosting the content on Github, automatically build on push via Travis-CI and deploy on Github pages to zonca.github.i…
Mar 9, 2020

Deploy CVMFS on Kubernetes

kubernetes
jetstream
jupyterhub
CVMFS is a software distribution service, it is used by High Energy Physics experiments at CERN to synchronize software environments across the whole collaborations.
Feb 26, 2020

Organize calendars for a large scientific collaboration

openscience
Many scientific collaborations have a central calendar, often hosted on Google Calendar, to coordinate Teleconferences, meetings and events across timezones.
Dec 2, 2019

Simulate users on JupyterHub

kubernetes
jetstream
jupyterhub
Updated January 2021
Oct 30, 2019

Execute Jupyter Notebooks not interactively

jupyter
Over the years, I…
Sep 23, 2019

Deploy Cluster Autoscaler for Kubernetes on Jetstream

kubernetes
jetstream
jupyterhub
The Kubernetes Cluster Autoscaler is a service that runs within a Kubernetes cluster and when there are not enough resources to…
Sep 12, 2019

Create a Github account for your research group with free private repositories

github
git
openscience
Github allows a research group to create their own webpage where they can host, share and develop their software using the git version control system and…
Aug 24, 2019

Ship large files with Python packages

python
It is often useful to ship large data files together with a Python package, a couple of scenarios are:
Aug 21, 2019

Deploy Kubernetes and JupyterHub on Jetstream with Magnum

kubernetes
jetstream
jupyterhub
This tutorial is obsolete, please checkout the updated version at https://zonca.dev/2020/05/kubernetes-jupyterhub-jetstream-magnum.html.
Jun 14, 2019

Webinar about distributed computing with Python

python
hpc
Recording available of the webinar I gave about “Distributed computing with Python”:
May 30, 2019

Kubernetes monitoring with Prometheus and Grafana

kubernetes
jetstream
jupyterhub
Updated September 2020
Apr 20, 2019

Inherit group permission in folder

hpc
linux
I have googled this so many times…
Mar 24, 2019

Kubernetes monitoring with Dashboard, Prometheus, and Grafana

kubernetes
jetstream
jupyterhub
Install with helm:
Feb 22, 2019

Scale Kubernetes manually on Jetstream

kubernetes
jetstream
jupyterhub
We would like to modify the number of Openstack virtual machines available to Kubernetes. Ideally we would like to do this automatically based on the load on JupyterHub…
Feb 22, 2019

Deploy Kubernetes with Kubespray 2.8.2 and JupyterHub with helm recipe 0.8 on Jetstream

kubernetes
jetstream
This tutorial is obsolete, check the updated version of the tutorial
Feb 22, 2019

Use the distributed file format Zarr on Jetstream Swift object storage, 2019

jupyterhub
jetstream
dask
This is an updated version of the 2018 edition
Jan 24, 2019

Deploy Pangeo on Kubernetes deployment on Jetstream created with Kubespray

kubernetes
jetstream
jupyterhub
The Pangeo collaboration for Big Data Geoscience maintains a helm chart with a prefigured JupyterHub deployment on Kubernetes which also supports launching private dask workers. This is very useful because the Jupyter…
Dec 20, 2018

Setup two factor authentication for UCSD, and Lastpass

Starting at the end of January 2019 UCSD requires every employee to have activated two factor authentication.
Dec 12, 2018

Deploy JupyterHub on a Supercomputer for a workshop or tutorial 2018 edition

jupyterhub
hpc
I described how to deploy JupyterHub with each user session running on a different node of a Supercomputer in my paper for PEARC18, however things are moving fast in the space and I am employing a…
Nov 7, 2018

Deploy JupyterHub on a Supercomputer for a workshop or tutorial 2018 edition

jupyterhub
hpc
I described how to deploy JupyterHub with each user session running on a different node of a Supercomputer in my paper for PEARC18, however things are moving fast in the space and I am employing a…
Nov 7, 2018

Advanced pandas with Astrophysics example Notebook

jupyter
astrophysics
Taught a lesson today on advanced python and pandas based on an example application in Astrophysics with simulations of data from the Planck Satellite, features also a Binder button to run it yourself.…
Oct 26, 2018

Bring your computing to the San Diego Supercomputer Center

hpc
I am often asked what computing resources are available at the San Diego Supercomputer Center for scientists and what is the best way to be…
Oct 24, 2018

Deploy JupyterHub on Kubernetes deployment on Jetstream created with Kubespray 3/3

kubernetes
jetstream
jupyterhub
All of the following assumes you are logged in to the master node of the Kubernetes cluster deployed with kubespray and checked out the repository:
Sep 24, 2018

Explore a Kubernetes deployment on Jetstream with Kubespray 2/3

kubernetes
jetstream
This is the second part of the tutorial on deploying Kubernetes with kubespray and JupyterHub on Jetstream.
Sep 23, 2018

Deploy Kubernetes on Jetstream with Kubespray 1/3

kubernetes
jetstream
This tutorial is obsolete, check the updated version of the tutorial
Sep 23, 2018

PEARC18 Paper on Deploying Jupyterhub at scale on XSEDE

singularity
jetstream
jupyterhub
kubernetes
Bob Sinkovits and I are presenting a paper at PEARC18 about:
Jul 23, 2018

Updated Singularity images for Comet

singularity
hpc
Back in January 2017 I wrote a blog post about running Singularity on Comet.
Jul 22, 2018

Create DockerHub auto build

github
It is very convenient to create Autobuild repositories on DockerHub linked to a Github repository with a Dockerfile. Then every time you commit to Github, Dockerhub is going…
Jul 19, 2018

How to organize code and data for simulations at NERSC

hpc
python
nersc
I recently improved my strategy for organizing code and data for simulations run at NERSC, I’ll write it here for reference.
Jun 20, 2018

Setup private dask clusters in Kubernetes alongside JupyterHub on Jetstream

jupyterhub
kubernetes
jetstream
dask
In this post we will leverage software made available by the Pangeo community to allow each user of a Jupyterhub instance deployed on Jetstream on top of Kubernetes to…
Jun 7, 2018

How to post a PEARC18 paper pre-print to Arxiv

openscience
Follows the step-by-step version:
May 12, 2018

Launch a shared dask cluster in Kubernetes alongside JupyterHub on Jetstream

jupyterhub
jetstream
dask
Let’s assume we have already a Kubernetes deployment and have installed JupyterHub, see for example my previous tutorial on Jetstream. Now that users can login and access a Jupyter Notebook, we would also like to provide them more computing power for their interactive…
May 4, 2018

Install a BOINC server on Jetstream

jetstream
cloudcomputing
BOINC is the leading platform for volunteer computing.
Mar 29, 2018

Use the distributed file format Zarr on Jetstream Swift object storage

jupyter
jetstream
cloudcomputing
Updated again in January 2019
Mar 3, 2018

Use the distributed file format Zarr on Jetstream Swift object storage

jupyter
jetstream
Zarr is a pretty new file format designed for cloud computing, see documentation and a webinar for more details.
Mar 3, 2018

Install custom Python environment on Jupyter Notebooks at NERSC

jupyterhub
python
hpc
nersc
NERSC has…
Dec 21, 2017

ECSS Symposium about Jupyterhub deployments on XSEDE

jupyterhub
jetstream
cloudcomputing
ECSS Symposium, 19 December 2017, Web presentation to the XSEDE Extended Collaborative Support Services.
Dec 15, 2017

Deploy scalable Jupyterhub with Kubernetes on Jetstream

jupyterhub
jetstream
cloudcomputing
The best infrastructure available to deploy Jupyterhub at scale…
Dec 5, 2017

Store a conda environment inside a Notebook

jupyter
python
Last August, during the Container Analysis Environments Workshop held at Urbana-Champaign, we had…
Dec 4, 2017

How to modify Singularity images on a Supercomputer

singularity
hpc
Singularity allows to run your own OS within most Supercomputers, see my previous post about Running Ubuntu on Comet via Singularity
Nov 6, 2017

Deploy scalable Jupyterhub on Docker Swarm mode

jupyterhub
jetstream
cloudcomputing
Jupyterhub genrally requires…
Oct 26, 2017

Setup automated testing on a Github repository with Travis-ci

github
git
Start from the CUDA 8 image from NVIDIA:
Oct 25, 2017

Setup automated testing on a Github repository with Travis-ci

github
git
It…
Sep 6, 2017

Deployment of Jupyterhub with Globus Auth to spawn Notebook on Comet in Singularity containers

jupyterhub
singularity
hpc
Follow the instructions at https://github.com/zonca/singularity-comet to build images from the ubuntu_anaconda_jupyterhub.def and centos_anaconda_jupyterhub.def definition files, or use the containers I have…
Aug 11, 2017

How to create pull requests on Github

git
github
Pull Requests are the web-based version of sending software patches…
Jun 30, 2017

How to create pull requests on Github

git
github
cloudcomputing
Pull Requests are the web-based version of sending software patches via email to code maintainers. They allow athat has no access to a code repository to submit a code…
Jun 30, 2017

Deploy Jupyterhub on a Supercomputer with SSH Authentication

python
jupyterhub
hpc
The best way to deploy Jupyterhub with an interface to a Supercomputer is through the use of batchspawner. I have a sample deployment explained in an older blog post: https://zonca.github.io/2017/02/sample-deployment-jupyterhub…
May 16, 2017

Configure Globus on your local machine for GridFTP with XSEDE authentication

linux
All the commands are executed on your local machine, the purpose of this tutorial is to be able to use globus-url-copy to copy efficiently data back and forth between your…
Apr 19, 2017

Sample deployment of Jupyterhub in HPC on SDSC Comet

python
jupyterhub
hpc
I have deployed an experimental Jupyterhub service (ask me privately if you would like access) installed on a SDSC Cloud virtual machine that spawns…
Feb 26, 2017

Customize your Python environment in Jupyterhub

python
jupyterhub
Usecase: You have access to a Jupyterhub server and you would like to install some packages but cannot use pip install and modify the systemwide Python installation.
Feb 24, 2017

Automated deployment of Jupyterhub with Ansible

jupyterhub
cloudcomputing
Last year I wrote some tutorials on simple deployments of Jupyterhub on Ubuntu 16.04 on the OpenStack deployment SDSC Cloud, even if most of the steps…
Feb 3, 2017

How to publish your research software to Github

git
github
openscience
Nowadays many journals require that the software used to…
Feb 1, 2017

Run Ubuntu in HPC with Singularity

singularity
hpc
If your answer to any of those question is yes, read on! Otherwise, well, still read on, it’s awesome!
Jan 13, 2017

Jupyterhub Docker Spawner with GPU support

python
jupyterhub
Docker Spawner allows users of Jupyterhub to run Jupyter Notebook inside isolated Docker Containers. Access to the host NVIDIA GPU was not allowed until NVIDIA release the NVIDIA-do…
Oct 12, 2016

Jupyterhub deployment on multiple nodes with Docker Swarm

jupyterhub
cloudcomputing
This post is part of a series on deploying Jupyterhub on OpenStack tailored at workshops, in the previous posts I showed:
May 24, 2016

Quick Jupyterhub deployment for workshops with pre-built image

python
jupyterhub
This tutorial explains how to use a OpenStack image I already built to quickly…
Apr 28, 2016

Deploy Jupyterhub on a Virtual Machine for a Workshop

python
jupyterhub
This tutorial describes the steps to install a Jupyterhub instance on a single machine suitable for hosting a workshop…
Apr 16, 2016

Use your own Python installation (kernel) in Jupyterhub

python
jupyterhub
Updated February 2017
Oct 5, 2015

IPython/Jupyter notebook setup on NERSC Edison

jupyter
python
hpc
nersc
This tutorial explains the setup to run an IPython Notebook on a computing node on the…
Sep 24, 2015

IPython/Jupyter notebook setup on SDSC Comet

python
hpc
jupyter
This tutorial explains the setup to run…
Sep 17, 2015

Run Jupyterhub on a Supercomputer

python
jupyterhub
hpc
Summary: I developed a plugin for Jupyterhub: RemoteSpawner, it has a proof-of-concept interface with the Supercomputer Gordon at UC San Diego to…
Apr 2, 2015

Accelerate groupby operation on pixels with Numba

python
astrophysics
Download the original IPython notebook
Mar 24, 2015

Software Carpentry setup for Chromebook

linux
python
In this post I’ll provide instructions on how to install the main requirements of a Software Carpentry workshop on a Chromebook. Bash, git, IPython notebook and R.
Feb 10, 2015

Zero based indexing

python
For Europeans zero based indexing feels reasonable if we think of floors in a house, the lowest floor is ground floor, then 1st floor and so on.
Oct 22, 2014

Write unit tests as cells of IPython notebooks

python
jupyter
Plugin for py.test to write unit tests as cells in IPython notebooks:
Sep 30, 2014

How to perform code review for scientific software

github
git
openscience
Code review is the formal process where a programmer inspects in detail a piece of…
Aug 28, 2014

Create a Github account for your research group with free private repositories

github
git
openscience
See the updated version at https://zonca.github.io/2019/08/github-for-research-groups.html
Aug 19, 2014

Thoughts on a career as a computational scientist

hpc
Recently I’ve been asked what are the prospects of a wannabe computational scientist, both in terms of training and in terms of job opportunities.
Jun 5, 2014

Machine learning at scale with Python

python
hpc
My talk for the San Diego Data Science meetup: http://www.meetup.com/San-Diego-Data-Science-R-Users-Group/events/170967362/
Mar 20, 2014

Python on Gordon

hpc
python
Gordon has already a python environment setup which can be activated by loading the python module:
Mar 20, 2014

Build Software Carpentry lessons with Pelican

python
openscience
Software Carpentry offers bootcamps for scientist to teach basic programming skills. All the material, mainly about bash, git, Python and R is available on Github under…
Feb 26, 2014

openproceedings - Github/FigShare based publishing platform for conference proceedings

python
openscience
Github provides a great interface for gathering, peer reviewing and accepting papers for conference proceedings, the…
Feb 13, 2014

wget file from google drive

linux
Sometimes it is useful, even more if you have a chromebook, to upload a file to Google Drive and then use wget to retrieve it from a server remotely.
Jan 31, 2014

Run IPython Notebook on a HPC Cluster via PBS

python
hpc
The IPython notebook is a great tool for data exploration and visualization. It is suitable in particular for analyzing a large amount of data remotely on a computing node of a HPC cluster and…
Dec 18, 2013

Joining San Diego Supercomputer Center

TL;DR Left UCSB…
Dec 10, 2013

Published paper on Destriping Cosmic Microwave Background Polarimeter data

python
cosmology
TL;DR version:
Nov 20, 2013

Jiffylab multiuser IPython notebooks

python
jiffylab is a very interesting project by Preston Holmes to provide sandboxed IPython notebooks instances on a server using docker. There are several user cases, for example:
Oct 14, 2013

How to log exceptions in Python

python
Sometimes it is useful to just catch any exception, write details to a log file and continue execution.
Oct 1, 2013

Google Plus comments plugin for Pelican

python
github
There has been recently several discussions about whether comments are any useful on blogs I think it is important to find better ways to connect blogs to social networks. In my opinion the most suitable social network for this is…
Sep 27, 2013

How to automatically build your Pelican blog and publish it to Github Pages

python
git
Something I like a lot about Jekyll, the Github static blog generator, is that you just push commits to your repository and Github takes care of re-building…
Sep 26, 2013

clviewer, interactive plot of CMB spectra

python
astrophysics
Today it was HackDay at .Astronomy, so I felt compelled to hack something around myself, creating something I have been thinking for a while after my previous work on Interactive CMB power spectra in…
Sep 17, 2013

Planck CMB map at high resolution

python
astrophysics
healpy
Pro…
Sep 10, 2013

Run Hadoop Python jobs on Amazon with MrJob

git
python
MrJob makes the implementation of the reducer particularly simple. Using hadoop-streaming directly, we…
Sep 2, 2013

Interactive figures in the browser - CMB Power Spectra

astrophysics
healpy
For a long time I’ve been curious about trying out d3.js , the javascript plotting library which is becoming the standard for interactive plotting in the browser.
Aug 30, 2013

Planck CTP angular power spectrum ell binning

healpy
Planck released a binning of the angular power spectrum in the Explanatory supplement,
unfortunately the file is in PDF format, non easily machine-readable:

http://www.sciops.esa.int/wikiSI/planckpla/index.php?title=Frequency_maps_angular_power_spectra&instance=Pla…
Aug 20, 2013

HEALPix map of the Earth using healpy

healpy
HEALPix maps can also be used to create equal-area pixelized maps of the Earth, RGB colors are not supported in healpy, so we need to convert the image to colorscale.
The best user case is for using spherical harmonic transforms, e.g. apply a smoothing filter, in this case HEALPix/healpy tools are really…
Aug 8, 2013

Export google analytics data via API with Python

python
Fun weekend hacking project: export google analytics data using the google APIs.

Clone the latest version of the API client from:

https://co…
Aug 4, 2013

Processing sources in Planck maps with Hadoop and Python

hpc
python
The purpose of this post is to investigate…
Jul 15, 2013

How to use the IPython notebook on a small computing cluster

hpc
python
The IPython notebook is a powerful and easy to use interface for using Python and…
Jun 22, 2013

IPython parallell setup on Carver at NERSC

hpc
python
IPython parallel is one of the easiest ways to spawn several Python sessions on a Supercomputing cluster and process jobs in parallel.

On Carver, the basic setup is running a controller on the login node, and submit…
Apr 11, 2013

Simple Mixin usage in python

python
One situation where Mixins are useful in Python is when you need to modify  a method of similar classes that you are importing from a package.
Apr 8, 2013

Noise in spectra and map domain

healpy
astrophysics
\(C_\ell\) of the variance map is just the variance map multiplied by the pixel area divided by the integration time.

\[C_\ell = \Omega_{\rm pix} \langle \frac{\sigma^2}{\tau} \rangle = \Omega_{\rm…
Apr 8, 2013

Basic fork/pull git workflow

git
Typical simple workflow for a (github) repository with few users.
Apr 6, 2013

Interactive 3D plot of a sky map

python
astrophysics
Mar 12, 2013

How to cite HDF5 in bibtex

here the bibtex entry:


reference:
http://www.hdfgroup.org/HDF5-FAQ.html#gcite
Feb 27, 2013

Compile healpix C++ to javascript

linux
Compile C++ -> LLVM with clang

Convert LLVM -> Javascript:
https://github.com/kripken/emscripten/wiki/Tutorial
Jan 28, 2013

Elliptic beams, FWHM and ellipticity

astrophysics
The relationship between the Full Width Half Max, FWHM (min, max, and average) and the
ellipticity is:

FWHM = sqrt(FWHM_min * FWHM_max)
e = FWHM_max/FWHM_min

Jan 18, 2013

Ubuntu PPA for HEALPix and healpy

healpy
linux

Dec 17, 2012

Butterworth filter with Python

python
Using IPython notebook of course:

http://nbviewer.ipython.org/3843014/
Oct 6, 2012

IPython.parallel for Planck data analysis at NERSC

python
Planck is a Space mission for high precision measurements of the Cosmic Microwave Background (CMB), data are received as timestreams of output voltages from…
Sep 27, 2012

homepage on about.me

moved my homepage to about.me:

http://about.me/andreazonca

it is quite nice, and…
Sep 26, 2012

doctests and unittests happiness 2

python
Aug 16, 2012

compile python module with mpi support

CC=mpicc LDSHARED=“mpicc -shared” python setup.py build_ext -i
Jul 6, 2012

some python resources

python
python tutorial:
http://docs.python.org/tutorial/
numpy tutorial [arrays]:
http://www.scipy.org/Tentative_NumPy_Tutorial
plotting tutorial:
http://matplotlib.sourceforge.net/u…
Nov 1, 2011

cfitsio wrapper in python

python
After several issues with pyfits, and tired of it being so overengineered, I’ve wrote my own fits I/O package in python, wrapping the C library cfitsio with ctypes.

Pretty easy, first version completely developed…
Jun 21, 2011

unit testing happiness

python
Jun 21, 2011

Pink noise (1/f noise) simulations in numpy

python
https://gist.github.com/979729

May 18, 2011

Vim regular expressions

linux
very good reference of the usage of regular expressions in VIM:

http://www.softpanorama.org/Editors/Vimorama/vim_regular_expressions.shtml
Apr 29, 2011

set python logging level

python
often using logging.basicConfig is useless because if the logging module is already configured upfront by one of the imported libraries this is ignored.

The solution is to set the level…
Apr 13, 2011

pyfits memory leak in new_table

python
astrophysics
I found a memory leakage issue in pyfits.new_table, data were NOT deleted when the table was deleted, I prepared a test on github, using objgraph , which shows…
Mar 28, 2011

ipython and PyTrilinos

python
Feb 16, 2011

git make local branch tracking origin

git
git branch –set-upstream master origin/master

you obtain the same result as initial cloning
Feb 2, 2011

memory map npy files

python
Mem-map the stored array, and then access the second row directly from disk:

X = np.load(‘/tmp/123.npy’, mmap_mode=‘r’)
Jan 7, 2011

force local install of python module

python
python setup.py install –prefix FOLDER


creates lib/python2.6/site-packages, to force a local install you should use:

python setup.py install –install-lib FOLDER
Dec 3, 2010

gnome alt f2 popup launcher

linux

gnome-panel-control –run-dialog
Aug 31, 2010

switch to interactive backend with ipython -pylab

python
objective:
Aug 21, 2010

numpy dtypes and fits keywords

python
bool: ‘L’,
uint8: ‘B’,
int16: ‘I’,
int32: ‘J’,
int64: ‘K’,
float32: ‘E’,
float64: ‘D’,
complex64: ‘C’,
complex128: ‘M’
Aug 4, 2010

count hits with numpy

python
I have an array where I record hits
a=np.zeros(5)
and an array with the indices of the hits, for example I have 2 hits on index 2
hits=np.array([2,2])
so I want to increase…
Jul 23, 2010

change column name in a fits with pyfits

python
no way to change it manipulating the dtype of the data array.
a=pyfits.open(‘filename.fits’)
a[1].header.update(‘TTYPE1’,‘newname’)
you need to change the header, using the update method of the right TTYPE and then…
Jun 30, 2010

healpix coordinates

astrophysics
Healpix considers latitude theta from 0 on north pole to pi south pole,
so the conversion is:
theta = pi/2 - latitude
longitude and phi…
Jun 23, 2010

parallel computing the python way

python
forget MPI:
http://showmedo.com/videotutorials/series?name=N49qyIFOh
Jun 21, 2010

quaternions for python

python
the situation is pretty problematic, I hope someday scipy will add a python package for rotating and interpolating quaternions, up to now:
Jun 21, 2010

change permission recursively to folders only

linux
find . -type d -exec chmod 777 {} ;
Mar 23, 2010

aptitude search ‘and’

linux
this is really something really annoying about aptitude, if you run:
aptitude search linux headers
it will make an ‘or’ search…to perform a…
Mar 16, 2010

using numpy dtype with loadtxt

linux
python
Let’s say you want to read a text file like this:


Mar 3, 2010

Stop ipcluster from a script

python
Ipcluster is easy to start but not trivial to stop from a script, after having finished the processing, here’s the solution:
from IPython.kernel import client
mec =…
Feb 19, 2010

Correlation

Expectation value or first moment of a random variable is the probability weighted sum of the possible values (weighted mean).
Expectat…
Jan 28, 2010

execute bash script remotely with ssh

linux
a bash script launched remotely via ssh does not load the environment, if this is an issue it is necessary to specify –login when calling bash:

ssh user@remoteserver.com ‘bash –login life_om/cronodproc’ | mail…
Jan 7, 2010

lock pin hold a package using apt on ubuntu

linux
set hold:
echo packagename hold | dpkg –set-selections

check, should be hi :
dpkg -l packagename

unset hold:
echo packagename install | dpkg –set-selections
Jan 7, 2010

load arrays from a text file with numpy

python
space separated text file with 5 arrays in columns:

[sourcecode language=“python”]
ods,rings,gains,offsets,rparams = np.loadtxt(filename,unpack=True)
[/sourcecode]

quite…
Jan 5, 2010

Latest Maxima and WxMaxima for Ubuntu Karmic

linux
http://zeus.nyf.hu/~blahota/maxima/karmic/

on maxima mailing…
Dec 15, 2009

number of files in a folder and subfolders

linux
folders are not counted
find . -type f | wc -l
Dec 10, 2009

forcefully unmount a disk partition

linux
check which processes are accessing a partition:

[sourcecode language=“python”]lsof | grep ‘/opt’[/sourcecode]

kill all the processes accessing the…
Sep 17, 2008

netcat, quickly send binaries through network

linux
just start nc in server mode on localhost:

[sourcecode language=‘python’] nc -l -p 3333 [/sourcecode]

send a string to localhost on port 3333:

[sourcecode language=‘python’] echo “hello world” | nc…
Apr 29, 2008

Decibels, dB and dBm, in terms of Power and Amplitude

astrophysics
It’s not difficult, just always having some doubts…
Mar 29, 2008

Relation between Power density and temperature in an antenna

astrophysics
Considering an antenna placed inside a blackbody enclosure at temperature T, the power received per unit bandwidth is:
\(latex \omega = kT\)

…
Mar 28, 2008

Producing PDF from XML files

linux
I need to produce formatted pdf from XML data input file.
The more standard way looks like to use XSL stylesheets.
Associating a XSL sheet to an XML file permits most browsers to render them directly…
Mar 28, 2008

vim costumization

linux
it is about perl but it suggests very useful tricks for programming with vim
http://mamchenkov.net/wordpress/2004/05/10/vim-for-perl-developers/
Oct 17, 2006

using gnu find

linux
list all the directories excluding “.”:
Oct 3, 2006

beginners bash guide

linux
great guide with many examples:

http://tille.xalasys.com/training/bash/
Oct 3, 2006

tar quickref

linux
compress: tar cvzf foo.tgz .cc .h
check inside: tar tzf foo.tgz | grep file.txt
extract: tar xvzf foo.tgz
extract 1 file only: tar xvzf foo.tgz path/to/file.txt
Sep 25, 2006

software carpentry

linux
basic software for scientists and engineers:
http://www.swc.scipy.org/
Sep 25, 2006

Software libero per il trattamento di dati scientifici

linux
nella ricerca del miglior ambiente per analisi di dati scientifici da leggere questi articoli:

http://www.pluto.it/files/journal/pj0501/swlibero-scie1.html

http://www.pluto.it/files/journal/pj0504/swli…
Sep 22, 2006

command line processing

linux
Very useful summary of many linux command line processing tools (great perl onliners)

http://grad.physics.sunysb.edu/~leckey/personal/forget/
Sep 22, 2006

awk made easy

linux
awk ’/REGEX/ {print NR “ $9” $4”_“$5 ;}’ file.txt
supports extended REGEX like perl ( e.g. [:blank:] Space or tab characters )
NR is line number
NF Number of fields
$n is the column to be printed, $0 is the…
Sep 22, 2006

pillole di astrofisica

astrophysics
curiosita’ ben spiegate da annibale d’ercole, interessante l’idea di avere un livello base e un livello avanzato
http://www.bo.astro.it/sait/spigolature/spigostart.html
Sep 20, 2006
No matching items