The Open Science Grid and IGWN-grid

The IGWN Grid is now the recommended way to submit bilby_pipe analyses using LVK resources. The IGWN Grid allows access to all dedicated resources (i.e. CIT, LHO), but also resources such as the Open Science Grid.

To run jobs through the OSG, login to on of the IGWN grid submit nodes ({ldas-osg,ldas-osg2,ldas-osg3}.ligo.caltech.edu at CIT), see here for more details

ssh albert.einstein@ldas-osg.ligo.caltech.edu

Then submit usual bilby_pipe jobs, but with the flag

osg = True

in your configuration (ini) files.

When running on the IGWN-grid, the software you run needs to be available on the compute nodes. A detailed guide to accessing software in the IGWN-grid is given on the IGWN computing docs. bilby_pipe supports all of the methods described on that page.

CVMFS

To use the IGWN conda distribution available through cvmfs you should either submit your job using an IGWN environment, or use the conda-env flag to choose one of the IGWN environments.

containers

Containers provide an isolated, distributable, computing environment. There are multiple methods to leverage containers in the IGWN grid and there are detailed instructions for how to build and distribute images on the IGWN computing docs.

Warning

In the future we plan to make production-ready containers available, however, this is currently a work in progress.

To use a singularity image you should set the container argument to point to the container. If the container is provided as a path in the local filesystem it will be automatically transferred to the job. Additionally, you will most likely need to specify the conda-env argument to point to the appropriate environment within the container.

Note

Using containers requires the use of the HTCondor file transfer mechanism.

Standalone executables

Specifying the analysis executable is another way to leverage grid computing. If an executable is provided to point to a CVMFS-distributed environment, that environment can then be used if the remainder of the workflow uses a local environment.

When using the IGWN-grid, you can specify which site you would like your job to run on by using the desired-sites option.