CernVM File System (CVMFS)¶

What is CVMFS?¶

The CernVM File System ("CernVM-FS" or "CVMFS") is a tool that allows for efficient global distribution of software and data that does not change frequently. Its name indicates its origins for use by virtual machines in use by the high energy physics community, however, it has wider applicability and usage. It caches files to disk so that, after the initial download, file access for the client is speedy.

Within IGWN, CVMFS is being used to distribute both instrument data ("frame files") and analysis software for use at the shared computing centres and by distributed workflows. You may elect to install CVMFS on your workstation for ease of access to recent data and software.

Platform compatibility

CVMFS is fully developed and tested for Linux, and works for the most part on macOS (with the explicit exception of X.509 authentication). There is no support for native Windows use, however CVMFS works well on the Windows Subsystem for Linux (WSL) version 2; users of Windows should follow the Linux instructions relevant for their WSL distribution of choice, noting any Extra steps for Windows users along the way.

Full documentation of CVMFS and its inner workings can be browsed online at https://cvmfs.readthedocs.io/.

Installing the CVMFS client¶

Install the client using the instructions specific to your platform and location:

Debian/UbuntumacOSRHEL | EuropeRHEL | Other

Configure the necessary extra Apt repository for CVMFS using these instructions

Install CVMFS:

sudo apt-get update
sudo apt-get install cvmfs

Configure the necessary extra Apt repository for cvmfs-contrib using these instructions.
Install the cvmfs-config-<domain> package that best matches your location. <domain> should be
- egi - for users in Europe
- osg - for everyone else
For example:
```
apt-get update
apt-get install cvmfs-config-osg
```

Install macFuse by downloading the .dmg installer from the latest release.

You may be asked to reboot your machine after this step.
Install the CVMFS client using these instructions.
Reboot your machine to finish installing CVMFS.

Manually configure the cvmfs-config repo that best matches your location:

EuropeOther

Users in Europe should configure their client to use the EGI configuration:

git clone https://github.com/cvmfs-contrib/cvmfs-config-egi.git /tmp/cvmfs-config-egi
sudo /bin/bash <<EOF
mkdir -p \
  /etc/cvmfs/default.d \
  /etc/cvmfs/config.d \
  /etc/cvmfs/keys/egi.eu
install -m 444 /tmp/cvmfs-config-egi/60-egi.conf /etc/cvmfs/default.d
install -m 444 /tmp/cvmfs-config-egi/config-egi.egi.eu.conf /etc/cvmfs/config.d
install -m 444 /tmp/cvmfs-config-egi/egi.eu.pub /etc/cvmfs/keys/egi.eu
EOF
rm -rf /tmp/cvmfs-config-osg

Then manually mount the config repo:

sudo mkdir -p /cvmfs/config-egi.egi.eu
sudo mount -t cvmfs config-egi.egi.eu /cvmfs/config-egi.egi.eu

Users outside of Europe should configure their client to use the OSG configuration:

git clone https://github.com/opensciencegrid/cvmfs-config-osg.git /tmp/cvmfs-config-osg
sudo /bin/bash <<EOF
mkdir -p \
  /etc/cvmfs/default.d \
  /etc/cvmfs/config.d \
  /etc/cvmfs/keys/opensciencegrid.org
install -m 444 /tmp/cvmfs-config-osg/60-osg.conf /etc/cvmfs/default.d
install -m 444 /tmp/cvmfs-config-osg/config-osg.opensciencegrid.org.conf /etc/cvmfs/config.d
install -m 444 /tmp/cvmfs-config-osg/opensciencegrid.org.pub /etc/cvmfs/keys/opensciencegrid.org
EOF
rm -rf /tmp/cvmfs-config-osg

Then manually mount the config repo:

sudo mkdir -p /cvmfs/config-osg.opensciencegrid.org
sudo mount -t cvmfs config-osg.opensciencegrid.org /cvmfs/config-osg.opensciencegrid.org

Configure the necessary extra Yum repository for CVMFS using these instructions
Install CVMFS:
```
dnf -y install cvmfs
```
Configure the necessary extra Yum repository for cvmfs-contrib using these instructions.
Install the cvmfs-config-egi configuration package
```
dnf -y install cvmfs-config-osg
```

Install and configure the OSG repositories using these instructions.
Install the relevant packages
```
yum -y install cvmfs cvmfs-config-osg
```

Next, configure the client (on all platforms) using the following short set of steps:
1. Run a basic setup:
```
sudo cvmfs_config setup
```
  Extra step for WSL users
  
  If you are running a Linux distribution using the Windows Subsystem for Linux, the above cvmfs_config setup call will do nothing, but will alert you to instead run this:
```
sudo cvmfs_config wsl2_start
```
2. Create a default.local configuration for CVMFS that references the repositories you care about:
```
sudo bash -c 'cat > /etc/cvmfs/default.local' << EOF
CVMFS_HTTP_PROXY=DIRECT
CVMFS_QUOTA_LIMIT=20000
CVMFS_REPOSITORIES=<my-cvmfs-repo>
EOF
```
  Configuring CVMFS_HTTP_PROXY
  
  The CVMFS_HTTP_PROXY variable defines what proxies (if any) CVMFS should use when attempting to download data over HTTP.
  
  The value DIRECT is a special case to avoid using a proxy altogether. For full details of how to configure this best for your client, see
  
  https://cvmfs.readthedocs.io/en/stable/cpt-configure.html#proxy-lists
  Configuring CVMFS_REPOSITORIES
  
  The CVMFS_REPOSITORIES variable should be defined as a comma-separated list of repository names, e.g.
```
CVMFS_REPOSITORIES=gwosc.osgstorage.org
```
  See the Useful CVMFS repositories section below for details of useful repositories and how to configure them.
3. Reload the configuration and verify the file system:
```
cvmfs_config probe
```

Mounting a CVMFS repository¶

Repositories can be mounted manually by adding their name to the CVMFS_REPOSITORIES variable in /etc/cvmfs/default.local, e.g:

CVMFS_REPOSITORIES="software.igwn.org"

See below for details about mouting repositories on your platform.

After attempting to mount a repository, you should probe the repository to assert that it works:

cvmfs_config probe software.igwn.org

Mounting a CVMFS repository on Linux¶

On Linux systems that use the EGI or OSG CVMFS configuration repositories, it is unlikely that you will need to manually mount the repository, or even add it to the CVMFS_REPOSITORIES variable.

autofs and cvmfs2 will automatically mount the repository as soon as any user attempts to access it.

Mounting a CVMFS repository on macOS¶

On macOS there is no autofs service to automatically mount CVMFS repositories, so they must be manually mounted as follows:

sudo mkdir -p /cvmfs/<repo-name>
sudo mount -t cvmfs <repo-name> /cvmfs/<repo-name>

For example:

sudo mkdir -p /cvmfs/software.igwn.org
sudo mount -t cvmfs software.igwn.org /cvmfs/software.igwn.org

Authorised access¶

Some CVMFS repositories are configured to require users to present authorisation credentials to gain access.

To configure CVMFS to enable access to restricted repositories:

Install cvmfs-x509-helper to support authorised access via SciTokens:

AptdnfYum

apt-get -y install cvmfs-x509-helper

dnf -y install cvmfs-x509-helper

yum -y install cvmfs-x509-helper

See How to generate a SciToken for details on how to generate a SciToken based on the (digital) identity you hold. Short version:
Generating a SciToken
```
conda install -c conda-forge htgettoken
htgettoken --scopes read:/kagra read:/ligo read:/virgo
```

Each repository below lists the relevant scope required for token-based authorisation.

Useful CVMFS repositories:¶

This section describes a number of useful CVMFS repositories, and how to configure them.

All of the repositories below should be automatically mounted on systems that have the CVMFS client configured using the EGI or OSG configuration repositories.

OASIS (`oasis.opensciencegrid.org`)¶

Name: oasis.opensciencegrid.org
Token scopes: -

OASIS is the OSG Application Software Installation Service, the recommended way to distribute software on the Open Science Grid.

IGWN Software (`software.igwn.org`)¶

Name: software.igwn.org
Token scopes: -

IGWN distributes its software in a dedicated CVMFS repository called software.igwn.org. This repo is the host for the IGWN Conda Distribution.

IGWN CVMFS software is built for linux

Most of the software available on software.igwn.org (including the IGWN Conda Distribution) is compiled for Linux use, so is unlikely to work on macOS, however, there are data and other files distributed in the OASIS repo that may be used on that platform.

You should now be able to see the IGWN Conda Distribution, as well as a few other pieces of software from IGWN:

$ /cvmfs/software.igwn.org/conda/condabin/conda --version
conda 4.9.2

Older IGWN Conda Distribution environments require OASIS

To use IGWN Conda Distribution environments older than 20230523, you will need to unsure that the oasis.opensciencegrid.org repository is also configured.

Public and Proprietary IGWN data¶

Previously it was possible to access public Gravitational Wave Open Science Center data via CVMFS, and for authorized users to access other proprietary data sets with appropriate credentials. However, as of 23 August 2025 the ability to access these data sets through CVMFS has been retired.

These data should now be accessed only through OSDF. There is also useful information in this guide on accessing data from HTCondor workflows.

OSG Singularity images (`singularity.opensciencegrid.org`)¶

Name: singularity.opensciencegrid.org
Token scopes: -

IGWN (and other groups) publish Singularity container images via CVMFS for distributed available, including to HTC grid jobs.

Publishing images to CVMFS

See Publishing Singularity Images To CVMFS for details on how to publish Docker images to CVMFS for distribution in singularity.opensciencegrid.org.