Server Setup¶

This page documents the installation and configuration of the rucio server and master database at:

    rucio.ligo.caltech.edu

With operating system:

    Scientific Linux release 7.6 (Nitrogen)

Install yum dependencies¶

Note: a Dockerfile with the recipe to install the server dependencies can be found here.

Update:

yum update

Enable additional repositories:

rpm -ivh http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm 
rpm -ivh http://repo.opensciencegrid.org/osg/3.4/osg-3.4-el7-release-latest.rpm
rpm -Uvh https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-redhat10-10-2.noarch.rpm

Update repository metadata

yum clean all
yum makecache

Yum-based dependencies and useful tools:

yum install -y git \
    vim \
    bzip2 \
    kernel-devel \
    gcc \
    make \
    perl \
    python-devel \
    python-psycopg2 \
    osg-pki-tools \
    fetch-crl \
    memcached \
    postgresql10-server \
    postgresql10 \
    gfal2 \
    gfal2-python \
    gfal2-util \
    gfal2-all \
    globus-proxy-utils \
    voms-clients-cpp \
    httpd-devel \
    openssl-devel \
    mod_wsgi \
    mod_auth_kerb \
    gridsite

pip dependencies¶

Do not install pip from RPM! Instead, get it from:

curl -O https://bootstrap.pypa.io/get-pip.py
python get-pip.py

python2-pip demands to install python-ipaddress which later conflicts with the pip installation of rucio itself.

Install all possible pip-packages required by rucio:

curl -O https://raw.githubusercontent.com/astroclark/rucio/ligo_lfn2pfn/requirements.readthedocs.txt
pip install -r requirements.readthedocs.txt supervisor

Finally, install ligo branch of rucio:

pip install git+https://github.com/astroclark/rucio.git@ligo_lfn2pfn

Set up logging directories:

mkdir /var/log/rucio /var/log/rucio/trace
chmod -R 777 /var/log/rucio

Make RESTful scripts executable:

chmod +x /usr/lib/python2.7/site-packages/rucio/web/rest/webpy/v1/*.py

Additional directory structures and contents:

ln -sf /usr/rucio /opt/rucio
mkdir /opt/rucio/lib
ln -sf /usr/lib/python2.7/site-packages/rucio /opt/rucio/lib/rucio

Clone the source repository for some supplementary files:

git clone git@github.com:astroclark/rucio.git

Copy the wsgi aliases from the source repository into place:

cp ~james.clark/rucio/etc/web/aliases-py27.conf /opt/rucio/etc/web

And edit /opt/rucio/etc/web/aliases-py27.conf to point to the correct locations. E.g.,:

/opt/rucio/.venv/lib/python2.7/site-packages/rucio/web/rest/account.py

Should be:

/opt/rucio/lib/rucio/web/rest/account.py

Service Configuration¶

Enable/start services:

systemctl enable memcached
systemctl start memcached
systemctl enable fetch-crl-boot
systemctl start fetch-crl-boot
systemctl enable fetch-crl-cron
systemctl start fetch-crl-cron

Apache Configuration¶

This step assumes the host is called rucio.ligo.caltech.edu with X509 certificate bundle:

ls /etc/httpd/x509-certs/
rucio.ligo.caltech.edu.cert  rucio.ligo.caltech.edu.key  rucio.ligo.caltech.edu.pem

Download the Apache configuration into place:

curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/rucio.conf -o /etc/httpd/conf.d/rucio.conf

Alternatively: download rucio.conf manually and copy to /etc/httpd/conf.d/rucio.conf

Make sure the following lines are commented out in /etc/httpd/conf/httpd.conf

#<Directory />
#    AllowOverride none
#    Require all denied
#</Directory>

Set SELinux to permissive:

setenforce permissive

Start up apache:

systemctl enable httpd
systemctl start httpd

PostgreSQL Configuration¶

Steps:

Initialise database
Open database authentication method for initial database creation
Create user and database for rucio
Finalise database access authentication

Initialise data directory:

/usr/pgsql-10/bin/postgresql-10-setup initdb

Edit /var/lib/pgsql/10/data/pg_hba.conf to modify database authentication method for initial database creation. Change /var/lib/pgsql/10/data/pg_hba.conf from:

# TYPE  DATABASE  USER  ADDRESS  METHOD
local   all       all            peer

to:

local   all       all            trust

Can be done by:

curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/pg_hba_trust.conf -o /var/lib/pgsql/10/data/pg_hba.conf

Start database:

systemctl restart postgresql-10

Create user and database for rucio:

psql -U postgres

postgres=# create role rucio password 'rucio' superuser inherit login;
CREATE ROLE
postgres=# create database rucio owner rucio encoding 'utf8';
CREATE DATABASE
postgres=# grant all on database rucio to rucio;
GRANT

Set authentication method to md5 and add a new host:

change method for everything to md5
Add new line: host all all 0.0.0.0/0 md5

E.g., download /var/lib/pgsql/10/data/pg_hba.conf in place:

curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/pg_hba_md5.conf -o /var/lib/pgsql/10/data/pg_hba.conf

Set addresses and port to listen to in /var/lib/pgsql/10/data/postgresql.conf:

listen_addresses = '*'
port = 5432 Equivalently:

curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/postgresql.conf -o /var/lib/pgsql/10/data/postgresql.conf

Now enable and start or restart the postgreql server:

systemctl enable postgresql-10
systemctl start postgresql-10

Rucio Configuration¶

Install rucio configuration files from gwrucio repository:

Add rucio.cfg to /opt/rucio/etc/
Add alembic.ini to /opt/rucio/ E.g.,:

curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/rucio.cfg -o /opt/rucio/etc/rucio.cfg
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/alembic.ini -o /opt/rucio/alembic.ini

Important Note: when first configuring the rucio server, ensure that the rucio.cfg is set to use userpass authentication with the original ddmlab user:

auth_type = userpass
username = ddmlab
password = secret

Test the RESTful interface¶

curl -k -X GET https://rucio.ligo.caltech.edu:443/ping
{"version": "1.13.1-2549-g78ef266-dev1555615471"}

Initialise the Rucio database¶

Manually copy the SQLAlchemy versions directory from the source repository:

cp -r ~james.clark/rucio/lib/rucio/db/sqla/migrate_repo/versions /opt/rucio/lib/rucio/db/sqla/migrate_repo/

Initialise the rucio database:

[root@rucio rucio]# cd /opt/rucio
[root@rucio rucio]# python tools/bootstrap.py
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 3345511706b8

Add Access Control List¶

Add a grid access control list file in /opt/rucio/etc/gacl:

curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/gacl -o /opt/rucio/etc/gacl

Root Account Setup¶

Full details of account setup can be found in rucio administration

These instructions assume an IGTF X509 host certificate has been installed on the server.

Creating the database adds a root account with default username/password. Add an X509 identity and then change that password ASAP. Add the host certificate X509 identity to the root account:

openssl x509 -in /etc/grid-security/hostcert.pem -noout -subject
subject= /DC=org/DC=incommon/C=US/ST=CA/L=Pasadena/O=California Institute of Technology/OU=Laser Interferometer Gravitational-Wave Observatory/CN=rucio.ligo.caltech.edu
[root@rucio rucio]# rucio-admin identity add --account root --type X509 --id "/DC=org/DC=incommon/C=US/ST=CA/L=Pasadena/O=California Institute of Technology/OU=Laser Interferometer Gravitational-Wave Observatory/CN=rucio.ligo.caltech.edu" --email james.clark@ligo.org
Added new identity to account: /DC=org/DC=incommon/C=US/ST=CA/L=Pasadena/O=California Institute of Technology/OU=Laser Interferometer Gravitational-Wave Observatory/CN=rucio.ligo.caltech.edu-root

Configure the client on the server to use the server’s host x509 certificate. Edit /opt/rucio/etc/rucio.cfg:

[client]
;auth_type = userpass
username = ddmlab
password = secret
auth_type = x509
client_cert = /etc/grid-security/hostcert.pem
client_key = /etc/grid-security/hostkey.pem

Delete default identities

rucio-admin identity delete --account root --type GSS --id "ddmlab@CERN.CH"
rucio-admin identity delete --account root --type X509 --id "/C=CH/ST=Geneva/O=CERN/OU=PH-ADP-CO/CN=DDMLAB Client Certificate/emailAddress=ph-adp-ddm-lab@cern.ch"
rucio-admin identity delete --account root --type USERPASS --id "ddmlab"
rucio-admin identity delete --account root --type SSH --id "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq5LySllrQFpPL614sulXQ7wnIr1aGhGtl8b+HCB/0FhMSMTHwSjX78UbfqEorZV16rXrWPgUpvcbp2hqctw6eCbxwqcgu3uGWaeS5A0iWRw7oXUh6ydnVy89zGzX1FJFFDZ+AgiZ3ytp55tg1bjqqhK1OSC0pJxdNe878TRVVo5MLI0S/rZY2UovCSGFaQG2iLj14wz/YqI7NFMUuJFR4e6xmNsOP7fCZ4bGMsmnhR0GmY0dWYTupNiP5WdYXAfKExlnvFLTlDI5Mgh4Z11NraQ8pv4YE1woolYpqOc/IMMBBXFniTT4tC7cgikxWb9ZmFe+r4t6yCDpX4IL8L5GOQ== ddmlab"

You may also add an X509 identity to the root account for remote administration. E.g.:

rucio-admin identity add --account root --type X509 \
    --id "/DC=org/DC=XXXXXXX/C=XX/O=XXXX/CN=Albert Einstein albert.einstein@ligo.org" \
    --email albert.einstein@ligo.org

In your local client configuration, change to x509_proxy authentication in the ${RUCIO_HOME}/etc/rucio.cfg

Daemon Configuration (SupervisorD)¶

Rucio daemons are controlled with supervisord. These directions assume supervisor has been installed via pip (yum installs supervisor dependencies which conflict with other pip packages).

Copy the supervisor configuration file into place:

curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/supervisord.conf -o /etc/supervisord.conf

Edit inet server and username/password (if you want the webui). Set an appropriate username/password in::

[inet_http_server]         ; inet (TCP) server disabled by default
port=rucio.ligo.caltech.edu:9001        ; (ip_address:port specifier, *:port for all iface)
username=user              ; (default is no username (open server))
password=123               ; (default is no password (open server))

Create log and run directories:

mkdir /var/log/supervisor
mkdir /var/run/supervisor

Add the rucio daemon configuration file:

mkdir /etc/supervisord.d
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/rucio-daemons.ini -o /etc/supervisord.d/rucio-daemons.ini

Check supervisord runs ok by launching in foreground:

supervisord --nodaemon

Starting supervisord at boot¶

The pip-installed supervisor does not provide init scripts to run supervisor under systemd. However, the supervisor documentation refers to a github repository where init scripts are provided. Download and install the CentOS script:

curl https://raw.githubusercontent.com/Supervisor/initscripts/master/centos-systemd-etcs -o  /usr/lib/systemd/system/supervisord.service

We should now be able to enable and start supervisord through systemctl

systemctl enable supervisord
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
systemctl start supervisord

Proxy Delegation For FTS¶

Third party transfers are performed via an FTS server. To authenticate 3rd party transfers:

Map the rucio server host identity to an account at each end point.
Create an X509 proxy certificate from the rucio server host certificate.
Delegate the proxy to the FTS server for authentication at each end point.

Note: the host certificate used here must be from a CA recognised by all sites in the rucio distribution network. We have found that LIGO robot certificates are not recognised by EGI sites and we need to use an IGTF-issued certificate, with a subject which follows the IGTF specifications (e.g., no postcode!).

Valid proxy creation and delegation is performed via a cronjob which executes a simple script with voms-proxy-init and fts-rest-delegate.

The script can be found here and downloaded / installed with

    curl https://git.ligo.org/james-clark/gwrucio/raw/master/bin/fts_proxy -o /opt/rucio/tools/fts_proxy

Edit the crontab to refresh the proxy and delegation at 6am and 6pm:

0 6,18 * * * /bin/bash /opt/rucio/tools/fts_proxy > /dev/null 2>&1

Support for EGI Sites¶

To authenticate with EGI sites, we need to register our certificate with the Virgo VOMs:

Create a pkcs12 certificate bundle for import into a browser:

openssl pkcs12 -export -out rucio-bundle.pfx -inkey /etc/grid-security/hostkey.pem -in /etc/grid-security/hostcert.pem

Download and import to a browser and navigate to: http://www.italiangrid.it/users/join_vo: request to join the Virgo VO.

Finally, add the following files to /etc/vomses:

mkdir /etc/vomses
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/virgo-voms.cnaf.infn.it -o /etc/vomses/virgo-voms.cnaf.infn.it
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/virgo-voms-01.pd.infn.it -o /etc/vomses/virgo-voms-01.pd.infn.it