# Server Setup
This page documents the installation and configuration of the rucio server and master database at:
```
    rucio.ligo.caltech.edu
```

With operating system:
```
    Scientific Linux release 7.6 (Nitrogen)
```

## Install yum dependencies
Note: a Dockerfile with the recipe to install the server dependencies can be found [here](https://git.ligo.org/james-clark/gwrucio/blob/master/server.Dockerfile).

Update:
```
yum update
```

Enable additional repositories:
```
rpm -ivh http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm 
rpm -ivh http://repo.opensciencegrid.org/osg/3.4/osg-3.4-el7-release-latest.rpm
rpm -Uvh https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-7-x86_64/pgdg-redhat10-10-2.noarch.rpm
```

Update repository metadata
```
yum clean all
yum makecache
```

Yum-based dependencies and useful tools:
```
yum install -y git \
    vim \
    bzip2 \
    kernel-devel \
    gcc \
    make \
    perl \
    python-devel \
    python-psycopg2 \
    osg-pki-tools \
    fetch-crl \
    memcached \
    postgresql10-server \
    postgresql10 \
    gfal2 \
    gfal2-python \
    gfal2-util \
    gfal2-all \
    globus-proxy-utils \
    voms-clients-cpp \
    httpd-devel \
    openssl-devel \
    mod_wsgi \
    mod_auth_kerb \
    gridsite
```

## pip dependencies
Do **not** install pip from RPM!  Instead, get it from:
```
curl -O https://bootstrap.pypa.io/get-pip.py
python get-pip.py
```
`python2-pip` demands to install `python-ipaddress` which later conflicts with
the pip installation of `rucio` itself.

Install all possible pip-packages required by rucio:
```
curl -O https://raw.githubusercontent.com/astroclark/rucio/ligo_lfn2pfn/requirements.readthedocs.txt
pip install -r requirements.readthedocs.txt supervisor
```

Finally, install ligo branch of rucio:
```
pip install git+https://github.com/astroclark/rucio.git@ligo_lfn2pfn
```

Set up logging directories:
```
mkdir /var/log/rucio /var/log/rucio/trace
chmod -R 777 /var/log/rucio
```

Make RESTful scripts executable:
```
chmod +x /usr/lib/python2.7/site-packages/rucio/web/rest/webpy/v1/*.py
```

Additional directory structures and contents:
```
ln -sf /usr/rucio /opt/rucio
mkdir /opt/rucio/lib
ln -sf /usr/lib/python2.7/site-packages/rucio /opt/rucio/lib/rucio
```

Clone the source repository for some supplementary files:
```
git clone git@github.com:astroclark/rucio.git
```

Copy the wsgi aliases from the *source* repository into place: 
```
cp ~james.clark/rucio/etc/web/aliases-py27.conf /opt/rucio/etc/web
```
And edit `/opt/rucio/etc/web/aliases-py27.conf` to point to the correct locations.  E.g.,:
```
/opt/rucio/.venv/lib/python2.7/site-packages/rucio/web/rest/account.py
```
Should be:
```
/opt/rucio/lib/rucio/web/rest/account.py
```

## Service Configuration

Enable/start services:
```
systemctl enable memcached
systemctl start memcached
systemctl enable fetch-crl-boot
systemctl start fetch-crl-boot
systemctl enable fetch-crl-cron
systemctl start fetch-crl-cron
```

### Apache Configuration
This step assumes the host is called `rucio.ligo.caltech.edu` with X509 certificate bundle:
```
ls /etc/httpd/x509-certs/
rucio.ligo.caltech.edu.cert  rucio.ligo.caltech.edu.key  rucio.ligo.caltech.edu.pem
```

Download the Apache configuration into place:
```
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/rucio.conf -o /etc/httpd/conf.d/rucio.conf
```
Alternatively: download
[`rucio.conf`](https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/rucio.conf)
manually and copy to `/etc/httpd/conf.d/rucio.conf`

Make sure the following lines are commented out in `/etc/httpd/conf/httpd.conf`
```
#<Directory />
#    AllowOverride none
#    Require all denied
#</Directory>
```

Set SELinux to permissive:
```
setenforce permissive
```

Start up apache:
```
systemctl enable httpd
systemctl start httpd
```


### PostgreSQL Configuration
Steps:
 1. Initialise database
 1. Open database authentication method for initial database creation
 1. Create user and database for rucio
 1. Finalise database access authentication

Initialise data directory:
```
/usr/pgsql-10/bin/postgresql-10-setup initdb
```

Edit `/var/lib/pgsql/10/data/pg_hba.conf` to modify database authentication
method for initial database creation.  Change
`/var/lib/pgsql/10/data/pg_hba.conf` from:
```
# TYPE  DATABASE  USER  ADDRESS  METHOD
local   all       all            peer
```
to:
```
local   all       all            trust
```
Can be done by:
```
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/pg_hba_trust.conf -o /var/lib/pgsql/10/data/pg_hba.conf
```

Start database:
```
systemctl restart postgresql-10
```

Create user and database for rucio:
```
psql -U postgres

postgres=# create role rucio password 'rucio' superuser inherit login;
CREATE ROLE
postgres=# create database rucio owner rucio encoding 'utf8';
CREATE DATABASE
postgres=# grant all on database rucio to rucio;
GRANT
```

Set authentication method to md5 and add a new host:
 * change method for *everything* to `md5`
 * Add new line: `host  all  all  0.0.0.0/0  md5`

E.g., download `/var/lib/pgsql/10/data/pg_hba.conf` in place:
```
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/pg_hba_md5.conf -o /var/lib/pgsql/10/data/pg_hba.conf
```

Set addresses and port to listen to in `/var/lib/pgsql/10/data/postgresql.conf`:
 * `listen_addresses = '*' `
 * `port = 5432`
Equivalently:
```
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/postgresql.conf -o /var/lib/pgsql/10/data/postgresql.conf
```

Now enable and start or restart the postgreql server:
```
systemctl enable postgresql-10
systemctl start postgresql-10
```

## Rucio Configuration
Install rucio configuration files from `gwrucio` repository:
 * Add [`rucio.cfg`](https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/rucio.cfg) to `/opt/rucio/etc/`
 * Add [`alembic.ini`](https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/alembic.ini) to `/opt/rucio/`
E.g.,:
```
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/rucio.cfg -o /opt/rucio/etc/rucio.cfg
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/alembic.ini -o /opt/rucio/alembic.ini
```
**Important Note**: when first configuring the rucio server, ensure that the
`rucio.cfg` is set to use `userpass` authentication with the original `ddmlab`
user:
```
auth_type = userpass
username = ddmlab
password = secret
```

### Test the RESTful interface
```
curl -k -X GET https://rucio.ligo.caltech.edu:443/ping
{"version": "1.13.1-2549-g78ef266-dev1555615471"}
```


### Initialise the Rucio database

Manually copy the SQLAlchemy versions directory from the source repository:
```
cp -r ~james.clark/rucio/lib/rucio/db/sqla/migrate_repo/versions /opt/rucio/lib/rucio/db/sqla/migrate_repo/
```

Initialise the rucio database:
```
[root@rucio rucio]# cd /opt/rucio
[root@rucio rucio]# python tools/bootstrap.py
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 3345511706b8
```

### Add Access Control List
Add a grid access control list file in `/opt/rucio/etc/gacl`:
```
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/gacl -o /opt/rucio/etc/gacl
```

### Root Account Setup
Full details of account setup can be found in [rucio administration](https://git.ligo.org/james-clark/gwrucio/wikis/rucio-administration)

These instructions assume an IGTF X509 host certificate has been installed on the server.

Creating the database adds a root account with default username/password.  Add
an X509 identity and then change that password ASAP.  Add the host certificate
X509 identity to the root account:
```
openssl x509 -in /etc/grid-security/hostcert.pem -noout -subject
subject= /DC=org/DC=incommon/C=US/ST=CA/L=Pasadena/O=California Institute of Technology/OU=Laser Interferometer Gravitational-Wave Observatory/CN=rucio.ligo.caltech.edu
[root@rucio rucio]# rucio-admin identity add --account root --type X509 --id "/DC=org/DC=incommon/C=US/ST=CA/L=Pasadena/O=California Institute of Technology/OU=Laser Interferometer Gravitational-Wave Observatory/CN=rucio.ligo.caltech.edu" --email james.clark@ligo.org
Added new identity to account: /DC=org/DC=incommon/C=US/ST=CA/L=Pasadena/O=California Institute of Technology/OU=Laser Interferometer Gravitational-Wave Observatory/CN=rucio.ligo.caltech.edu-root
```

Configure the client *on the server* to use the server's host x509 certificate. Edit `/opt/rucio/etc/rucio.cfg`:
```
[client]
;auth_type = userpass
username = ddmlab
password = secret
auth_type = x509
client_cert = /etc/grid-security/hostcert.pem
client_key = /etc/grid-security/hostkey.pem
```

Delete default identities
```
rucio-admin identity delete --account root --type GSS --id "ddmlab@CERN.CH"
rucio-admin identity delete --account root --type X509 --id "/C=CH/ST=Geneva/O=CERN/OU=PH-ADP-CO/CN=DDMLAB Client Certificate/emailAddress=ph-adp-ddm-lab@cern.ch"
rucio-admin identity delete --account root --type USERPASS --id "ddmlab"
rucio-admin identity delete --account root --type SSH --id "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq5LySllrQFpPL614sulXQ7wnIr1aGhGtl8b+HCB/0FhMSMTHwSjX78UbfqEorZV16rXrWPgUpvcbp2hqctw6eCbxwqcgu3uGWaeS5A0iWRw7oXUh6ydnVy89zGzX1FJFFDZ+AgiZ3ytp55tg1bjqqhK1OSC0pJxdNe878TRVVo5MLI0S/rZY2UovCSGFaQG2iLj14wz/YqI7NFMUuJFR4e6xmNsOP7fCZ4bGMsmnhR0GmY0dWYTupNiP5WdYXAfKExlnvFLTlDI5Mgh4Z11NraQ8pv4YE1woolYpqOc/IMMBBXFniTT4tC7cgikxWb9ZmFe+r4t6yCDpX4IL8L5GOQ== ddmlab"
```


You may also add an X509 identity to the root account for remote administration. E.g.:
```
rucio-admin identity add --account root --type X509 \
    --id "/DC=org/DC=XXXXXXX/C=XX/O=XXXX/CN=Albert Einstein albert.einstein@ligo.org" \
    --email albert.einstein@ligo.org
```
In *your* local client configuration, change to `x509_proxy` authentication in
the `${RUCIO_HOME}/etc/rucio.cfg`


### Daemon Configuration (SupervisorD)
Rucio daemons are controlled with supervisord.  These directions assume
supervisor has been installed via pip (yum installs supervisor dependencies
which conflict with other pip packages).

Copy the supervisor configuration file into place:
```
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/supervisord.conf -o /etc/supervisord.conf
```
Edit inet server and username/password (if you want the webui). Set an
appropriate username/password in::
```
[inet_http_server]         ; inet (TCP) server disabled by default
port=rucio.ligo.caltech.edu:9001        ; (ip_address:port specifier, *:port for all iface)
username=user              ; (default is no username (open server))
password=123               ; (default is no password (open server))
```

Create log and run directories:
```
mkdir /var/log/supervisor
mkdir /var/run/supervisor
```


Add the rucio daemon configuration file:
```
mkdir /etc/supervisord.d
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/rucio-daemons.ini -o /etc/supervisord.d/rucio-daemons.ini
```

Check supervisord runs ok by launching in foreground:
```
supervisord --nodaemon
```

#### Starting supervisord at boot
The pip-installed supervisor does not provide init scripts to run supervisor
under systemd.  However, the supervisor
[documentation](http://supervisord.org/index.html) refers to a [github
repository](https://github.com/Supervisor/initscripts) where init scripts are
provided.  Download and install the CentOS script:
```
curl https://raw.githubusercontent.com/Supervisor/initscripts/master/centos-systemd-etcs -o  /usr/lib/systemd/system/supervisord.service
```
We should now be able to enable and start `supervisord` through `systemctl`
```
systemctl enable supervisord
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
systemctl start supervisord
```

## Proxy Delegation For FTS
Third party transfers are performed via an [FTS
server](http://fts3-docs.web.cern.ch/fts3-docs/index.html).  To authenticate 3rd party transfers:
 * Map the rucio server host identity to an account at each end point.
 * Create an X509 proxy certificate from the rucio server host certificate.
 * Delegate the proxy to the FTS server for authentication at each end point.

**Note**: the host certificate used here must be from a CA recognised by all
sites in the rucio distribution network.  We have found that LIGO robot
certificates are not recognised by EGI sites and we need to use an IGTF-issued
certificate, with a subject which follows the IGTF specifications (e.g., no
postcode!).

Valid proxy creation and delegation is performed via a cronjob which executes a
simple script with `voms-proxy-init` and `fts-rest-delegate`.  

The script can be found [here](https://git.ligo.org/james-clark/gwrucio/blob/master/bin/fts_proxy) and downloaded / installed with

```
    curl https://git.ligo.org/james-clark/gwrucio/raw/master/bin/fts_proxy -o /opt/rucio/tools/fts_proxy
```

Edit the crontab to refresh the proxy and delegation at 6am and 6pm:
```
0 6,18 * * * /bin/bash /opt/rucio/tools/fts_proxy > /dev/null 2>&1
```

### Support for EGI Sites
To authenticate with EGI sites, we need to register our certificate with the Virgo VOMs:

Create a pkcs12 certificate bundle for import into a browser:
```
openssl pkcs12 -export -out rucio-bundle.pfx -inkey /etc/grid-security/hostkey.pem -in /etc/grid-security/hostcert.pem
```
Download and import to a browser and navigate to:
[http://www.italiangrid.it/users/join_vo](http://www.italiangrid.it/users/join_vo):
request to join the Virgo VO.

Finally, add the following files to `/etc/vomses`:
 * [virgo-voms.cnaf.infn.it]()
 * [virgo-voms-01.pd.infn.it]()
```
mkdir /etc/vomses
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/virgo-voms.cnaf.infn.it -o /etc/vomses/virgo-voms.cnaf.infn.it
curl https://git.ligo.org/james-clark/gwrucio/raw/master/configs/server/virgo-voms-01.pd.infn.it -o /etc/vomses/virgo-voms-01.pd.infn.it
```