How to Configure the DQR INI file

The DQR is configured using an INI file. This tells the listener which follow-up should be included in the list and how it should be organized (tiers and questions). It also tells the DQR which data quality products it needs to produce itself within a DAG and which it can expect to be provided externally.

Below, we describe the key sections and options within the INI file. We note that the DQR maintains a near-production ready dqr.ini example within ~etc/ for reference.

Required sections

[general]

This section specifies a few high-level options that are common to how the DQR schedules follow-up and where it will write its logging and intermediate data products. You must provide the following options

  • output_dir

    • The full path to the output directory into which logging and intermediate data products will be placed. The DQR will generate some basic directory structure underneath this directory, but that is managed internally.

  • output_url

    • The URL corresponding to output_dir. This may be needed if processes write large amounts of output that are not uploaded to GraceDb themselves, but instead links are placed in the DQR’s html. Note, this behavior should be avoided whenever possible and all relevent data products should be uploaded to GraceDb, meaning this option should mostly go unused. However, that can be impractical in some circumstances.

  • gracedb_url

    • The URL for the instance of GraceDb you’d like to use. This can be a local path instead of a server name, in which case the DQR will use an instance of FakeDb as a proxy for GraceDb’s Python REST API.

  • keyname (optional)

    • If posting to a logbook (–do-not-post was not specified), this is the username used with keytab to authenticate via kerberos. This will depend on how you’ve set up your keytab.

  • keytab (optional)

    • If posting to a logbook (–do-not-post was not specified), this is the keytab used with keyname to authenticate via kerberos. This will depend on where you store your keytab.

  • logbook_url (optional)

    • If posting to a logbook (–do-not-post was not specified), this is the main URL of that logbook from which the permanent link will be generated.

  • request_url (optional)

    • If posting to a logbook (–do-not-post was not specified), this is the url for the PHP request to generate a new post.

[tiers]

This section enumerates who should be emailed upon the completion of each tier in the report. Recipients should be specified as a space-delimited list for each tier. For example:

0 = alice@ligo.org bob@ligo.org
1 = albert@ligo.org karl@ligo.org bob@ligo.org

Note, if any tasks are assigned to a tier that is not present in this section, no email will be sent when that tier completes. One can also supply a name for each tier such as:

0 = alice@ligo.org bob@ligo.org
0 name = Low Latency

If no name is specified, the name defaults to “Tier tier” (e.g.: “Tier 0”).

Optional Sections

[DEFAULT]

This section specifies common options for all follow-up tasks. Alternatively, you can specify options in each follow-up task’s section separately. The latter is likely to be convenient when specifying the tier and question of each task.

Most commonly, [DEFAULT] will only contain condor_classads. You can specify an arbitrary number of classads with the following structure:

condor_classads =
  universe vanilla
  getenv True
  accounting_group some.accounting.tag
  accounting_group_user albert.einstein

Sections for follow-up task

Each follow-up task that is to be included in the DQR is specified with a separate section. The task’s name is specified as the section name (e.g., [Task Name]) and is used to predict the expected JSON filename uploaded to GraceDb as well as internal routing and lookup within the DQR’s scheduler. JSON reports’ filenames are predicted via:

"%s-%s.json"%(task_name.replace(" ",""), graceid)

so that Task Name for G123 will be rendered as TaskName-G123.json.

As a minimum, each of these sections must contain the following options

  • include_in_dag

    • A boolean flag that tells the DQR scheduler whether to attempt to write a SUB file for this task within its DAG. Some follow-up must be managed external to the DQR’s scheduler, but most follow-up should have include_in_dag = True.

  • tier

    • Specifies which tier this task is assigned in the DQR html. Tiers must be integers and tasks are inherited hierarchically through all tiers. That means that a task assigned tier=0 will be included in the html for all tier>=0. Tiers can take any value and do not have to be consecutive integers.

  • question

    • The high-level question this task attempts to answer. question should be a string and is used to organize output within each tier of the DQR’s html document. These can be any string, but you will likely use existing questions identified in the Hierarchical Layout.

  • allow

    • a space-delimited list of states this task is allowed to return. Note that error is automatically included and does not need to be specified. Thus, this should typically be a subset of: pass, fail, human_input_needed.

    • if a task returns a state outside of this list, it is rendered as a bad_state within the DQR html and treated as an error.

Additional special options are available

  • librarian

    • a string listing the point of contact for this task. This will be rendered for “missing” tasks within the html.

  • toggles

    • a space-delimited list of toggles to be included in the dqr.html document. These toggles allow users to filter which tasks are shown at a given time. Each toggles can be any string (as long as it does not contain spaces), but are typically IFOs (e.g.: H1)

  • email_upon_error

    • a single email address. If the associated task fails within the DAG (exit code != 0), this email will be notificed automatically by Condor. If no email is specified, no warning email will be sent.

    • we note that this will overwrite options passed through condor_classads, so the user should be careful!

  • pre

    • the path to an executable run as the Condor pre-script for this job. If not supplied, no pre-script is written in the DAG for this task. This could be used to, for example, source a particular environment needed for that job but which could conflict with other jobs.

  • post

    • the path to an executable run as the Condor post-script for this job. If not supplied, no post-script is written in the DAG for this task.

You can (and should) specify other options for your follow-up under the associated section. These will be passed to your executable as command-line arguments within the SUB file. See Adding a new check that will be run within the DQR’s DAG for more detail.