.. _configuring: How to Configure the DQR INI file ==================================================================================================== The DQR is configured using an INI file. This tells the listener which follow-up should be included in the list and how it should be organized (`tiers` and `questions`). It also tells the DQR which data quality products it needs to produce itself within a DAG and which it can expect to be provided externally. Below, we describe the key sections and options within the INI file. We note that the DQR maintains a near-production ready `dqr.ini` example within `~etc/` for reference. Required sections ---------------------------------------------------------------------------------------------------- [general] ____________________________________________________________________________________________________ This section specifies a few high-level options that are common to how the DQR schedules follow-up and where it will write its logging and intermediate data products. You must provide the following options * `output_dir` * The full path to the output directory into which logging and intermediate data products will be placed. The DQR will generate some basic directory structure underneath this directory, but that is managed internally. * `output_url` * The URL corresponding to `output_dir`. This may be needed if processes write large amounts of output that are not uploaded to GraceDb themselves, but instead links are placed in the DQR's html. *Note*, this behavior should be avoided whenever possible and all relevent data products should be uploaded to GraceDb, meaning this option should mostly go unused. However, that can be impractical in some circumstances. * `gracedb_url` * The URL for the instance of GraceDb you'd like to use. This can be a local path instead of a server name, in which case the DQR will use an instance of `FakeDb` as a proxy for GraceDb's Python REST API. * `keyname` (optional) * If posting to a logbook (`--do-not-post` was *not* specified), this is the username used with `keytab` to authenticate via kerberos. This will depend on how you've set up your keytab. * `keytab` (optional) * If posting to a logbook (`--do-not-post` was *not* specified), this is the keytab used with `keyname` to authenticate via kerberos. This will depend on where you store your keytab. * `logbook_url` (optional) * If posting to a logbook (`--do-not-post` was *not* specified), this is the main URL of that logbook from which the permanent link will be generated. * `request_url` (optional) * If posting to a logbook (`--do-not-post` was *not* specified), this is the url for the PHP request to generate a new post. [tiers] ____________________________________________________________________________________________________ This section enumerates who should be emailed upon the completion of each `tier` in the report. Recipients should be specified as a space-delimited list for each tier. For example:: 0 = alice@ligo.org bob@ligo.org 1 = albert@ligo.org karl@ligo.org bob@ligo.org Note, if any tasks are assigned to a `tier` that is not present in this section, no email will be sent when that `tier` completes. One can also supply a name for each tier such as:: 0 = alice@ligo.org bob@ligo.org 0 name = Low Latency If no name is specified, the name defaults to "Tier `tier`" (e.g.: "Tier 0"). Optional Sections ---------------------------------------------------------------------------------------------------- [DEFAULT] ____________________________________________________________________________________________________ This section specifies common options for all follow-up tasks. Alternatively, you can specify options in each follow-up task's section separately. The latter is likely to be convenient when specifying the `tier` and `question` of each task. Most commonly, `[DEFAULT]` will only contain `condor_classads`. You can specify an arbitrary number of classads with the following structure:: condor_classads = universe vanilla getenv True accounting_group some.accounting.tag accounting_group_user albert.einstein Sections for follow-up task ____________________________________________________________________________________________________ Each follow-up task that is to be included in the DQR is specified with a separate section. The task's name is specified as the section name (e.g., [Task Name]) and is used to predict the expected JSON filename uploaded to GraceDb as well as internal routing and lookup within the DQR's scheduler. JSON reports' filenames are predicted via:: "%s-%s.json"%(task_name.replace(" ",""), graceid) so that `Task Name` for `G123` will be rendered as `TaskName-G123.json`. As a minimum, each of these sections must contain the following options * `include_in_dag` * A boolean flag that tells the DQR scheduler whether to attempt to write a SUB file for this task within its DAG. Some follow-up must be managed external to the DQR's scheduler, but most follow-up should have `include_in_dag = True`. * `tier` * Specifies which `tier` this task is assigned in the DQR html. Tiers must be integers and tasks are inherited hierarchically through all tiers. That means that a task assigned `tier=0` will be included in the html for all `tier>=0`. Tiers can take any value and do not have to be consecutive integers. * `question` * The high-level question this task attempts to answer. `question` should be a string and is used to organize output within each `tier` of the DQR's html document. These can be any string, but you will likely use existing questions identified in the :ref:`layout`. * `allow` * a space-delimited list of states this task is allowed to return. Note that `error` is automatically included and does not need to be specified. Thus, this should typically be a subset of: `pass`, `fail`, `human_input_needed`. * if a task returns a state outside of this list, it is rendered as a `bad_state` within the DQR html and treated as an error. Additional special options are available * `librarian` * a string listing the point of contact for this task. This will be rendered for "missing" tasks within the html. * `toggles` * a space-delimited list of toggles to be included in the dqr.html document. These toggles allow users to filter which tasks are shown at a given time. Each toggles can be any string (as long as it does not contain spaces), but are typically IFOs (e.g.: H1) * `email_upon_error` * a single email address. If the associated task fails within the DAG (exit code != 0), this email will be notificed automatically by Condor. If no email is specified, no warning email will be sent. * we note that this will overwrite options passed through `condor_classads`, so the user should be careful! * `pre` * the path to an executable run as the Condor pre-script for this job. If not supplied, no pre-script is written in the DAG for this task. This could be used to, for example, source a particular environment needed for that job but which could conflict with other jobs. * `post` * the path to an executable run as the Condor post-script for this job. If not supplied, no post-script is written in the DAG for this task. You can (and should) specify other options for your follow-up under the associated section. These will be passed to your executable as command-line arguments within the SUB file. See :ref:`within-dag` for more detail.