Technical Design

The DQR works by uploading a simple html document, a supporting javascript library, and a JSON configuration file to each candidate event within GraceDb. Follow-up tasks are also scheduled within a DAG and submitted to the cluster for processing. Each follow-up task, upon completion, generates a standardized report and uploads it to GraceDb. This contains a state flag along with references to information necessary to justify automonous decisions made by that process or to inform analysts evaluating the results for themselves.

The javascript library running under the html document then identifies reports posted to GraceDb and parses out their results. This is done whenever the page is refreshed and does not rely on any external process to update the html or JSON configuration file as more processes finish. In this way, users will automatically see new results appear as soon as they are available. It also means the DQR is fire and forget, which reduces the monitoring and maintenance burden for developers.

The DQR can also include tasks run outside its DAG. These are included in the JSON configuration file uploaded to GraceDb and will therefore be rendered in the DQR’s html, but will not be managed by the DQR’s DAG. A prime example are the low-latency tasks performed under gwcelery before autonomous alerts are released; these must be reported to GraceDb in a DQR-compatible format so that they are rendered by the DQR even though the DQR did not run the tasks itself. This format is specific to the DQR and is not used elsewhere in GraceDB.

Below, we describe the file formats required by the DQR’s javascript library. We provide a simple Python package that supports these.

JSON Configuration format

The JSON configuration file uploaded by the DQR scheduler tells the javascript library what to render in the html. Users should not have to interact with these files, but we specify their structure below for completeness.

The config file contain a single JSON object with the following attributes

  • graceid (string)

  • hostname (string)

  • username (string)

  • date (string)

  • process_name (string)

  • process_version (string)

  • librarian (string)

  • tiers (array) * each element of the array must be an object with the following keys: name (string), id (string)

  • questions (array) * each element of the array must be an object with the following keys: name (string), id (string)

  • tasks (array) * each element of the array must be an object with the following keys: tier (string), question (string), name (string), id (string), allow (array of strings), toggels (array of strings), librrian (string)

An example config file is provided below:

{
  "graceid": "T000007",
  "hostname": "ldas-pcdev1",
  "username": "reed.essick",
  "date": "12:30:05 PDT Tue 15 May 2018",
  "process_name": "dqr-create",
  "process_version": "0.1",
  "librarian": "Reed Essick (reed.essick@ligo.org)"
  "tiers": [
      {"name": "low latency", "id": "0"},
      {"name": "high latency", "id": "1"},
      {"name": "offline", "id": "2"}
  ],
  "questions": [
      {"name": "a simple question", "id": "asimplequestion"},
      {"name": "another question", "id": "anotherquestion"}
  ],
  "tasks": [
      "allow": ["error", "fail", "pass"], "toggles": ["L1", "H1", "V1"]"}
      {"tier": "0", "question": "asimplequestion", "name": "example", "id": "example-T000007", "allow": ["error", "fail", "pass"], "toggles": ["L1", "H1", "V1"], "librarian":"no librarian specified"},
      {"tier": "0", "question": "anotherquestion", "name": "check_vector", "id": "check_vector-T000007","allow": ["error", "fail", "pass"], "toggles": ["L1"], "librarian":"no librarian specified"},
      {"tier": "1", "question": "asimplequestion", "name": "omegascans", "id": "omegascans-T000007", "allow": ["error", "human_input_needed", "fail", "pass"], "toggles": ["H1"], "librarian":"no librarian specified"},
      {"tier": "2", "question": "anotherquestion", "name": "some thing", "id": "something-T000007", "allow":["human_input_needed"], toggles=[], "librarian":"Al Einstein (albert.einstein@ligo.org)"},
  ],
}

JSON Report format

Follow-up reports are uploaded as files. The naming convention for each file is not strictly enforced as long as the id is correctly specified in the DQR’s JSON config file, but typically a follow-up process called a great process run for candidate T011020 should upload a file named agreatprocess-T011020.json.

Each report is a single JSON object with the following attributes.

  • state (string)

  • process_name (string)

  • process_version (string)

  • librarian (string)

  • date (string)

  • username (string)

  • hostname (string)

  • summary (string)

  • figures (array)

    • each element of the array must be an object with the format specified below.

  • tables (array)

    • each element of the array must be an object with the format specified below.

  • links (array)

    • each element of the array must be an object with the format specified below.

  • extra (unspecified)

    • no information included here will be used to render the report in the html document, but developers can include extra information in any format here. It will be stored in GraceDb along with the report for posterity.

An example JSON report is provided below:

{
  "state": "pass",
  "date": "09:34:54 PST Thu 04 Jan 2018",
  "username": "reed.essick",
  "hostname": "ldas-pcdev1",
  "process_name": "check_vector",
  "process_version": "0.1",
  "librarian": "Reed Essick (reed.essick@ligo.org)"
  "summary": "active fraction > 0.000e+00 --> pass event",
  "tables": [
      {
          "columns": ["active time", "duration", "active fraction", "activity threshold"]
          "data": [
              [0.0, 10.0, 0.0, 0.0],
              [0.1, 10.0, 0.01, 0.5]
          ],
          "stats": [
              [null, null, null, null],
              ["pass", "fail", "human_input_needed", "error"]
          ]
      }
  ],
  "figures": [
      {
          "src": "check_vectorA-T000023.png"
          "alt": "check_vectorA",
          "width": 500,
          "height": 500,
      },
      {
          "src": "check_vectorB-T000023.png"
          "alt": "check_vectorB",
          "width": 500,
          "height": 500,
      }
  ],
  "links": [
      {
          "href": "https://dcc.ligo.org",
          "innerHTML": "a link to the DCC which should contain some documentation for this process"
      },
      {
          "href": "https://git.ligo.org/detchar/data-quality-report",
          "innerHTML": "a link to the DQR repo"
      }
  ],
  "extra": [],
}

Although not strictly required, developers are strongly encouraged to always include a link to their product’s documentation within their report. This will provide analysts with immediate access to all relevant documentation directly from the DQR’s html page for each candidate.

Links, tables, and figures are specified as lists of JSON objects, with each object following the format outlined below.

Figure format

Each figure object must have the following attributes, all of which are strings:

  • src (string)

  • alt (string)

  • width (int, float)

  • height (int, float)

These are rendered as:

`<img src="src", alt="alt" width="width" height="height">``

An example is shown in JSON Report format.

Table format

Each table object must have the following attributes:

  • columns (array of strings)

    • the coloumn headers for the table

  • data (array of arrays of unspecified type)

    • each element of the array must be another array, and each element of those arrays will be rendered in a table data tag

  • states (array of arrays of (null, string)

    • each element of the array must be another array, and each element of those arrays must be either null or one of the allowed states (specified as a string)

We note that the shape of data and states must match. This is managed automatically if users use the Python library provided within this package. By specifying states, users can assign colors to each cell in the table and highlight specific data; colors follow the same conventions as the return states for each task.

An example is shown in JSON Report format.

JSON Override format

Override files are uploaded whenever an analyst wishes to change the state of a particular technical solution. If that technical solution’s report is named agreatprocess-T011020.json, then the JSON override file must be named agreatprocess-T011020_override.json.

The formatting and naming conventions are handled within dqr-interact so users do not need to construct these themselves. However, we include the specification here for completeness.

Override files contain a single JSON object with the following attributes, each of which is a string.

  • state (string; the “new state” for this task)

  • name (string; the “full neame” for the person applying this override)

  • date (string)

  • username (string)

  • hostname (string)

  • process_name (string; refers to the interaction command-line utility)

  • process_version (string; refers to the interaction command-line utility)

  • librarian (string; refers to the interaction command-line utility)

JSON Comment format

Comment files are uploaded whenever an analyst wishes to comment on a particular technial solution. If that technical solution’s report is named agreatprocess-T011020.json, then the JSON comment file must be named agreatprocess-T011020_comment.json. If multiple comments are uploaded, the DQR relies on GraceDb’s internal indexing to discover the order of comments and render them correctly.

The formatting and naming conventions are handled within dqr-interact so users do not need to construct these themselves. However, we include the specification here for completeness.

Note, GraceDb retains copies of files with the same filename by appending a suffix indicating the order in which they were uploaded. The DQR’s javascript library uses this to determine the order in which it should render comments, and therefore does not look for files named agreatprocess-T011020_comment.json, but instead looks for files named agreatprocess-T011020_comment.json,0, agreatprocess-T011020_comment.json,1, and so on.

Comment files contain a single JSON object with the following attributes, each of which is a string.

  • comment (string)

  • name (string; the “full name” for the person making this comment)

  • date (string)

  • username (string)

  • hostname (string)

  • process_name (string; refers to the interaction command-line utility)

  • process_version (string; refers to the interaction command-line utility)

  • librarian (string; refers to the interaction command-line utility)