Testing a Technical Solution

There are many ways to test and debug your code. We list a few common “tricks” that might be useful, but your reviewers will almost certainly require more than just these. We strongly suggest starting with existing Test events before moving on to more complicated workflows involving creating simulated events!

Basic debugging with existing Test events

Because creating simulated events can be a hassle, it is often convenient to find exising Test events and use those. Typically, one should look in gracedb-test.ligo.org or gracedb-playground.ligo.org. Do not use Test events in gracedb.ligo.org (the production server) unless you are absolutely sure you know what you’re doing!

Test events are easy enough to find. Simply navigate to either the Latest or Search tabs in the GraceDb web interface and search for “Test”. This will return you a set of Test events, from which you can pick a victim. Note, the GraceID should start with “T” (e.g.: T12345). If it does not, you should pick another event.

Once you’ve chosen your victim, say “T12345” in gracedb-test.ligo.org, you should test your new executable (call it new-exe-dqr for concreteness) with something like:

new-exe-dqr T12345 --gracedb-url https://gracedb-test.ligo.org/api/ [--other-options]

The exact syntax will depend strongly on how you’ve written your task. This should run your executable on T12345 and upload a DQR JSON report to that event in gracedb-test.ligo.org. Before conducting any further testing, you should make sure this work reliably!

Once you’re confident your individual executable works, you can generate an entire DQR workflow with your task configured in /path/to/dqr.ini via:

dqr-create -v /path/to/dqr.ini T12345 --do-not-submit --do-not-upload

This should create a DAG for your workflow. Check that the DAG looks reasonable, and then either launch it directly or run:

dqr-create -v /path/to/dqr.ini T12345

This will re-write the DAG, submit it, and also upload the required html, javascript, and JSON config files to T12345. Remember to correctly set gracedb_url in your INI file!

If the DAG completes, you should see your new executable’s results rendered in the DQR html page linked from the “data_quality” section of T12345’s landing page (e.g, https://gracedb-test.ligo.org/events/T12345/view/).

Testing with simulated events you create

The following is a rather advanced workflow and you may not need it. Users may want to primarily test their code with existing Test events before embarking on this route.

Besides standard debugging procedures, the main package that will be useful when testing your technical solution is LVAlertTest. Below, we describe a basic workflow using LVAlertTest, and more information can be found within LVAlertTest’s documentation. We also describe how one can integrate LVAlertTest with an instance of GraceDb itself, particularly GraceDb Test.

Using LVAlertTest’s simulate.py

Note, simulate.py is not included in this repository. Instead, you will need to install LVAlertTest and use it to generate simulated events and alerts.

simulate.py provides a convenient way to generate mock data similar to what would appear in GraceDb “in the wild.” Users can specify what type of events they’d like simulated, some basic properties of those events, along with the type of follow-up they’d like mimicked. For your purposes, you should only have to create events and not simulate follow-up, so your simulate.py config files should have a single section in them. See LVAlertTest’s documentation for more details.

What’s more, simulate.py allows you to use either remote GraceDb instances or local FakeDb instances. FakeDb is part of a simple Python library that mimics GraceDb’s REST interface but works with your local file system instead of a remote server. It allows you to thoroughly test your code’s behavior in a variety of situations without worrying about affecting remote servers. This also means you can develop and test your code while you’re offline. Again, see LVAlertTest’s documentation for more details on exactly how FakeDb works.

Integration with lvalertTest_listen and FakeDb

As a first step, you should test your code with FakeDb. If you’re developing in Python, or at least using GraceDb’s Python REST interface, the setup needed for FakeDb will not require any changes to your production code. Instead, you just need to specify a few command line options.

Within your code, you can use dqr.gracedb.init to instantiate either a remote connection to an instance of GraceDb or a local copy of FakeDb based on the gracedb_url. If the url is a local path, an instance of FakeDb will be returned. FakeDb’s API is identical to GraceDb’s Python REST API, so the rest of your code will be transparent to this change.

You will also need to set up your follow-up scheduler so it listens to FakeDb instead of GraceDb. This can be done with lvalertTest_listen, which is included within LVAlertTest. The workflow is similar to setting up production listeners, and we therefore refer you to both LVAlertTest’s documentation and GraceDb’s LVAlert tutorial.

When you’re ready to test your workflow, first launch a lvalertTest_listen instance pointing at your local FakeDb directory and then run simulate.py to simulate events. lvalertTest_listen will respond to events, scheduling and executing your workflow.:

>> lvalertTest_listen -c path/to/dqr_listener.ini -f path/to/FakeDb &
>> simulate.py -g path/to/FakeDb --num-events 1 path/to/simulate_config.ini

Be sure to clean up your instance of lvalertTest_listen running in the background when you’re done!

You should be able to demonstate your code’s end-to-end functionality, including error handling and reporting. An example of a complete testing workflow can be found here.

Integration with lvalert_listen and GraceDb Test

If you would like to use simulate.py with an actual instance of GraceDb, you can switch over from FakeDb easily. When running simulate.py, specify the URL for GraceDb Test instead of a local path for your FakeDb:

>> simulate.py -G https://gracedb-test.ligo.org/api/ ...

You will also need to set up your listener to listen to GraceDb Test. This can be done by using lvalert_listen instead of lvalertTest_listen and supplying the same INI file:

>> lvalert_listen -c path/to/you/config.ini -s lvalert-test.cgca.uwm.edu

Note: you will need to specify the LVAlert server associated with GraceDb Test; the default server is associated with the production GraceDb instead of GraceDb Test.

You should NEVER use the production GraceDb server for testing unless you have contacted the GraceDb developers beforehand!