.. _testing: Testing a Technical Solution ==================================================================================================== There are many ways to test and debug your code. We list a few common "tricks" that might be useful, but your reviewers will almost certainly require more than just these. **We strongly suggest starting with existing Test events before moving on to more complicated workflows involving creating simulated events!** Basic debugging with existing Test events ---------------------------------------------------------------------------------------------------- Because creating simulated events can be a hassle, it is often convenient to find exising Test events and use those. Typically, one should look in `gracedb-test.ligo.org` or `gracedb-playground.ligo.org`. **Do not use Test events in** `gracedb.ligo.org` (the production server) **unless you are absolutely sure you know what you're doing!** Test events are easy enough to find. Simply navigate to either the *Latest* or *Search* tabs in the GraceDb web interface and search for "Test". This will return you a set of Test events, from which you can pick a victim. **Note**, the GraceID should start with "T" (e.g.: T12345). If it does not, you should pick another event. Once you've chosen your victim, say "T12345" in `gracedb-test.ligo.org`, you should test your new executable (call it `new-exe-dqr` for concreteness) with something like:: new-exe-dqr T12345 --gracedb-url https://gracedb-test.ligo.org/api/ [--other-options] The exact syntax will depend strongly on how you've written your task. This should run your executable on T12345 and upload a DQR JSON report to that event in `gracedb-test.ligo.org`. **Before conducting any further testing, you should make sure this work reliably!** Once you're confident your individual executable works, you can generate an entire DQR workflow with your task configured in `/path/to/dqr.ini` via:: dqr-create -v /path/to/dqr.ini T12345 --do-not-submit --do-not-upload This should create a DAG for your workflow. Check that the DAG looks reasonable, and then either launch it directly or run:: dqr-create -v /path/to/dqr.ini T12345 This will re-write the DAG, submit it, and also upload the required html, javascript, and JSON config files to T12345. Remember to correctly set `gracedb_url` in your INI file! If the DAG completes, you should see your new executable's results rendered in the DQR html page linked from the "data_quality" section of T12345's landing page (e.g, `https://gracedb-test.ligo.org/events/T12345/view/`). Testing with simulated events you create ---------------------------------------------------------------------------------------------------- **The following is a rather advanced workflow and you may not need it.** Users may want to primarily test their code with existing Test events before embarking on this route. Besides standard debugging procedures, the main package that will be useful when testing your technical solution is `LVAlertTest `_. Below, we describe a basic workflow using `LVAlertTest `_, and more information can be found within `LVAlertTest's documentation `_. We also describe how one can integrate `LVAlertTest `_ with an instance of GraceDb itself, particularly `GraceDb Test `_. Using LVAlertTest's `simulate.py` ____________________________________________________________________________________________________ **Note**, `simulate.py` is *not included in this repository*. Instead, you will need to install `LVAlertTest `_ and use it to generate simulated events and alerts. `simulate.py` provides a convenient way to generate mock data similar to what would appear in GraceDb "in the wild." Users can specify what type of events they'd like simulated, some basic properties of those events, along with the type of follow-up they'd like mimicked. For your purposes, you should only have to create events and not simulate follow-up, so your `simulate.py` config files should have a single section in them. See `LVAlertTest's documentation `_ for more details. What's more, `simulate.py` allows you to use either remote GraceDb instances or local `FakeDb` instances. `FakeDb` is part of a simple Python library that mimics GraceDb's REST interface but works with your local file system instead of a remote server. It allows you to thoroughly test your code's behavior in a variety of situations without worrying about affecting remote servers. This also means you can develop and test your code while you're offline. Again, see `LVAlertTest's documentation `_ for more details on exactly how `FakeDb` works. Integration with lvalertTest_listen and FakeDb ____________________________________________________________________________________________________ As a first step, you should test your code with `FakeDb`. If you're developing in Python, or at least using GraceDb's Python REST interface, the setup needed for `FakeDb` will not require any changes to your production code. Instead, you just need to specify a few command line options. Within your code, you can use `dqr.gracedb.init` to instantiate either a remote connection to an instance of GraceDb or a local copy of FakeDb based on the `gracedb_url`. If the `url` is a local path, an instance of `FakeDb` will be returned. `FakeDb`'s API is identical to GraceDb's Python REST API, so the rest of your code will be transparent to this change. You will also need to set up your follow-up scheduler so it listens to `FakeDb` instead of GraceDb. This can be done with `lvalertTest_listen`, which is included within `LVAlertTest `_. The workflow is similar to setting up production listeners, and we therefore refer you to both `LVAlertTest's documentation `_ and `GraceDb's LVAlert tutorial `_. When you're ready to test your workflow, first launch a `lvalertTest_listen` instance pointing at your local `FakeDb` directory and then run `simulate.py` to simulate events. `lvalertTest_listen` will respond to events, scheduling and executing your workflow.:: >> lvalertTest_listen -c path/to/dqr_listener.ini -f path/to/FakeDb & >> simulate.py -g path/to/FakeDb --num-events 1 path/to/simulate_config.ini Be sure to clean up your instance of `lvalertTest_listen` running in the background when you're done! You should be able to demonstate your code's end-to-end functionality, including error handling and reporting. An example of a complete testing workflow can be found `here `_. Integration with lvalert_listen and GraceDb Test ____________________________________________________________________________________________________ If you would like to use `simulate.py` with an actual instance of GraceDb, you can switch over from FakeDb easily. When running `simulate.py`, specify the URL for GraceDb Test instead of a local path for your FakeDb:: >> simulate.py -G https://gracedb-test.ligo.org/api/ ... You will also need to set up your listener to listen to GraceDb Test. This can be done by using `lvalert_listen` instead of `lvalertTest_listen` and supplying the same INI file:: >> lvalert_listen -c path/to/you/config.ini -s lvalert-test.cgca.uwm.edu **Note**: you will need to specify the LVAlert server associated with GraceDb Test; the default server is associated with the production GraceDb instead of GraceDb Test. **You should NEVER use the production GraceDb server for testing unless you have contacted the GraceDb developers beforehand!**