bilby_pipe.htcondor_sync
========================

.. py:module:: bilby_pipe.htcondor_sync

.. autoapi-nested-parse::

   bilby_pipe_htcondor_sync can be used to sync jobs running under HTCondor where
   transfer-files=True (i.e. they do not use a virtual file system). It will use
   rsync to copy the results from remote worker nodes or spool back to the execute
   node.  The executable is assumed to be run from the submit directory (i.e.,
   where the .ini file is) and on the submit machine (i.e. where the job was
   submitted from).

   ..
       !! processed by numpydoc !!


Attributes
----------

.. autoapisummary::

   bilby_pipe.htcondor_sync.methods


Functions
---------

.. autoapisummary::

   bilby_pipe.htcondor_sync.get_cluster_id_list
   bilby_pipe.htcondor_sync.get_cluster_id
   bilby_pipe.htcondor_sync.rsync_via_ssh
   bilby_pipe.htcondor_sync.rsync_via_spool
   bilby_pipe.htcondor_sync.main


Module Contents
---------------

.. py:function:: get_cluster_id_list(outdir)

   
   Get a list of cluster IDs for every running analysis job
















   ..
       !! processed by numpydoc !!

.. py:function:: get_cluster_id(logfile)

   
   Read a log files to determine the latest cluster ID

   Extract the HTCondor cluster ID from the .log file. For example, if the log
   file reads

   ```
   001 (100503183.000.000) 2022-03-07 15:22:49 Job executing on host: <10.14.4.164...>
   ```
   Then this function return the cluster ID 100503183

   :Parameters:

       **logfile: str**
           A path to a HTCondor log file



   :Returns:

       cluster_id: str
           The cluster ID. If not ID is found, None is returned and a log message
           is printed.











   ..
       !! processed by numpydoc !!

.. py:function:: rsync_via_ssh(cluster_id, outdir, verbose=False, timeout=30)

   
   Attempt to rsync the local (submit) directory to current running worker nodes

   This method applies when the job is actively executing on a remote worker
   node and condor_ssh_to_job is possible. The method works by using
   condor_ssh_to_job and rsync as described in the HTCondor documentation:
   https://htcondor.readthedocs.io/en/latest/man-pages/condor_ssh_to_job.html.

   :Parameters:

       **cluster_id: int**
           The HTCondor clusterId

       **outdir: str**
           The top-level outdir of the bilby_pipe job

       **verbose: bool**
           If true, print explicit error messages

       **timeout: int**
           The timeout interval



   :Returns:

       success: bool
           True if the method was successful.











   ..
       !! processed by numpydoc !!

.. py:function:: rsync_via_spool(cluster_id, outdir, verbose=False)

   
   Attempt to rsync the local (submit) directory to the spool

   This method applies when the job is not actively executing on a remote
   worker, but is idle.  In this instance, any files produced by the job will
   be stored in the spool (a local directory on the submit machine). This
   methods identifies the spool location, based on the cluster_id, and attempts
   to rsync the data.

   :Parameters:

       **cluster_id: int**
           The HTCondor clusterId

       **outdir: str**
           The top-level outdir of the bilby_pipe job



   :Returns:

       success: bool
           True if the method was successful.











   ..
       !! processed by numpydoc !!

.. py:data:: methods

.. py:function:: main()