Re: [Condor-users] desire to Condor control via Python

["David E. Konerding" <dekonerding@xxxxxxx>]

Kipp Cannon wrote:

> Hi,
>
> My name is Kipp Cannon and I'm a member of the LIGO scientific
> collaboration.  We are in the process of developing the infrastructure to
> perform analysis of the gravitational wave detector outputs in (near)  
> realtime.  To this end, we have a need to control condor from within other
> software.  We need to be able to submit dags, query status, terminate all
> jobs associated with a dag, etc..  In particular, we would like to be able
> to do this from within Python.
>
> What tools exist already for this?  I know that pyGlobus provides Python 
> bindings for the Globus Tool Kit, and the Globus "GRAM" manager is 
> aparently able to control Condor but this seems like a circuitous route 
> to the solution.  Is there a more direct solution --- just some Python 
> code that can run condor_submit_dag, parse log files, etc., to provide 
> something of a scriptable interface to the whole show?
>
> Thanks,
>
> 	

Kipp,

The dumb-first approach is to just wrap the existing condor shell 
commands using Python.  We've done this for a few
commands (such as condor_submit_dag).  We haven't bothered with wrapping 
condor_q or condor_rm, though those are fairly straightfoward as well.
It appears that there is already such a module for Perl in the lib 
directory of the condor installation ("Condor.pm")

Another approach might be to wrap the condor API using SWIG to submit 
and delete jobs.  I haven't actually seen any header or libraries that
expose the job API to the user, so I don't know if this is possible.

Condor already exposes a log-file parsing API, and there is an XML log 
file format which we use.  One problem is the XML file format isn't 
strictly correct (XML validators choke on it because it doesn't wrap the 
document in a top-level tag, which is normally required for XML).

Lastly there is 'condor birdbath' (check google), which exposes a SOAP 
interface for Condor.  This is is the most promising way to interact 
with Condor; you
could use ZSI or other Python SOAP implementations to communicate with 
Condor.  I've used this but ultimately ran into trouble with the 
implementation
crashing on me; I don't have really good feedback on when this will be a 
reliable feature.

Dave