[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] desire to Condor control via Python

Kipp Cannon wrote:


My name is Kipp Cannon and I'm a member of the LIGO scientific
collaboration. We are in the process of developing the infrastructure to
perform analysis of the gravitational wave detector outputs in (near) realtime. To this end, we have a need to control condor from within other
software. We need to be able to submit dags, query status, terminate all
jobs associated with a dag, etc.. In particular, we would like to be able
to do this from within Python.

What tools exist already for this? I know that pyGlobus provides Python bindings for the Globus Tool Kit, and the Globus "GRAM" manager is aparently able to control Condor but this seems like a circuitous route to the solution. Is there a more direct solution --- just some Python code that can run condor_submit_dag, parse log files, etc., to provide something of a scriptable interface to the whole show?



The dumb-first approach is to just wrap the existing condor shell commands using Python. We've done this for a few
commands (such as condor_submit_dag). We haven't bothered with wrapping condor_q or condor_rm, though those are fairly straightfoward as well.
It appears that there is already such a module for Perl in the lib directory of the condor installation ("Condor.pm")

Another approach might be to wrap the condor API using SWIG to submit and delete jobs. I haven't actually seen any header or libraries that
expose the job API to the user, so I don't know if this is possible.

Condor already exposes a log-file parsing API, and there is an XML log file format which we use. One problem is the XML file format isn't strictly correct (XML validators choke on it because it doesn't wrap the document in a top-level tag, which is normally required for XML).

Lastly there is 'condor birdbath' (check google), which exposes a SOAP interface for Condor. This is is the most promising way to interact with Condor; you
could use ZSI or other Python SOAP implementations to communicate with Condor. I've used this but ultimately ran into trouble with the implementation
crashing on me; I don't have really good feedback on when this will be a reliable feature.