[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] API
- Date: Mon, 18 Dec 2017 15:44:28 -0600 (CST)
- From: Todd L Miller <tlmiller@xxxxxxxxxxx>
- Subject: Re: [HTCondor-users] API
I have a python script and it kicks off a bunch of threads. Each of
those threads kicks off another bunch of threads. Each of those
threads runs a python script using Popen. Currently this is run from
cron on one host. I would like to move this to condor to take
advantage of multiple machines and achieve parallelism.
Does using the API for this seem like the best way?
Generally, the right API depends on how much communication is
necessary between the different threads (and which direction it is), and
how long each thread lasts. I tend to think that HTCondor works best when
scheduling jobs that don't need to communicate with each other and take
about an hour to run. If that describes the last step of your application
(the Python script called by popen()), then you might benefit from
replacing that popen() with an HTCondor job.
It it takes a while to get to that point, you could consider
replacing the second-level threads with HTCondor jobs as well. If the
second level takes a while, but the scripts it popen()s do not, maybe you
want to start HTCondor jobs instead of second-level threads but not
replace the popen()s.
Can anyone point me at an example for using the API?
Brian Bockleman has been working on improving the documentation
for the Python API. The documentation is here:
His tutorial from earlier this year has links to earlier tutorials and