[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] API



Hello IÂm new in this topic, but IÂm using HTCondor with python, In my case I use a library that execute command in the terminal, so you have to install HTCondor and make the configuration, then you can make command like condor_q, condor_status and condor_submit, here is my code that I have in github I did it with Flask and I want to do a GUI.
https://github.com/fabianMM04/h/blob/master/wms.py


2017-12-18 16:44 GMT-05:00 Todd L Miller <tlmiller@xxxxxxxxxxx>:
I have a python script and it kicks off a bunch of threads. Each of
those threads kicks off another bunch of threads. Each of those
threads runs a python script using Popen. Currently this is run from
cron on one host. I would like to move this to condor to take
advantage of multiple machines and achieve parallelism.
...
Does using the API for this seem like the best way?

    Generally, the right API depends on how much communication is necessary between the different threads (and which direction it is), and how long each thread lasts. I tend to think that HTCondor works best when scheduling jobs that don't need to communicate with each other and take about an hour to run. If that describes the last step of your application (the Python script called by popen()), then you might benefit from replacing that popen() with an HTCondor job.

    It it takes a while to get to that point, you could consider replacing the second-level threads with HTCondor jobs as well. If the second level takes a while, but the scripts it popen()s do not, maybe you want to start HTCondor jobs instead of second-level threads but not replace the popen()s.

Can anyone point me at an example for using the API?

    Brian Bockleman has been working on improving the documentation for the Python API. The documentation is here:

https://htcondor-python.readthedocs.io/en/latest/

His tutorial from earlier this year has links to earlier tutorials and
other resources:

http://research.cs.wisc.edu/htcondor/HTCondorWeek2017/presentations/TueBockelman_Python.pdf

- ToddM

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxx.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/