[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Thread safety of htcondor python bindings



Hi John,

Thanks for the reply. Consider this small example:

```
def run(njobs):
  schedd = htcondor.Submit()
  sub = htcondor.Submit(...) # details ommitted
  with schedd.transaction() as txn:
    sub.queue(txn, njobs)


p = Pool()
p.map(run, [njobs] * 2]
p.close()
```

These are different processes (not threads). Considering that bindings eventually call `condor_submit`, I am under the assumption that this is a safe thing to do. Could you please confirm that?

If it is not safe, than I will use a synchronization mechanism to serialize the calls. (Unfortunately, I can't fundamentally change our code such that all the submissions are from one process)


Thanks!
Sandeep



On Tue, Feb 27, 2018 at 9:47 AM, John M Knoeller <johnkn@xxxxxxxxxxx> wrote:

The bindings themselves are thread safe as far as we know, but the HTCondor daemons are not multithreaded.ÂÂ In particular, you *cannot*Â have multiple python threads sending commands to the same HTCondor daemon simultaneously.Â

Â

You should design your python programs so that only one thread does job submission for any given schedd, or you are very likely to have intermittent failures of submission and disconnection by the schedd.

Â

You can *probably* get away with multiple threads doing queries to the collector or schedd at the same time, but even then I would encourage you to try and avoid this.

Â

-tj

Â

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Sandeep Gupta
Sent: Tuesday, February 27, 2018 7:50 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Thread safety of htcondor python bindings

Â

Hello Biruk,

Â

I am a new user of the python bindings of ht condor. I primarily use it via multiprocessing module, and have seen no issues so far. Here is the source code for python bindings:Âhttps://github.com/htcondor/htcondor/tree/master/src/python-bindings

There are some synchronization mechanisms, but I am not aware of any thread safety guarantees. (I could be wrong though).

Â

Â

Sandeep.

Â

On Mon, Feb 26, 2018 at 9:15 PM, Biruk Mammo <birukw@xxxxxxxxxx> wrote:

Hello,

Â

Is the Python htcondor module guaranteed to be thread-safe?

Â

Â


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

Â


This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited.Â


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/


This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited.Â