Re: [HTCondor-users] Requests and HTCondor Python bindings seem to fight over certificate

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

> On Oct 27, 2015, at 8:34 AM, Todd Tannenbaum <tannenba@xxxxxxxxxxx> wrote:
> 
> On 10/27/2015 8:04 AM, Todd Tannenbaum wrote:
>>> Can anyone shed any light?
>>> 
>>> Thanks,
>>> 
>> 
>> Just a guess, but is your Python script and attempting to communicate
>> to both HTCondor (with GSI or SSL) and Puppet from different threads
>> in the same process?  If so, perhaps there is thread safety problem,
>> for instance perhaps the HTCondor client libraries that the Python
>> interface uses are not thread safe with respect to OpenSSL and/or
>> GSI.
>> 
>> regards Todd
> 
> Followup to my own post above :), but some evidence that my guess is
> correct -  see
> 
>  https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=4464
> 
> (the implementation of ticket 4464 changed some in ticket 4483, but I
> think the implication is the same).
> 
> Maybe you could try setting environment variable GLOBUS_THREAD_MODEL to "pthread", but it looks like the code purposefully overides that setting to prevent hard-to-diagnose problems if the Globus native packages are not installed or other such shared library problems--- a Linux-style spin on Windows DLL hell :)
> 
> So without doing any tests myself, I think at the current time htcondor's python bindings are not thread safe with respect to GSI or OpenSSL. Eventually, we should do something about this, but we could do something about it sooner rather than later if you cannot work-around this on your side..
> 

So, the HTCondor python module keeps a lock such that only one call to a HTCondor library can occur at a given time.  When this lock is taken, the global python interpreter lock (GIL) is released - meaning other python threads can proceed.

Since OpenSSL tends support obscure use cases (bizarre embedded systems that have no thread support) at the cost of the common use case, all locks default to no-ops (arenât you glad your security library is âunsafe by defaultâ?   Grumble grumble vent grumbleâ).

So, I suspect that the built-in python module and htcondor module are both invoking OpenSSL at the same time.

Luckily, thereâs an easy fix - we can actually initialize OpenSSL multithreading support.  Todd, can you open a ticket?

Brian