[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] how to knock at shared port daemon for initiating/debugging SSL handshake



Hi Thomas,

Unfortunately, older versions of OpenSSL do poor jobs when presented with multiple certification paths (often it'll follow one chain to the end and, if it doesn't like it, will reject instead of looking for other routes).  Is it possible this was one of the IGTF+CAB CAs?  I wouldn't be surprised if you look at the chain and found the one in the host certificate file was rooted in non-IGTF.

When there were similar multiple validation path issues inside IGTF itself, I knew which clients / servers had which bugs and could provide concrete advice on what versions to avoid.  I've since forgotten that -- but simply, when using IGTF, you shouldn't need the intermediate / root certificates.

Hope this helps,

Brian

> On Apr 11, 2022, at 9:30 AM, Thomas Hartmann <thomas.hartmann@xxxxxxx> wrote:
> 
> Hi Brian,
> 
> thanks for the info
> 
> I think we have identified at least a part of the issue - which seems to
> relate to the structure of the hosts' certificate bag structure.
> 
> On Friday noon, we deployed  new host certificates on our production
> CEs, where the host certificate pem file consisted of the actual host
> certificate followed by two BEGIN/END CERTIFICATE blocks with the
> issuers certs.
> 
> Since then handshakes failed for the Belle2 group (trying to gather
> their SSL libs in use) and partly locally [1]. After I now truncated the
> cert bag structs to the host cert only removing the tailing issuer
> certs, the handshakes became successful again.
> 
> However, during the period other users however other groups like ATLAS,
> CMS, LHCb (where LHCb should be in principle using the same framework as
> Belle2) saw no problems and handshaked as usual with the hostcert+issuer
> bag struct.
> I.e., some SSL libraries(?)/implementations seem to work while others
> fail - the condor and ssl packages on the CE as well as our local client
> look like [2], where both are CentOS 7.9.2009.
> 
> Cheers,
>   Thomas
> 
> [1]
> It failed at us for for condor_ce_{trace,ping} with X509 authz for proxy
> certs wfrom the same CA as the CE's host cert - while it worked when
> using grid proxy cert originating from another CA.
> 
> 
> [2]
> [2.CondorCE]
> htcondor-ce-bdii-5.1.3-1.el7.noarch
> condor-classads-9.0.11-1.el7.x86_64
> htcondor-ce-client-5.1.3-1.el7.noarch
> python3-condor-9.0.11-1.el7.x86_64
> htcondor-ce-view-5.1.3-1.el7.noarch
> python2-condor-9.0.11-1.el7.x86_64
> htcondor-ce-condor-5.1.3-1.el7.noarch
> condor-procd-9.0.11-1.el7.x86_64
> condor-externals-9.0.11-1.el7.x86_64
> condor-boinc-7.16.16-1.el7.x86_64
> htcondor-ce-5.1.3-1.el7.noarch
> condor-9.0.11-1.el7.x86_64
> htcondor-ce-apel-5.1.3-1.el7.noarch
> globus-gsi-openssl-error-4.3-1.el7.x86_64
> globus-gsi-proxy-ssl-6.5-1.el7.x86_64
> globus-openssl-module-5.2-1.el7.x86_64
> openssl-1.0.2k-25.el7_9.x86_64
> openssl-libs-1.0.2k-25.el7_9.x86_64
> perl-Crypt-OpenSSL-X509-1.803-4.el7.x86_64
> perl-IO-Socket-SSL-1.94-7.el7.noarch
> perl-Net-SSLeay-1.55-6.el7.x86_64
> python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch
> 
> 
> [2.Client]
> condor-9.0.11-1.el7.x86_64
> condor-classads-9.0.11-1.el7.x86_64
> condor-externals-9.0.11-1.el7.x86_64
> condor-procd-9.0.11-1.el7.x86_64
> htcondor-ce-client-5.1.3-1.el7.noarch
> python2-condor-9.0.11-1.el7.x86_64
> python3-condor-9.0.11-1.el7.x86_64
> globus-gsi-openssl-error-4.3-1.el7.x86_64
> globus-gsi-proxy-ssl-6.5-1.el7.x86_64
> globus-openssl-module-5.2-1.el7.x86_64
> gskssl64-8.0-55.24.x86_64
> mod_ssl-2.4.6-97.el7.centos.5.x86_64
> openssl098e-0.9.8e-29.el7.centos.3.x86_64
> openssl-1.0.2k-25.el7_9.x86_64
> openssl-devel-1.0.2k-25.el7_9.x86_64
> openssl-libs-1.0.2k-25.el7_9.x86_64
> perl-Crypt-OpenSSL-X509-1.803-4.el7.x86_64
> perl-IO-Socket-SSL-1.94-7.el7.noarch
> perl-Net-SSLeay-1.55-6.el7.x86_64
> python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch
> 
> 
> 
> On 11/04/2022 15.50, Bockelman, Brian wrote:
>> Hi Thomas,
>> 
>> Unfortunately, there's no simple way to debug - the TLS handshake occurs in the middle of HTCondor's binary protocol.
>> 
>> I usually debug such things by setting TOOL_DEBUG=D_FULLDEBUG,D_SECURITY.  At that level, the client does a reasonably good job of at least emitting the OpenSSL error messages -- and debug from there.
>> 
>> Brian
>> 
>>> On Apr 11, 2022, at 6:54 AM, Thomas Hartmann <thomas.hartmann@xxxxxxx> wrote:
>>> 
>>> Hi all,
>>> 
>>> is there a way to connect with `openssl s_client` to a CondorCE running
>>> a shared port daemon?
>>> 
>>> I.e., I would like to debug a probable certificate issue for one of our
>>> VOs, where their connection fails early - and my suspicion is, that
>>> their trusted CA chain is not in order. Thus, I would like them just to
>>> initiate the SSL handshake for more details.
>>> However, no SSL handshake is initiated when going directly for the
>>> CE:port - as I suppose that one needs to knock correctly for the shared
>>> port daemon, or?
>>> 
>>> Cheers,
>>> Thomas
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>> 
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>> 
>> 
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>> 
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/