[HTCondor-users] dagman fails to authenticate to schedd and update its job ClassAds

["Rundall, Jacob D" <rundall@xxxxxxxxxxxx>]

It appears that condor_dagman is having trouble authenticating to the schedd:

07/29/20 15:56:09 AUTH_ERROR: Generic preauthentication failure

07/29/20 15:56:09 SECMAN: required authentication with schedd at <141.142.181.239:9618> failed, so aborting command QMGMT_WRITE_CMD.

07/29/20 15:56:09 WARNING: failed to connect to queue manager (AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using KERBEROS)

 

This output occurs when running a very simple diamond DAG, for instance. I see it ~4 times right away, and then occasionally later on. With DAGMAN_QUEUE_UPDATE_INTERVAL set to the default of 300 it pretty much only reoccurs at the end of the DAGâs run. When I shorten the DAGMAN_QUEUE_UPDATE_INTERVAL to 10 this reoccurs more frequently (not exactly every 10 seconds, but maybe around the time each time a node in the DAG completes).

 

BTW, we noticed this issue because dagman job ClassAds are seemingly not being updated, i.e., the DAG_ attributes are not getting added as listed here:

https://htcondor.readthedocs.io/en/stable/users-manual/dagman-applications.html#status-information-for-the-dag-in-a-classad

 

And weâre suspicious that these authentication errors may point to the underlying reason.

 

Does anyone have any input towards this issue/these issues? Thanks!