It appears that condor_dagman is having trouble authenticating to the schedd:
07/29/20 15:56:09 AUTH_ERROR: Generic preauthentication failure
07/29/20 15:56:09 SECMAN: required authentication with schedd at <18.104.22.168:9618> failed, so aborting command QMGMT_WRITE_CMD.
07/29/20 15:56:09 WARNING: failed to connect to queue manager (AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using KERBEROS)
This output occurs when running a very simple diamond DAG, for instance. I see it ~4 times right away, and then occasionally later on. With DAGMAN_QUEUE_UPDATE_INTERVAL set to the default of 300 it pretty much only reoccurs at the end of
the DAGâs run. When I shorten the DAGMAN_QUEUE_UPDATE_INTERVAL to 10 this reoccurs more frequently (not exactly every 10 seconds, but maybe around the time each time a node in the DAG completes).
BTW, we noticed this issue because dagman job ClassAds are seemingly not being updated, i.e., the DAG_ attributes are not getting added as listed here:
And weâre suspicious that these authentication errors may point to the underlying reason.
Does anyone have any input towards this issue/these issues? Thanks!