Mailing List Archives

Public Access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Upgrade of HTCondor-CE from 5 to 6 broke my CE

It appears you have strong authentication disabled in your pool.
I think the quickest way to get the CE working is to configure it to accept no authentication when talking with your HTCondor pool. Try adding these config params to you CE configuration (in a file under /etc/condor-ce/config.d/):

SEC_CLIENT_ENCRYPTION=optional
SEC_CLIENT_INTEGRITY=optional
SEC_CLIENT_AUTHENTICATION=optional

 - Jaime

On Mar 5, 2024, at 6:02âAM, David Cohen <cdavid@xxxxxxxxxxxxxxxxxxxxxx> wrote:

Hi Jaime,
Any update on this?


On Sat, Mar 2, 2024 at 6:31âAM David Cohen <cdavid@xxxxxxxxxxxxxxxxxxxxxx> wrote:
Hi Jaime,
The upgrade was from 9.0.20 to 10.9.0. Job submissions from the other schdd still work.
Please see the output bellow:


tau-htc ~]# condor_q -pool tau-cm.hep.tau.ac.il:9618 -name tau-htc.hep.tau.ac.il -debug:D_SECURITY:2
03/02/24 06:25:26 KEYCACHE: created: 0x8e1210
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission ALLOW
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission READ
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission WRITE
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission NEGOTIATOR
03/02/24 06:25:26 ipverify: NEGOTIATOR optimized to deny everyone
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission ADMINISTRATOR
03/02/24 06:25:26 ipverify: ADMINISTRATOR optimized to deny everyone
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission CONFIG
03/02/24 06:25:26 ipverify: CONFIG optimized to deny everyone
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission DAEMON
03/02/24 06:25:26 ipverify: DAEMON optimized to deny everyone
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission SOAP
03/02/24 06:25:26 ipverify: SOAP optimized to deny everyone
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission DEFAULT
03/02/24 06:25:26 ipverify: DEFAULT optimized to deny everyone
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission CLIENT
03/02/24 06:25:26 IPVERIFY: allow CLIENT: * (from config value ALLOW_CLIENT)
03/02/24 06:25:26 ipverify: CLIENT optimized to allow anyone
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission ADVERTISE_STARTD
03/02/24 06:25:26 ipverify: ADVERTISE_STARTD optimized to deny everyone
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission ADVERTISE_SCHEDD
03/02/24 06:25:26 ipverify: ADVERTISE_SCHEDD optimized to deny everyone
03/02/24 06:25:26 IPVERIFY: Subsystem TOOL
03/02/24 06:25:26 IPVERIFY: Permission ADVERTISE_MASTER
03/02/24 06:25:26 ipverify: ADVERTISE_MASTER optimized to deny everyone
03/02/24 06:25:26 Initialized the following authorization table:
03/02/24 06:25:26 Authorizations yet to be resolved:
03/02/24 06:25:26 SECMAN: command 6 QUERY_SCHEDD_ADS to collector at <192.114.100.129:9618> from TCP port 26415 (blocking).
03/02/24 06:25:26 Filtering authentication methods (FS,TOKEN,SCITOKENS,SSL,IDTOKENS,PASSWORD) prior to offering them remotely.
03/02/24 06:25:26 Can try token auth because we have at least one named credential.
03/02/24 06:25:26 Will try IDTOKENS auth.
03/02/24 06:25:26 Can try token auth because we have at least one named credential.
03/02/24 06:25:26 Will try IDTOKENS auth.
03/02/24 06:25:26 Inserting pre-auth metadata for TOKEN.
03/02/24 06:25:26 Inserting pre-auth metadata for TOKEN.
03/02/24 06:25:26 SECMAN: no cached key for {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<6>}.
03/02/24 06:25:26 SECMAN: Security Policy:
AuthMethods = "FS,TOKEN,SCITOKENS,SSL,TOKEN,PASSWORD"
Authentication = "OPTIONAL"
CryptoMethods = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BABrYaeUqNGomF/DFLWl/N/sAbc+LTLQfL155RumG0S2mE48rvTW9tfkMqCoQqWp18RPVSzUCdPHMgzUTHHeLqM="
Enact = "NO"
Encryption = "OPTIONAL"
Integrity = "OPTIONAL"
IssuerKeys = "POOL, POOL.puppet-bak"
NegotiatedSession = true
NewSession = "YES"
OutgoingNegotiation = "PREFERRED"
ServerPid = 1135683
SessionDuration = "60"
SessionLease = 3600
Subsystem = "TOOL"
TrustDomain = "hep.tau.ac.il"
03/02/24 06:25:26 SECMAN: negotiating security for command 6.
03/02/24 06:25:26 SECMAN: sending DC_AUTHENTICATE command
03/02/24 06:25:26 SECMAN: sending following classad:
AuthMethods = "FS,TOKEN,SCITOKENS,SSL,TOKEN,PASSWORD"
Authentication = "OPTIONAL"
Command = 6
ConnectSinful = "<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>"
CryptoMethods = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BABrYaeUqNGomF/DFLWl/N/sAbc+LTLQfL155RumG0S2mE48rvTW9tfkMqCoQqWp18RPVSzUCdPHMgzUTHHeLqM="
Enact = "NO"
Encryption = "OPTIONAL"
Integrity = "OPTIONAL"
IssuerKeys = "POOL, POOL.puppet-bak"
NegotiatedSession = true
NewSession = "YES"
OutgoingNegotiation = "PREFERRED"
RemoteVersion = "$CondorVersion: 10.9.0 2023-09-28 BuildID: 678228 PackageID: 10.9.0-1 $"
ServerPid = 1135683
SessionDuration = "60"
SessionLease = 3600
Subsystem = "TOOL"
TrustDomain = "hep.tau.ac.il"
03/02/24 06:25:26 SECMAN: server responded with:
AuthMethods = "FS"
AuthMethodsList = "FS,TOKEN,TOKEN,SCITOKENS,SSL"
Authentication = "NO"
CryptoMethodsList = "AES,BLOWFISH,3DES"
Enact = "YES"
Encryption = "NO"
Integrity = "NO"
IssuerKeys = "POOL, POOL.puppet-bak"
NegotiatedSession = true
RemoteVersion = "$CondorVersion: 10.9.0 2023-09-28 BuildID: 678228 PackageID: 10.9.0-1 $"
SessionDuration = "60"
SessionLease = 3600
TrustDomain = "hep.tau.ac.il"
03/02/24 06:25:26 SECMAN: received post-auth classad:
ReturnCode = "AUTHORIZED"
Sid = "tau-cm:12529:1709353526:9418902"
User = "unauthenticated@unmapped"
ValidCommands = "60007,457,60020,68,5,6,7,9,12,43,20,46,78,50,56,48,71,74"
03/02/24 06:25:26 SECMAN: policy to be cached:
AuthMethods = "FS"
AuthMethodsList = "FS,TOKEN,TOKEN,SCITOKENS,SSL"
Authentication = "NO"
Command = 6
ConnectSinful = "<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>"
CryptoMethodsList = "AES,BLOWFISH,3DES"
Enact = "YES"
Encryption = "NO"
Integrity = "NO"
IssuerKeys = "POOL, POOL.puppet-bak"
MyRemoteUserName = "unauthenticated@unmapped"
NegotiatedSession = true
OutgoingNegotiation = "PREFERRED"
RemoteVersion = "$CondorVersion: 10.9.0 2023-09-28 BuildID: 678228 PackageID: 10.9.0-1 $"
SessionDuration = "60"
SessionLease = 3600
Sid = "tau-cm:12529:1709353526:9418902"
Subsystem = "TOOL"
TrackState = true
TrustDomain = "hep.tau.ac.il"
UseSession = "YES"
User = "unauthenticated@unmapped"
ValidCommands = "60007,457,60020,68,5,6,7,9,12,43,20,46,78,50,56,48,71,74"
03/02/24 06:25:26 SESSION: client checking key type: -1
03/02/24 06:25:26 SECMAN: added session tau-cm:12529:1709353526:9418902 to cache for 60 seconds (3600s lease).
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<60007>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<457>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<60020>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<68>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<5>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<6>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<7>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<9>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<12>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<43>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<20>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<46>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<78>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<50>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<56>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<48>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<71>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: command {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<74>} mapped to session tau-cm:12529:1709353526:9418902.
03/02/24 06:25:26 SECMAN: startCommand succeeded.
03/02/24 06:25:26 Authorizing server 'unauthenticated@unmapped/192.114.100.129'.
03/02/24 06:25:26 SECMAN: command 519 QUERY_JOB_ADS_WITH_AUTH to schedd at <192.114.100.130:9618> from TCP port 3031 (blocking).
03/02/24 06:25:26 Filtering authentication methods (FS,TOKEN,SCITOKENS,SSL,IDTOKENS,PASSWORD) prior to offering them remotely.
03/02/24 06:25:26 Can try token auth because we have at least one named credential.
03/02/24 06:25:26 Will try IDTOKENS auth.
03/02/24 06:25:26 Can try token auth because we have at least one named credential.
03/02/24 06:25:26 Will try IDTOKENS auth.
03/02/24 06:25:26 Inserting pre-auth metadata for TOKEN.
03/02/24 06:25:26 Inserting pre-auth metadata for TOKEN.
03/02/24 06:25:26 SECMAN: no cached key for {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<519>}.
03/02/24 06:25:26 SECMAN: Security Policy:
AuthMethods = "FS,TOKEN,SCITOKENS,SSL,TOKEN,PASSWORD"
Authentication = "OPTIONAL"
CryptoMethods = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BEZY2obAJ19KaFg9XXS3+ugAs9Z0bRblNY+pBB4022ehqiLjqU4b6eqJ7l1z/UY+kIy9FUKDkNyqbFaHhjiVlI0="
Enact = "NO"
Encryption = "OPTIONAL"
Integrity = "OPTIONAL"
IssuerKeys = "POOL, POOL.puppet-bak"
NegotiatedSession = true
NewSession = "YES"
OutgoingNegotiation = "PREFERRED"
ServerPid = 1135683
SessionDuration = "60"
SessionLease = 3600
Subsystem = "TOOL"
TrustDomain = "hep.tau.ac.il"
03/02/24 06:25:26 SECMAN: negotiating security for command 519.
03/02/24 06:25:26 SECMAN: sending DC_AUTHENTICATE command
03/02/24 06:25:26 SECMAN: sending following classad:
AuthMethods = "FS,TOKEN,SCITOKENS,SSL,TOKEN,PASSWORD"
Authentication = "OPTIONAL"
Command = 519
ConnectSinful = "<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>"
CryptoMethods = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BEZY2obAJ19KaFg9XXS3+ugAs9Z0bRblNY+pBB4022ehqiLjqU4b6eqJ7l1z/UY+kIy9FUKDkNyqbFaHhjiVlI0="
Enact = "NO"
Encryption = "OPTIONAL"
Integrity = "OPTIONAL"
IssuerKeys = "POOL, POOL.puppet-bak"
NegotiatedSession = true
NewSession = "YES"
OutgoingNegotiation = "PREFERRED"
RemoteVersion = "$CondorVersion: 10.9.0 2023-09-28 BuildID: 678228 PackageID: 10.9.0-1 $"
ServerPid = 1135683
SessionDuration = "60"
SessionLease = 3600
Subsystem = "TOOL"
TrustDomain = "hep.tau.ac.il"
03/02/24 06:25:26 SECMAN: server responded with:
AuthMethods = "FS"
AuthMethodsList = "FS,TOKEN,TOKEN,SCITOKENS,SSL"
Authentication = "YES"
CryptoMethods = "AES"
CryptoMethodsList = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BCJZ9sY8BIN37bzcYWEZhgJpaSfD4fEW7VZlQ6xRExbt91vyCooyeEqWANcpNFUI6GJwOM5D1uDqPPgb/VqrJqk="
Enact = "YES"
Encryption = "YES"
Integrity = "YES"
IssuerKeys = "POOL, POOL.puppet-bak"
NegotiatedSession = true
RemoteVersion = "$CondorVersion: 10.9.0 2023-09-28 BuildID: 678228 PackageID: 10.9.0-1 $"
SessionDuration = "60"
SessionLease = 3600
TrustDomain = "hep.tau.ac.il"
03/02/24 06:25:26 SECMAN: new session, doing initial authentication.
03/02/24 06:25:26 SECMAN: authenticating RIGHT NOW.
03/02/24 06:25:26 SECMAN: AuthMethodsList: FS,TOKEN,TOKEN,SCITOKENS,SSL
03/02/24 06:25:26 SECMAN: Auth methods: FS,TOKEN,TOKEN,SCITOKENS,SSL
03/02/24 06:25:26 AUTHENTICATE: setting timeout for <192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e> to 20.
03/02/24 06:25:26 AUTHENTICATE: in authenticate( addr == '<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>', methods == 'FS,TOKEN,TOKEN,SCITOKENS,SSL'
)
03/02/24 06:25:26 AUTHENTICATE: can still try these methods: FS,TOKEN,TOKEN,SCITOKENS,SSL
03/02/24 06:25:26 HANDSHAKE: in handshake(my_methods = 'FS,TOKEN,TOKEN,SCITOKENS,SSL')
03/02/24 06:25:26 HANDSHAKE: handshake() - i am the client
03/02/24 06:25:26 Setting SciTokens cache directory to /var/run/condor/cache
03/02/24 06:25:26 HANDSHAKE: sending (methods == 6404) to server
03/02/24 06:25:26 HANDSHAKE: server replied (method = 4)
03/02/24 06:25:26 AUTHENTICATE: will try to use 4 (FS)
03/02/24 06:25:26 AUTHENTICATE: do_authenticate is 1.
03/02/24 06:25:26 AUTHENTICATE_FS: used dir /tmp/FS_XXXzvi6oY, status: 1
03/02/24 06:25:26 AUTHENTICATE: auth_status == 4 (FS)
03/02/24 06:25:26 Authentication was a Success.
03/02/24 06:25:26 AUTHENTICATION: setting default map to (null)
03/02/24 06:25:26 AUTHENTICATION: post-map: current user is '(null)'
03/02/24 06:25:26 AUTHENTICATION: post-map: current domain is '(null)'
03/02/24 06:25:26 AUTHENTICATION: post-map: current FQU is '(null)'
03/02/24 06:25:26 AUTHENTICATE: Exchanging keys with remote side.
03/02/24 06:25:26 AUTHENTICATE: Result of end of authenticate is 1.
03/02/24 06:25:26 SECMAN: generating AES key for session with schedd at <192.114.100.130:9618>...
03/02/24 06:25:26 SECMAN: about to enable encryption.
03/02/24 06:25:26 CRYPTO: New crypto state with protocol AES
03/02/24 06:25:26 SECMAN: successfully enabled encryption!
03/02/24 06:25:26 SECMAN: about to enable message authenticator with key type 3
03/02/24 06:25:26 SECMAN: because protocal is AES, not using other MAC.
03/02/24 06:25:26 SECMAN: successfully enabled message authenticator!
03/02/24 06:25:26 SECMAN: received post-auth classad:
ReturnCode = "AUTHORIZED"
Sid = "tau-htc:777771:1709353526:23785"
TriedAuthentication = true
User = "condor@xxxxxxxxxxxxx"
ValidCommands = "60007,457,60020,443,441,6,12,5,515,516,519,540,1111"
03/02/24 06:25:26 SECMAN: policy to be cached:
AuthMethods = "FS"
AuthMethodsList = "FS,TOKEN,TOKEN,SCITOKENS,SSL"
Authentication = "YES"
Command = 519
ConnectSinful = "<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>"
CryptoMethods = "AES"
CryptoMethodsList = "AES,BLOWFISH,3DES"
Enact = "YES"
Encryption = "YES"
Integrity = "YES"
IssuerKeys = "POOL, POOL.puppet-bak"
MyRemoteUserName = "condor@xxxxxxxxxxxxx"
NegotiatedSession = true
OutgoingNegotiation = "PREFERRED"
RemoteVersion = "$CondorVersion: 10.9.0 2023-09-28 BuildID: 678228 PackageID: 10.9.0-1 $"
SessionDuration = "60"
SessionLease = 3600
Sid = "tau-htc:777771:1709353526:23785"
Subsystem = "TOOL"
TrackState = true
TriedAuthentication = true
TrustDomain = "hep.tau.ac.il"
UseSession = "YES"
User = "unauthenticated@unmapped"
ValidCommands = "60007,457,60020,443,441,6,12,5,515,516,519,540,1111"
03/02/24 06:25:26 SESSION: client checking key type: 3
03/02/24 06:25:26 SESSION: fallback crypto method would be BLOWFISH.
03/02/24 06:25:26 SESSION: found list: AES,BLOWFISH,3DES.
03/02/24 06:25:26 SESSION: client duplicated AES to BLOWFISH key for UDP.
03/02/24 06:25:26 SECMAN: added session tau-htc:777771:1709353526:23785 to cache for 60 seconds (3600s lease).
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<60007>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<457>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<60020>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<443>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<441>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<6>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<12>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<5>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<515>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<516>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<519>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<540>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: command {<192.114.100.130:9618?addrs=192.114.100.130-9618+[2001-bf8-900-c-2--82]-9618&alias=tau-htc.hep.tau.ac.il&noUDP&sock=schedd_777728_4c3e>,<1111>} mapped to session tau-htc:777771:1709353526:23785.
03/02/24 06:25:26 SECMAN: startCommand succeeded.
03/02/24 06:25:26 Authorizing server 'unauthenticated@unmapped/192.114.100.130'.


-- Schedd: tau-htc.hep.tau.ac.il : <192.114.100.130:9618?... @ 03/02/24 06:25:26
OWNER BATCH_NAME      SUBMITTED   DONE   RUN    IDLE   HOLD  TOTAL JOB_IDS

Total for query: 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended  
Total for condor: 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended  
Total for all users: 825 jobs; 825 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended


On Sat, Mar 2, 2024 at 12:45âAM Jaime Frey <jfrey@xxxxxxxxxxx> wrote:
Part of the problem here is that the default security setup changed between 8.8 and 10.0 (to be more secure by default). For some existing pools, that may require some tweaks to their configuration files. The HTCondor-CEâs configuration may need similar tweaks.

Can you run the following command as well:
condor_q -pool tau-cm.hep.tau.ac.il:9618 -name tau-htc.hep.tau.ac.il -d:D_SECURITY:2

That should help me determine what you need to change to fix the failures.

 - Jaime

On Feb 29, 2024, at 8:56âPM, David Cohen <cdavid@xxxxxxxxxxxxxxxxxxxxxx> wrote:

Hi Jaime,
Please see the output bellow:
tau-htc ~]# condor_ce_q -pool tau-cm.hep.tau.ac.il:9618 -name tau-htc.hep.tau.ac.il -debug:D_SECURITY:2
03/01/24 04:51:16 KEYCACHE: created: 0x1fa8210
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission ALLOW
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission READ
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission WRITE
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission NEGOTIATOR
03/01/24 04:51:16 ipverify: NEGOTIATOR optimized to deny everyone
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission ADMINISTRATOR
03/01/24 04:51:16 ipverify: ADMINISTRATOR optimized to deny everyone
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission CONFIG
03/01/24 04:51:16 ipverify: CONFIG optimized to deny everyone
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission DAEMON
03/01/24 04:51:16 ipverify: DAEMON optimized to deny everyone
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission SOAP
03/01/24 04:51:16 ipverify: SOAP optimized to deny everyone
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission DEFAULT
03/01/24 04:51:16 ipverify: DEFAULT optimized to deny everyone
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission CLIENT
03/01/24 04:51:16 IPVERIFY: allow CLIENT: * (from config value ALLOW_CLIENT)
03/01/24 04:51:16 IPVERIFY: deny CLIENT: anonymous@*, unmapped@* (from config value DENY_CLIENT)
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission ADVERTISE_STARTD
03/01/24 04:51:16 ipverify: ADVERTISE_STARTD optimized to deny everyone
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission ADVERTISE_SCHEDD
03/01/24 04:51:16 ipverify: ADVERTISE_SCHEDD optimized to deny everyone
03/01/24 04:51:16 IPVERIFY: Subsystem TOOL
03/01/24 04:51:16 IPVERIFY: Permission ADVERTISE_MASTER
03/01/24 04:51:16 ipverify: ADVERTISE_MASTER optimized to deny everyone
03/01/24 04:51:16 Initialized the following authorization table:
03/01/24 04:51:16 Authorizations yet to be resolved:
03/01/24 04:51:16 deny CLIENT:  anonymous@*/* unmapped@*/*
03/01/24 04:51:16 SECMAN: command 6 QUERY_SCHEDD_ADS to collector at <192.114.100.129:9618> from TCP port 24398 (blocking).
03/01/24 04:51:16 Filtering authentication methods (FS,TOKEN,SCITOKENS,SSL,IDTOKENS,PASSWORD) prior to offering them remotely.
03/01/24 04:51:16 Can try token auth because we have at least one named credential.
03/01/24 04:51:16 Will try IDTOKENS auth.
03/01/24 04:51:16 Can try token auth because we have at least one named credential.
03/01/24 04:51:16 Will try IDTOKENS auth.
03/01/24 04:51:16 Inserting pre-auth metadata for TOKEN.
03/01/24 04:51:16 Inserting pre-auth metadata for TOKEN.
03/01/24 04:51:16 SECMAN: no cached key for {<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>,<6>}.
03/01/24 04:51:16 SECMAN: Security Policy:
AuthMethods = "FS,TOKEN,SCITOKENS,SSL,TOKEN,PASSWORD"
Authentication = "REQUIRED"
CryptoMethods = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BFg9bf3LhfTFHhABkjSvHlpR7Zu9hyg5fkMDfldGaeFppyl/DGhjdvZmu7piW4bvxrmfwkiPxEKw1pC1DUxK+qY="
Enact = "NO"
Encryption = "REQUIRED"
Integrity = "REQUIRED"
IssuerKeys = "POOL"
NegotiatedSession = true
NewSession = "YES"
OutgoingNegotiation = "REQUIRED"
ServerPid = 943148
SessionDuration = "60"
SessionLease = 3600
Subsystem = "TOOL"
TrustDomain = "users.htcondor.org"
03/01/24 04:51:16 SECMAN: negotiating security for command 6.
03/01/24 04:51:16 SECMAN: sending DC_AUTHENTICATE command
03/01/24 04:51:16 SECMAN: sending following classad:
AuthMethods = "FS,TOKEN,SCITOKENS,SSL,TOKEN,PASSWORD"
Authentication = "REQUIRED"
Command = 6
ConnectSinful = "<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>"
CryptoMethods = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BFg9bf3LhfTFHhABkjSvHlpR7Zu9hyg5fkMDfldGaeFppyl/DGhjdvZmu7piW4bvxrmfwkiPxEKw1pC1DUxK+qY="
Enact = "NO"
Encryption = "REQUIRED"
Integrity = "REQUIRED"
IssuerKeys = "POOL"
NegotiatedSession = true
NewSession = "YES"
OutgoingNegotiation = "REQUIRED"
RemoteVersion = "$CondorVersion: 10.9.0 2023-09-28 BuildID: 678228 PackageID: 10.9.0-1 $"
ServerPid = 943148
SessionDuration = "60"
SessionLease = 3600
Subsystem = "TOOL"
TrustDomain = "users.htcondor.org"
03/01/24 04:51:16 SECMAN: server responded with:
AuthMethods = "FS"
AuthMethodsList = "FS,TOKEN,TOKEN,SCITOKENS,SSL"
Authentication = "YES"
CryptoMethods = "AES"
CryptoMethodsList = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BJ7lTON+wfXCPQrChtgWop2nBDpJ2ECeaRbRaqxsoBKjcelGqKWeYKi7VEh8UCC2D9UGW1sb+pXXAHhgtoLyyXw="
Enact = "YES"
Encryption = "YES"
Integrity = "YES"
IssuerKeys = "POOL, POOL.puppet-bak"
NegotiatedSession = true
RemoteVersion = "$CondorVersion: 10.9.0 2023-09-28 BuildID: 678228 PackageID: 10.9.0-1 $"
SessionDuration = "60"
SessionLease = 3600
TrustDomain = "hep.tau.ac.il"
03/01/24 04:51:16 SECMAN: new session, doing initial authentication.
03/01/24 04:51:16 SECMAN: authenticating RIGHT NOW.
03/01/24 04:51:16 SECMAN: AuthMethodsList: FS,TOKEN,TOKEN,SCITOKENS,SSL
03/01/24 04:51:16 SECMAN: Auth methods: FS,TOKEN,TOKEN,SCITOKENS,SSL
03/01/24 04:51:16 AUTHENTICATE: setting timeout for <192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il> to 20.
03/01/24 04:51:16 AUTHENTICATE: in authenticate( addr == '<192.114.100.129:9618?alias=tau-cm.hep.tau.ac.il>', methods == 'FS,TOKEN,TOKEN,SCITOKENS,SSL')
03/01/24 04:51:16 AUTHENTICATE: can still try these methods: FS,TOKEN,TOKEN,SCITOKENS,SSL
03/01/24 04:51:16 HANDSHAKE: in handshake(my_methods = 'FS,TOKEN,TOKEN,SCITOKENS,SSL')
03/01/24 04:51:16 HANDSHAKE: handshake() - i am the client
03/01/24 04:51:16 Setting SciTokens cache directory to /var/run/condor-ce/cache
03/01/24 04:51:16 HANDSHAKE: sending (methods == 6404) to server
03/01/24 04:51:16 HANDSHAKE: server replied (method = 4)
03/01/24 04:51:16 AUTHENTICATE: will try to use 4 (FS)
03/01/24 04:51:16 AUTHENTICATE: do_authenticate is 1.
03/01/24 04:51:16 AUTHENTICATE_FS: used dir /tmp/FS_XXXgveiLN, status: 0
03/01/24 04:51:16 AUTHENTICATE: method 4 (FS) failed.
03/01/24 04:51:16 AUTHENTICATE: can still try these methods: TOKEN,TOKEN,SCITOKENS,SSL
03/01/24 04:51:16 HANDSHAKE: in handshake(my_methods = 'TOKEN,TOKEN,SCITOKENS,SSL')
03/01/24 04:51:16 HANDSHAKE: handshake() - i am the client
03/01/24 04:51:16 HANDSHAKE: sending (methods == 6400) to server
03/01/24 04:51:16 HANDSHAKE: server replied (method = 2048)
03/01/24 04:51:16 Will use issuer hep.tau.ac.il for remote server.
03/01/24 04:51:16 AUTHENTICATE: will try to use 2048 (IDTOKENS)
03/01/24 04:51:16 AUTHENTICATE: do_authenticate is 1.
03/01/24 04:51:16 PW.
03/01/24 04:51:16 PW: getting name.
03/01/24 04:51:16 Looking for tokens in directory /etc/condor-ce/tokens.d for issuer hep.tau.ac.il
03/01/24 04:51:16 TOKEN: No token found.
03/01/24 04:51:16 PW: Failed to fetch a login name
03/01/24 04:51:16 PW: Generating ra.
03/01/24 04:51:16 PW: Client sending.
03/01/24 04:51:16 Client error: NULL in send?
03/01/24 04:51:16 Client sending: -1, 0(), 0
03/01/24 04:51:16 PW: Client receiving.
03/01/24 04:51:16 Server sent status indicating not OK.
03/01/24 04:51:16 PW: Client received ERROR from server, propagating
03/01/24 04:51:16 PW: CLient sending two.
03/01/24 04:51:16 In client_send_two.
03/01/24 04:51:16 Client error: don't know my own name?
03/01/24 04:51:16 Can't send null for random string.
03/01/24 04:51:16 Client error: I have no name?
03/01/24 04:51:16 Client sending: 0() 0 0
03/01/24 04:51:16 Sent ok.
03/01/24 04:51:16 AUTHENTICATE: method 2048 (IDTOKENS) failed.
03/01/24 04:51:16 AUTHENTICATE: can still try these methods: SCITOKENS,SSL
03/01/24 04:51:16 HANDSHAKE: in handshake(my_methods = 'SCITOKENS,SSL')
03/01/24 04:51:16 HANDSHAKE: handshake() - i am the client
03/01/24 04:51:16 HANDSHAKE: sending (methods == 4352) to server
03/01/24 04:51:16 HANDSHAKE: server replied (method = 4096)
03/01/24 04:51:16 AUTHENTICATE: will try to use 4096 (SCITOKENS)
03/01/24 04:51:16 AUTHENTICATE: do_authenticate is 1.
03/01/24 04:51:16 CAFILE:     '/etc/pki/tls/certs/ca-bundle.crt'
03/01/24 04:51:16 CADIR:      '/etc/grid-security/certificates'
03/01/24 04:51:16 CIPHERLIST: 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RS
A-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA
-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS'
03/01/24 04:51:16 SSL client host check: using host alias tau-cm.hep.tau.ac.il for peer 192.114.100.129
03/01/24 04:51:16 SSL Auth: No SciToken file provided
03/01/24 04:51:16 SSL Auth: SSL Authentication fails, terminating
03/01/24 04:51:16 AUTHENTICATE: method 4096 (SCITOKENS) failed.
03/01/24 04:51:16 AUTHENTICATE: can still try these methods: SSL
03/01/24 04:51:16 HANDSHAKE: in handshake(my_methods = 'SSL')
03/01/24 04:51:16 HANDSHAKE: handshake() - i am the client
03/01/24 04:51:16 HANDSHAKE: sending (methods == 256) to server
03/01/24 04:51:16 HANDSHAKE: server replied (method = 256)
03/01/24 04:51:16 AUTHENTICATE: will try to use 256 (SSL)
03/01/24 04:51:16 AUTHENTICATE: do_authenticate is 1.
03/01/24 04:51:16 CAFILE:     '/etc/pki/tls/certs/ca-bundle.crt'
03/01/24 04:51:16 CADIR:      '/etc/grid-security/certificates'
03/01/24 04:51:16 CERTFILE:   '/etc/grid-security/hostcert.pem'
03/01/24 04:51:16 KEYFILE:    '/etc/grid-security/hostkey.pem'
03/01/24 04:51:16 CIPHERLIST: 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RS
A-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA
-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS'
03/01/24 04:51:16 SSL client host check: using host alias tau-cm.hep.tau.ac.il for peer 192.114.100.129
03/01/24 04:51:16 SSL Auth: Trying to connect.
03/01/24 04:51:16 Tried to connect: -1
03/01/24 04:51:16 SSL Auth: SSL: trying to continue reading.
03/01/24 04:51:16 Round 1.
03/01/24 04:51:16 Send message (2).
03/01/24 04:51:16 Status (c: 2, s: 0)
03/01/24 04:51:16 SSL Auth: Trying to connect.
03/01/24 04:51:16 Tried to connect: -1
03/01/24 04:51:16 SSL Auth: SSL: trying to continue reading.
03/01/24 04:51:16 Round 2.
03/01/24 04:51:16 SSL Auth: Receive message.
03/01/24 04:51:16 Received message (2).
03/01/24 04:51:16 Status (c: 2, s: 2)
03/01/24 04:51:16 SSL Auth: Trying to connect.
03/01/24 04:51:16 -Error with certificate at depth: 1
03/01/24 04:51:16   issuer   = /O=condor/CN=hep.tau.ac.il
03/01/24 04:51:16   subject  = /O=condor/CN=hep.tau.ac.il
03/01/24 04:51:16   err 19:self signed certificate in certificate chain
03/01/24 04:51:16 Tried to connect: -1
03/01/24 04:51:16 SSL: library failure: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed
03/01/24 04:51:16 Round 3.
03/01/24 04:51:16 Send message (3).
03/01/24 04:51:16 Status (c: 3, s: 2)
03/01/24 04:51:16 SSL Auth: SSL Authentication failed
03/01/24 04:51:16 AUTHENTICATE: method 256 (SSL) failed.
03/01/24 04:51:16 AUTHENTICATE: can still try these methods:  
03/01/24 04:51:16 HANDSHAKE: in handshake(my_methods = '')
03/01/24 04:51:16 HANDSHAKE: handshake() - i am the client
03/01/24 04:51:16 HANDSHAKE: sending (methods == 0) to server
03/01/24 04:51:16 HANDSHAKE: server replied (method = 0)
03/01/24 04:51:16 AUTHENTICATE: no available authentication methods succeeded!
03/01/24 04:51:16 SECMAN: required authentication with collector at <192.114.100.129:9618> failed, so aborting command QUERY_SCHEDD_ADS.
03/01/24 04:51:16 ERROR: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using SSL|AUTHENTICATE:1004:Failed to authenticate using SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using
IDTOKENS|AUTHENTICATE:1004:Failed to authenticate using FS
Error: Couldn't contact the condor_collector on tau-cm.hep.tau.ac.il:9618.  

Extra Info: the condor_collector is a process that runs on the central  
manager of your Condor pool and collects the status of all the machines and  
jobs in the Condor pool. The condor_collector might not be running, it might  
be refusing to communicate with you, there might be a network problem, or  
there may be some other problem. Check with your system administrator to fix  
this problem.  

If you are the system administrator, check that the condor_collector is  
running on tau-cm.hep.tau.ac.il:9618, check the ALLOW/DENY configuration in  
your condor_config, and check the MasterLog and CollectorLog files in your  
log directory for possible clues as to why the condor_collector is not  
responding. Also see the Troubleshooting section of the manual.

On Thu, Feb 29, 2024 at 10:07âPM Jaime Frey via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:
Can you try running this command:
condor_ce_q -pool tau-cm.hep.tau.ac.il:9618 -name tau-htc.hep.tau.ac.il -d:D_SECURITY:2

This does the same query thatâs failing for the job router and should fail in the same way, with extra details.

 - Jaime

On Feb 26, 2024, at 1:14âAM, David Cohen <cdavid@xxxxxxxxxxxxxxxxxxxxxx> wrote:

Hi,
Last week the HTCondor was upgraded from 8.8 to 10.9 and HTCondor-CE from 5 to 6.
Since then I see in the CE /var/log/condor-ce/JobRouterLog:
2/26/24 09:05:22 Unable to find address of tau-htc.hep.tau.ac.il at tau-cm.hep.tau.ac.il:9618
02/26/24 09:05:22 JobRouter (src="" failed to remove dest job: Unable to find address of tau-htc.hep.tau.ac.il at tau-cm.hep.tau.ac.il:9618
02/26/24 09:05:22 JobRouter (src="" removing orphaned destination job with no matching source job.
02/26/24 09:05:22 SECMAN: required authentication with collector at <192.114.100.129:9618> failed, so aborting command QUERY_SCHEDD_ADS.
02/26/24 09:05:22 ERROR: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using SSL|AUTHENTICATE:1004:Failed to authenticate using SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using
IDTOKENS|AUTHENTICATE:1004:Failed to authenticate using FS
02/26/24 09:05:22 Unable to find address of tau-htc.hep.tau.ac.il at tau-cm.hep.tau.ac.il:9618
02/26/24 09:05:22 JobRouter (src="" failed to remove dest job: Unable to find address of tau-htc.hep.tau.ac.il at tau-cm.hep.tau.ac.il:9618
02/26/24 09:05:22 JobRouter (src="" removing orphaned destination job with no matching source job.

And on the Central manager /var/log/condor/CollectorLog:
02/26/24 09:10:18 DC_AUTHENTICATE: required authentication of 192.114.100.130 failed: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using SSL|AUTHENTICATE:1004:Failed to authenticate us
ing SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using IDTOKENS|AUTHENTICATE:1004:Failed to authenticate using FS|FS:1004:Unable to lstat(/tmp/FS_XXX8hoSrF)
02/26/24 09:10:18 DC_AUTHENTICATE: required authentication of 192.114.100.130 failed: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using SSL|AUTHENTICATE:1004:Failed to authenticate us
ing SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using IDTOKENS|AUTHENTICATE:1004:Failed to authenticate using FS|FS:1004:Unable to lstat(/tmp/FS_XXXJf0649)

Naturally no grid jobs are running and the cluster is idle.
Any ideas on what went wrong?

Thanks,
David


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/