Mailing List Archives Public Access	UW Madison Computer Sciences Department Computer Systems Lab

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[HTCondor-users] job failed to submit to CE with SCIToken only

Date: Tue, 13 Jun 2023 18:17:27 +0800 (GMT+08:00)
From: JIANG Xiaowei <jiangxw@xxxxxxxxxx>
Subject: [HTCondor-users] job failed to submit to CE with SCIToken only

Dear Experts,

I am facing a wierd problem that the cms sam job can not be submitted to our CE with only SCIToken.

On sam schedd side, there are some errors like [1].

On my CE collector, the CollectorLog is posted in the attachment and no clue in SchedLog.

The related configurations are like:

[root@condorce02 config.d]# cat /etc/condor-ce/mapfiles.d/10-scitokens.conf

# CMS SAM ##
SCITOKENS /^https\:\/\/cms-auth\.web\.cern\.ch\/,08ca855e-d715-410e-a6ff-ad77306e1763$/ cmssgm006
## ATLAS SAM ##
SCITOKENS /^https:\/\/atlas-auth\.web\.cern\.ch\/,5c5d2a4d-9177-3efa-912f-1b4e5c9fb660$/ atlassgm007

[root@condorce02 config.d]# condor_ce_config_val -dump Collector.SEC
COLLECTOR.SEC_ADVERTISE_STARTD_AUTHENTICATION_METHODS = FS,TOKEN,SCITOKENS,GSI,SSL
COLLECTOR.SEC_READ_AUTHENTICATION_METHODS = FS,TOKEN,SCITOKENS,GSI,SSL
COLLECTOR.SEC_WRITE_AUTHENTICATION_METHODS = FS,TOKEN,SCITOKENS,GSI,SSL

The condor_versions are:

[root@condorce02 config.d]# condor_ce_version
$HTCondorCEVersion: 5.1.6 $
$CondorVersion: 9.0.17 May 27 2023 BuildID: 649540 PackageID: 9.0.17-3 $

Hope to get help from your expert side! Thanks!

Regards,

Xiaowei

[1] -

06/07/23 13:23:07 [117315] SECMAN: required authentication with collector at <202.122.33.23:9619> failed, so aborting command QUERY_SCHEDD_ADS. 06/07/23 13:23:07 [117315] ERROR: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using SSL|AUTHENTICATE:1004:Failed to authenticate using SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using IDTOKENS|AUTHENTICATE:1004:Failed to authenticate using FS 06/07/23 13:23:07 [117315] Error locating schedd condorce02.ihep.ac.cn 06/07/23 13:23:07 [117315] Can't find address of queue manager 06/07/23 13:23:07 [117315] Error connecting to schedd condorce02.ihep.ac.cn:

06/13/23 17:54:57 DC_AUTHENTICATE: received DC_AUTHENTICATE from <188.184.81.101:45985>
06/13/23 17:54:57 DC_AUTHENTICATE: received following ClassAd:
AuthMethods = "FS,TOKEN,KERBEROS,SCITOKENS,SSL,CLAIMTOBE"
Authentication = "REQUIRED"
Command = 6
ConnectSinful = "<202.122.33.23:9619?alias=condorce02.ihep.ac.cn>"
CryptoMethods = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BGDKiENXWoso32JgqTq16m02VHcT0wWpBLp5ZIwbrFFiTSHHE9XuSxvz0mGWTzRGcS3QTEsK6JTYGarRbHE3BB0="
Enact = "NO"
Encryption = "REQUIRED"
Integrity = "REQUIRED"
IssuerKeys = "POOL"
NegotiatedSession = true
NewSession = "YES"
OutgoingNegotiation = "REQUIRED"
ParentUniqueID = "etf-01:467255:1686650085"
RemoteVersion = "$CondorVersion: 10.0.3 2023-04-06 BuildID: 638290 PackageID: 10.0.3-1 $"
ServerCommandSock = "<188.184.81.101:49586?addrs=188.184.81.101-49586&alias=etf-01.cern.ch>"
ServerPid = 467256
SessionDuration = "1800"
SessionLease = 3600
Subsystem = "C_GAHP_WORKER_THREAD"
TrustDomain = "etf-01.cern.ch"
06/13/23 17:54:57 DC_AUTHENTICATE: our_policy:
AuthMethods = "FS,TOKEN,SCITOKENS,GSI,SSL"
Authentication = "OPTIONAL"
CryptoMethods = "AES,BLOWFISH,3DES"
Enact = "NO"
Encryption = "OPTIONAL"
Integrity = "OPTIONAL"
IssuerKeys = "POOL"
OutgoingNegotiation = "PREFERRED"
ParentUniqueID = "condorce02:80255:1686645361"
ServerPid = 80306
SessionDuration = "86400"
SessionLease = 3600
Subsystem = "COLLECTOR"
TrustDomain = "condorce02.ihep.ac.cn:9619"
06/13/23 17:54:57 DC_AUTHENTICATE: the_policy:
AuthMethods = "FS"
AuthMethodsList = "FS,TOKEN,SCITOKENS,SSL"
Authentication = "YES"
CryptoMethods = "AES,BLOWFISH,3DES"
CryptoMethodsList = "AES,BLOWFISH,3DES"
Enact = "YES"
Encryption = "YES"
Integrity = "YES"
IssuerKeys = "POOL"
SessionDuration = "1800"
SessionLease = 3600
TrustDomain = "condorce02.ihep.ac.cn:9619"
06/13/23 17:54:57 DC_AUTHENTICATE: generating AES-GCM key for session condorce02:80306:1686650097:489...
06/13/23 17:54:57 SECMAN: Sending following response ClassAd:
AuthMethods = "FS"
AuthMethodsList = "FS,TOKEN,SCITOKENS,SSL"
Authentication = "YES"
CryptoMethods = "AES"
CryptoMethodsList = "AES,BLOWFISH,3DES"
Enact = "YES"
Encryption = "YES"
Integrity = "YES"
IssuerKeys = "POOL"
RemoteVersion = "$CondorVersion: 9.0.17 May 27 2023 BuildID: 649540 PackageID: 9.0.17-3 $"
SessionDuration = "1800"
SessionLease = 3600
TrustDomain = "condorce02.ihep.ac.cn:9619"
06/13/23 17:54:57 SECMAN: new session, doing initial authentication.
06/13/23 17:54:57 Returning to DC while we wait for socket to authenticate.
06/13/23 17:54:57 DC_AUTHENTICATE: authenticating RIGHT NOW.
06/13/23 17:54:57 AUTHENTICATE: setting timeout for (unknown) to 20.
06/13/23 17:54:57 AUTHENTICATE: in authenticate( addr == '(unknown)', methods == 'FS,TOKEN,SCITOKENS,SSL')
06/13/23 17:54:57 AUTHENTICATE: can still try these methods: FS,TOKEN,SCITOKENS,SSL
06/13/23 17:54:57 HANDSHAKE: in handshake(my_methods = 'FS,TOKEN,SCITOKENS,SSL')
06/13/23 17:54:57 HANDSHAKE: handshake() - i am the server
06/13/23 17:54:57 HANDSHAKE: client sent (methods == 6404)
06/13/23 17:54:57 HANDSHAKE: i picked (method == 4)
06/13/23 17:54:57 HANDSHAKE: client received (method == 4)
06/13/23 17:54:57 AUTHENTICATE: will try to use 4 (FS)
06/13/23 17:54:57 AUTHENTICATE: do_authenticate is 1.
06/13/23 17:54:57 FS: client template is /tmp/FS_XXXXXXXXX
06/13/23 17:54:57 FS: client filename is /tmp/FS_XXXuhlvx7
06/13/23 17:54:57 Will return to DC because authentication is incomplete.
06/13/23 17:54:57 AUTHENTICATE_FS: used dir /tmp/FS_XXXuhlvx7, status: 0
06/13/23 17:54:57 AUTHENTICATE: do_authenticate is 0.
06/13/23 17:54:57 AUTHENTICATE: method 4 (FS) failed.
06/13/23 17:54:57 AUTHENTICATE: can still try these methods: FS,TOKEN,SCITOKENS,SSL
06/13/23 17:54:57 HANDSHAKE: in handshake(my_methods = 'FS,TOKEN,SCITOKENS,SSL')
06/13/23 17:54:57 AUTHENTICATE: handshake would block
06/13/23 17:54:57 Will return to DC to continue authentication..
06/13/23 17:54:58 HANDSHAKE: handshake() - i am the server
06/13/23 17:54:58 HANDSHAKE: client sent (methods == 6400)
06/13/23 17:54:58 HANDSHAKE: i picked (method == 2048)
06/13/23 17:54:58 HANDSHAKE: client received (method == 2048)
06/13/23 17:54:58 AUTHENTICATE: can still try these methods: FS,TOKEN,SCITOKENS,SSL
06/13/23 17:54:58 Will use issuer condorce02.ihep.ac.cn:9619 for remote server.
06/13/23 17:54:58 AUTHENTICATE: will try to use 2048 (IDTOKENS)
06/13/23 17:54:58 AUTHENTICATE: do_authenticate is 1.
06/13/23 17:54:58 PW.
06/13/23 17:54:58 Will return to DC to continue authentication..
06/13/23 17:54:58 PASSWORD: entered authenticate_continue, state==100
06/13/23 17:54:58 PW: Server receiving 1.
06/13/23 17:54:58 Received: -1, 0(), 0
06/13/23 17:54:58 PW: Server received ERROR from client, propagating
06/13/23 17:54:58 PW: Server sending.
06/13/23 17:54:58 In server_send: -1.
06/13/23 17:54:58 Server send '', '', 0 0 0
06/13/23 17:54:58 PASSWORD: leaving authenticate_continue, state==101, return=2
06/13/23 17:54:58 AUTHENTICATE: auth would still block
06/13/23 17:54:58 Will return to DC to continue authentication..
06/13/23 17:54:58 PASSWORD: entered authenticate_continue, state==101
06/13/23 17:54:58 PW: Server receiving 2.
06/13/23 17:54:58 Error from client.
06/13/23 17:54:58 PW: client in mode 2048 and ID (null).
06/13/23 17:54:58 PASSWORD: leaving authenticate_continue, state==101, return=0
06/13/23 17:54:58 AUTHENTICATE: do_authenticate is 0.
06/13/23 17:54:58 AUTHENTICATE: method 2048 (IDTOKENS) failed.
06/13/23 17:54:58 AUTHENTICATE: can still try these methods: FS,TOKEN,SCITOKENS,SSL
06/13/23 17:54:58 HANDSHAKE: in handshake(my_methods = 'FS,TOKEN,SCITOKENS,SSL')
06/13/23 17:54:58 HANDSHAKE: handshake() - i am the server
06/13/23 17:54:58 HANDSHAKE: client sent (methods == 4352)
06/13/23 17:54:58 HANDSHAKE: i picked (method == 4096)
06/13/23 17:54:58 HANDSHAKE: client received (method == 4096)
06/13/23 17:54:58 AUTHENTICATE: will try to use 4096 (SCITOKENS)
06/13/23 17:54:58 AUTHENTICATE: forcing do_authenticate to true.
06/13/23 17:54:58 AUTHENTICATE: do_authenticate is 1.
06/13/23 17:54:58 CADIR:      '/etc/grid-security/certificates'
06/13/23 17:54:58 CERTFILE:   '/etc/grid-security/hostcert.pem'
06/13/23 17:54:58 KEYFILE:    '/etc/grid-security/hostkey.pem'
06/13/23 17:54:58 CIPHERLIST: 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS'
06/13/23 17:54:58 Will return to DC to continue authentication..
06/13/23 17:54:58 SSL Auth: Trying to accept.
06/13/23 17:54:58 Accept returned -1.
06/13/23 17:54:58 SSL Auth: SSL: trying to continue reading.
06/13/23 17:54:58 Round 0.
06/13/23 17:54:58 SSL Auth: Receive message.
06/13/23 17:54:58 Received message (2).
06/13/23 17:54:58 Status (c: 2, s: 2)
06/13/23 17:54:58 SSL Auth: Trying to accept.
06/13/23 17:54:58 Accept returned -1.
06/13/23 17:54:58 SSL Auth: SSL: trying to continue reading.
06/13/23 17:54:58 Round 1.
06/13/23 17:54:58 Send message (2).
06/13/23 17:54:58 Status (c: 2, s: 2)
06/13/23 17:54:58 SSL Auth: Trying to accept.
06/13/23 17:54:58 Accept returned -1.
06/13/23 17:54:58 SSL Auth: SSL: trying to continue reading.
06/13/23 17:54:58 Round 2.
06/13/23 17:54:58 SSL Auth: Would block when trying to receive message
06/13/23 17:54:58 AUTHENTICATE: auth would still block
06/13/23 17:54:58 Will return to DC to continue authentication..
06/13/23 17:54:58 SSL Auth: Trying to accept.
06/13/23 17:54:58 Accept returned -1.
06/13/23 17:54:58 SSL Auth: SSL: trying to continue reading.
06/13/23 17:54:58 Round 2.
06/13/23 17:54:58 SSL Auth: Receive message.
06/13/23 17:54:58 Received message (3).
06/13/23 17:54:58 Status (c: 3, s: 2)
06/13/23 17:54:58 SSL Auth: SSL Authentication failed
06/13/23 17:54:58 AUTHENTICATE: do_authenticate is 0.
06/13/23 17:54:58 AUTHENTICATE: method 4096 (SCITOKENS) failed.
06/13/23 17:54:58 AUTHENTICATE: can still try these methods: FS,TOKEN,SCITOKENS,SSL
06/13/23 17:54:58 HANDSHAKE: in handshake(my_methods = 'FS,TOKEN,SCITOKENS,SSL')
06/13/23 17:54:58 AUTHENTICATE: handshake would block
06/13/23 17:54:58 Will return to DC to continue authentication..
06/13/23 17:54:58 HANDSHAKE: handshake() - i am the server
06/13/23 17:54:58 HANDSHAKE: client sent (methods == 256)
06/13/23 17:54:58 HANDSHAKE: i picked (method == 256)
06/13/23 17:54:58 HANDSHAKE: client received (method == 256)
06/13/23 17:54:58 AUTHENTICATE: can still try these methods: FS,TOKEN,SCITOKENS,SSL
06/13/23 17:54:58 AUTHENTICATE: will try to use 256 (SSL)
06/13/23 17:54:58 AUTHENTICATE: do_authenticate is 1.
06/13/23 17:54:58 CADIR:      '/etc/grid-security/certificates'
06/13/23 17:54:58 CERTFILE:   '/etc/grid-security/hostcert.pem'
06/13/23 17:54:58 KEYFILE:    '/etc/grid-security/hostkey.pem'
06/13/23 17:54:58 CIPHERLIST: 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS'
06/13/23 17:54:58 Will return to DC to continue authentication..
06/13/23 17:54:58 SSL Auth: Trying to accept.
06/13/23 17:54:58 Accept returned -1.
06/13/23 17:54:58 SSL Auth: SSL: trying to continue reading.
06/13/23 17:54:58 Round 0.
06/13/23 17:54:58 SSL Auth: Receive message.
06/13/23 17:54:58 Received message (2).
06/13/23 17:54:58 Status (c: 2, s: 2)
06/13/23 17:54:58 SSL Auth: Trying to accept.
06/13/23 17:54:58 Accept returned -1.
06/13/23 17:54:58 SSL Auth: SSL: trying to continue reading.
06/13/23 17:54:58 Round 1.
06/13/23 17:54:58 Send message (2).
06/13/23 17:54:58 Status (c: 2, s: 2)
06/13/23 17:54:58 SSL Auth: Trying to accept.
06/13/23 17:54:58 Accept returned -1.
06/13/23 17:54:58 SSL Auth: SSL: trying to continue reading.
06/13/23 17:54:58 Round 2.
06/13/23 17:54:58 SSL Auth: Would block when trying to receive message
06/13/23 17:54:58 AUTHENTICATE: auth would still block
06/13/23 17:54:58 Will return to DC to continue authentication..
06/13/23 17:54:59 SSL Auth: Trying to accept.
06/13/23 17:54:59 Accept returned -1.
06/13/23 17:54:59 SSL Auth: SSL: trying to continue reading.
06/13/23 17:54:59 Round 2.
06/13/23 17:54:59 SSL Auth: Receive message.
06/13/23 17:54:59 Received message (3).
06/13/23 17:54:59 Status (c: 3, s: 2)
06/13/23 17:54:59 SSL Auth: SSL Authentication failed
06/13/23 17:54:59 AUTHENTICATE: do_authenticate is 0.
06/13/23 17:54:59 AUTHENTICATE: method 256 (SSL) failed.
06/13/23 17:54:59 AUTHENTICATE: can still try these methods: FS,TOKEN,SCITOKENS,SSL
06/13/23 17:54:59 HANDSHAKE: in handshake(my_methods = 'FS,TOKEN,SCITOKENS,SSL')
06/13/23 17:54:59 AUTHENTICATE: handshake would block
06/13/23 17:54:59 Will return to DC to continue authentication..
06/13/23 17:54:59 HANDSHAKE: handshake() - i am the server
06/13/23 17:54:59 HANDSHAKE: client sent (methods == 0)
06/13/23 17:54:59 HANDSHAKE: i picked (method == 0)
06/13/23 17:54:59 HANDSHAKE: client received (method == 0)
06/13/23 17:54:59 AUTHENTICATE: can still try these methods: FS,TOKEN,SCITOKENS,SSL
06/13/23 17:54:59 AUTHENTICATE: no available authentication methods succeeded!
06/13/23 17:54:59 DC_AUTHENTICATE: required authentication of 188.184.81.101 failed: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using SSL|AUTHENTICATE:1004:Failed to authenticate using SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using IDTOKENS|AUTHENTICATE:1004:Failed to authenticate using FS|FS:1004:Unable to lstat(/tmp/FS_XXXuhlvx7)

Follow-Ups:
- Re: [HTCondor-users] job failed to submit to CE with SCIToken only
  - From: Bockelman, Brian
- Re: [HTCondor-users] job failed to submit to CE with SCIToken only
  - From: Maarten Litmaath

Prev by Date: Re: [HTCondor-users] DiskUsage
Next by Date: Re: [HTCondor-users] job failed to submit to CE with SCIToken only
Previous by thread: Re: [HTCondor-users] Condor user priority query
Next by thread: Re: [HTCondor-users] job failed to submit to CE with SCIToken only
Index(es):
- Date
- Thread

Mailing List Archives

Public Access

[HTCondor-users] job failed to submit to CE with SCIToken only