Re: [HTCondor-users] job failed to submit to CE with SCIToken only

["Bockelman, Brian" <BBockelman@xxxxxxxxxxxxx>]

Hi Xiaowei,

This is a completely different response - the token was passed to the server (authenticated) but the authorization was denied.

You may need to ask the ETF admins to try the test I suggested; it could be something in their hostâs configuration.

Brian

Sent from my iPhone

> On Jun 14, 2023, at 3:06 AM, JIANG Xiaowei <jiangxw@xxxxxxxxxx> wrote:
> 
> ïHi Brian, Todd, Maarten,
> 
> Thanks to all of you! Following your suggestion, I did some test with scitoken on CERN's lxplus node.
> 
> Using a cms user's scitoken with scopes (compute.read), run the command Brain suggested and the submit command Maarten's suggested, got the same log:
> 
> 06/14/23 09:22:35 SECMAN: received post-auth classad:
> ReturnCode = "DENIED"
> Sid = "condorce02:80306:1686727354:6858"
> TriedAuthentication = true
> User = "lhcb048@xxxxxxxxxxxxxxxxxx"
> ValidCommands = "60007,457,60020,68,5,6,7,9,12,43,20,46,78,50,56,48,71,74"
> 06/14/23 09:22:35 SECMAN: FAILED: Received "DENIED" from server for user lhcb048@xxxxxxxxxxxxxxxxxx using method SCITOKENS.
> Error: communication error
> SECMAN:2010:Received "DENIED" from server for user lhcb048@xxxxxxxxxxxxxxxxxx using method SCITOKENS.
> Error: Couldn't contact the condor_collector on condorce02.ihep.ac.cn
> 
> The CE looks like successfully recoganize the token and mapping to the local user lhcb048 (it has improved better than my test before). And the allow_* and deny_* on CE side are (some configurations are temporary for debuging the issue):
> ALLOW_ADMIN_COMMANDS = true
> ALLOW_ADMINISTRATOR = $(SUPERUSERS)
> ALLOW_CLIENT = *
> ALLOW_DAEMON = $(FRIENDLY_DAEMONS)
> ALLOW_NEGOTIATOR = $(SUPERUSERS)
> ALLOW_OWNER = $(SUPERUSERS)
> ALLOW_READ = *
> ALLOW_WRITE = *
> COLLECTOR.ALLOW_ADVERTISE_MASTER = $(FRIENDLY_DAEMONS)
> COLLECTOR.ALLOW_ADVERTISE_SCHEDD = $(FRIENDLY_DAEMONS)
> COLLECTOR.ALLOW_ADVERTISE_STARTD = $(UNMAPPED_USERS), $(USERS)
> COLLECTOR.ALLOW_READ = *
> SCHEDD.ALLOW_NEGOTIATOR = condor@xxxxxxxxxxxxxxxxxxx/$(FULL_HOSTNAME)
> SCHEDD.ALLOW_WRITE = *
> SCHEDD_ALLOW_LATE_MATERIALIZE = true
> DENY_ADMINISTRATOR = anonymous@*, unmapped@*
> DENY_CLIENT = anonymous@*, unmapped@*
> DENY_DAEMON = anonymous@*, unmapped@*
> DENY_NEGOTIATOR = anonymous@*, unmapped@*
> DENY_OWNER = anonymous@*, unmapped@*
> DENY_WRITE = anonymous@*, unmapped@* */134.158.151.140 */31.147.202.178
> 
> I don't know if the log "SECMAN:2010:Received "DENIED" from server for user lhcb048@xxxxxxxxxxxxxxxxxx using method SCITOKENS" is related to my allow/deny policy or scitoken's scopes. Is it possible to fix the 'DENIED' problem on the CE side in this case?
> 
> Besides, I am asking the CMS friends to run the similar test on the ETF host.
> 
> Regards,
> Xiaowei
> 
> 
> 
>> -----ååéä-----
>> åää: "Bockelman, Brian" <BBockelman@xxxxxxxxxxxxx>
>> åéæé: 2023-06-14 09:47:47 (ææä)
>> æää: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
>> æé: 
>> äé: Re: [HTCondor-users] job failed to submit to CE with SCIToken only
>> 
>> Hi Xiaowei,
>> 
>> From the server-side logfile you share, the error is on the client side.  For both SSL/TLS and SCITOKENS authentication, the client sends a message that it's giving up prior to completing the SSL handshake.  Since it's that early, you can eliminate any current problems with the token itself or the authorization configuration.
>> 
>> I queried from a personal dev host and it seems to have given a reasonable response.
>> 
>> You may ask the administrator of etf-01.cern.ch to try sending you the output of the following:
>> 
>> _CONDOR_AUTH_SSL_CLIENT_CADIR=/etc/grid-security/certificates/ _CONDOR_SEC_CLIENT_AUTHENTICATION_METHODS=SCITOKENS _CONDOR_TOOL_DEBUG=D_SECURITY:2 condor_status -debug -pool condorce02.ihep.ac.cn:9619
>> 
>> and see if the client is producing more useful debug outputs at the higher logging level.
>> 
>> For example, if AUTH_SSL_CLIENT_CADIR is not set to /etc/grid-security/certificates (as suggested in Maarten's later link) then I can reproduce what you see rather easily.
>> 
>> Brian
>> 
>>>> On Jun 13, 2023, at 5:17 AM, JIANG Xiaowei <jiangxw@xxxxxxxxxx> wrote:
>>> 
>>> Dear Experts, 
>>> 
>>> I am facing a wierd problem that the cms sam job can not be submitted to our CE with only SCIToken.  
>>> On sam schedd side, there are some errors like [1]. 
>>> On my CE collector, the CollectorLog is posted in the attachment and no clue in SchedLog. 
>>> The related configurations are like: 
>>> [root@condorce02 config.d]# cat /etc/condor-ce/mapfiles.d/10-scitokens.conf
>>> # CMS SAM ##
>>> SCITOKENS /^https\:\/\/cms-auth\.web\.cern\.ch\/,08ca855e-d715-410e-a6ff-ad77306e1763$/ cmssgm006
>>> ## ATLAS SAM ##
>>> SCITOKENS /^https:\/\/atlas-auth\.web\.cern\.ch\/,5c5d2a4d-9177-3efa-912f-1b4e5c9fb660$/ atlassgm007
>>> [root@condorce02 config.d]# condor_ce_config_val -dump Collector.SEC
>>> COLLECTOR.SEC_ADVERTISE_STARTD_AUTHENTICATION_METHODS = FS,TOKEN,SCITOKENS,GSI,SSL
>>> COLLECTOR.SEC_READ_AUTHENTICATION_METHODS = FS,TOKEN,SCITOKENS,GSI,SSL
>>> COLLECTOR.SEC_WRITE_AUTHENTICATION_METHODS = FS,TOKEN,SCITOKENS,GSI,SSL
>>> The condor_versions are:  
>>> [root@condorce02 config.d]# condor_ce_version
>>> $HTCondorCEVersion: 5.1.6 $
>>> $CondorVersion: 9.0.17 May 27 2023 BuildID: 649540 PackageID: 9.0.17-3 $
>>> Hope to get help from your expert side! Thanks! 
>>> 
>>> Regards, 
>>> Xiaowei 
>>> 
>>> [1] -  
>>> 06/07/23 13:23:07 [117315] SECMAN: required authentication with collector at <202.122.33.23:9619> failed, so aborting command QUERY_SCHEDD_ADS. 06/07/23 13:23:07 [117315] ERROR: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using SSL|AUTHENTICATE:1004:Failed to authenticate using SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using IDTOKENS|AUTHENTICATE:1004:Failed to authenticate using FS 06/07/23 13:23:07 [117315] Error locating schedd condorce02.ihep.ac.cn 06/07/23 13:23:07 [117315] Can't find address of queue manager 06/07/23 13:23:07 [117315] Error connecting to schedd condorce02.ihep.ac.cn: <collector.log>
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>> 
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>> 
>> 
>> 
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>> 
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/