[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] IDTOKENS and cli



Hi Jaime,

I spent more time on this today. The condor_ping tool that Todd pointed me to is very helpful. While I am now able to get the schedd talking to the collector and unprivileged users can submit jobs, I still can't get the root user to query the pool from the schedd.

schedd# # condor_ping -type collector  -table all
         Instruction Authentication Encryption Integrity Decision Identity
ALLOW IDTOKENS AES AES ALLOW condor@xxxxxxxxxxxxxxxxxxxxxxxx READ FAIL FAIL FAIL FAIL FAIL (use -verbose for more info) WRITE FAIL FAIL FAIL FAIL FAIL (use -verbose for more info) NEGOTIATOR FAIL FAIL FAIL FAIL FAIL (use -verbose for more info) ADMINISTRATOR FAIL FAIL FAIL FAIL FAIL (use -verbose for more info) OWNER FAIL FAIL FAIL FAIL FAIL (use -verbose for more info) CONFIG FAIL FAIL FAIL FAIL FAIL (use -verbose for more info) DAEMON FAIL FAIL FAIL FAIL FAIL (use -verbose for more info) ADVERTISE_STARTD FAIL FAIL FAIL FAIL FAIL (use -verbose for more info) ADVERTISE_SCHEDD FAIL FAIL FAIL FAIL FAIL (use -verbose for more info) ADVERTISE_MASTER IDTOKENS AES AES ALLOW condor@xxxxxxxxxxxxxxxxxxxxxxxx

The logs on the collector show the authentication succeeding, but being denied access:

[...]
04/29/22 14:16:05 HANDSHAKE: handshake() - i am the server
04/29/22 14:16:05 HANDSHAKE: client sent (methods == 2048)
04/29/22 14:16:05 HANDSHAKE: i picked (method == 2048)
04/29/22 14:16:05 HANDSHAKE: client received (method == 2048)
04/29/22 14:16:05 PW.
04/29/22 14:16:05 Will return to DC to continue authentication..
04/29/22 14:16:05 PASSWORD: entered authenticate_continue, state==100
04/29/22 14:16:05 PW: Server receiving 1.
04/29/22 14:16:05 Received: 0, 31(condor@xxxxxxxxxxxxxxxxxxxxxxxx), 256
04/29/22 14:16:05 PW: Server fetching password.
04/29/22 14:16:05 getTokenSigningKey(): for id=POOL, pool=1 v84mode=0 reading /etc/condor/condor_cred
04/29/22 14:16:05 PW: Server generating rb.
04/29/22 14:16:05 PW: Server sending.
04/29/22 14:16:05 In server_send: 0.
04/29/22 14:16:05 Calculating hkt 'condor@xxxxxxxxxxxxxxxxxxxxxxxx' (31), 'condor_pool@xxxxxxxxxxxxxxxxxxxxxxxx' (36). 04/29/22 14:16:05 Server send 'condor@xxxxxxxxxxxxxxxxxxxxxxxx', 'condor_pool@xxxxxxxxxxxxxxxxxxxxxxxx', 256 256 20 04/29/22 14:16:05 PASSWORD: leaving authenticate_continue, state==101, return=2
04/29/22 14:16:05 AUTHENTICATE: auth would still block
04/29/22 14:16:05 Will return to DC to continue authentication..
04/29/22 14:16:05 PASSWORD: entered authenticate_continue, state==101
04/29/22 14:16:05 PW: Server receiving 2.
04/29/22 14:16:05 PW: Server checking hk.
04/29/22 14:16:05 In calculate_hk.
04/29/22 14:16:05 Setting session key.
04/29/22 14:16:05 Key length: 32
04/29/22 14:16:05 PW: Server set session key.
04/29/22 14:16:05 PASSWORD: leaving authenticate_continue, state==101, return=1
04/29/22 14:16:05 Authentication was a Success.
04/29/22 14:16:05 AUTHENTICATION: setting default map to condor@xxxxxxxxxxxxxxxxxxxxxxxx 04/29/22 14:16:05 AUTHENTICATION: post-map: current FQU is 'condor@xxxxxxxxxxxxxxxxxxxxxxxx'
04/29/22 14:16:05 AUTHENTICATE: Exchanging keys with remote side.
04/29/22 14:16:05 AUTHENTICATE: Result of end of authenticate is 1.
04/29/22 14:16:05 DC_AUTHENTICATE: authentication of 10.13.5.58 complete.
04/29/22 14:16:05 DC_AUTHENTICATE: encryption enabled for session ldas-condor:215706:1651259765:6878 04/29/22 14:16:05 DC_AUTHENTICATE: message authenticator enabled with key id ldas-condor:215706:1651259765:6878.
04/29/22 14:16:05 DC_AUTHENTICATE: Success.
04/29/22 14:16:05 DC_AUTHENTICATE: authentication of <10.13.5.58:5037> was successful but resulted in a limited authorization which did not include this command (60020 DC_NOP_READ), so aborting.
04/29/22 14:16:05 DC_AUTHENTICATE: Command not authorized, done!

...even though the collector is supposed to allow read from everyone:

# condor_config_val -dump ALLOW_
# Configuration from machine: ldas-condor.ldas.ligo-la.caltech.edu

# Parameters with names that match ALLOW_:
ALLOW_ADMIN_COMMANDS = true
ALLOW_ADMINISTRATOR = root@*, condor@*, condor_pool@*, schedd@$(UID_DOMAIN)
ALLOW_CLIENT = *
ALLOW_DAEMON = condor@*, condor_pool@*, schedd@$(UID_DOMAIN)
ALLOW_NEGOTIATOR = condor@*, condor_pool@*, schedd@$(UID_DOMAIN)
ALLOW_READ = *
ALLOW_WRITE = *
DAGMAN_ALLOW_ANY_NODE_NAME_CHARACTERS = false
DAGMAN_ALLOW_EVENTS = 114
FS_ALLOW_UNSAFE = false
NEGOTIATOR_ALLOW_QUOTA_OVERSUBSCRIPTION = true
SCHEDD_ALLOW_LATE_MATERIALIZE = true
SEC_SCITOKENS_ALLOW_EXTRA_SLASH = false
SUBMIT_ALLOW_GETENV = true
XEN_ALLOW_HARDWARE_VT_SUSPEND = false
[...]


After some strace digging, I found that the cli is loading the token /etc/condor/tokens.d/MASTER_auto_generated_token, which a jwt decoder confirms has limited privileges:

"scope": "condor:/ADVERTISE_MASTER",

A token that I created with 'condor_token_create' on the schedd has more privileges, but condor_ping seems to want to use these auth-generated tokens instead.

How can I allow the root user to default to using a token with more privileges?

--Mike

On 4/27/22 11:14, Jaime Frey wrote:
To allow users to query the collector (i.e. use condor_status) without a token, we suggest allowing the ANONYMOUS or CLAIMTOBE authentication method for the READ and CLIENT authorization levels. If you donât set the authentication methods, CLAIMTOBE is enabled by default. If you use the get_htcondor tool to configure your machines, ANONYMOUS is enabled.

If youâre setting the authentication methods explicitly on your config file, then try using these values:

SEC_READ_AUTHENTICATION_METHODS = IDTOKENS, FS, ANONYMOUS
SEC_CLIENT_AUTHENTICATION_METHODS = IDTOKENS, FS, ANONYMOUS

The -authz argument to condor_token_create specifies authorization levels, not command names. UPDATE_SCHEDD_AD, QUERY_STARTD_ADS, and UPDATE_AD_GENERIC are command names and thus wonât do anything in the token.
The authorization levels are described here: https://htcondor.readthedocs.io/en/latest/admin-manual/security.html#access-level-descriptions

  - Jaime

On Apr 25, 2022, at 4:46 PM, Michael Thomas <wart@xxxxxxxxxxx<mailto:wart@xxxxxxxxxxx>> wrote:

I'm still flailing about trying to get idtokens working on a new 9.0.12 schedd in my cluster.

I was finally able to get the schedd talking to the collector by issuing a new token with condor_token_create (and a long list of permissions), then signing it on the CM:

condor_token_create -identity schedd@xxxxxxxxxxxxxxxxxxxxxxxx<mailto:schedd@xxxxxxxxxxxxxxxxxxxxxxxx> -authz DAEMON -authz UPDATE_SCHEDD_AD -authz READ -authz WRITE -authz QUERY_STARTD_ADS -authz UPDATE_AD_GENERIC -authz ADMINISTRATOR

However, I'm unable to get some CLI commands to work.  Notably, the 'condor_status' command, when run as root, returns:

# condor_status
Error: communication error
SECMAN:2010:Received "DENIED" from server for user condor@xxxxxxxxxxxxxxxxxxxxxxxx<mailto:condor@xxxxxxxxxxxxxxxxxxxxxxxx> using method IDTOKENS.

The collector reports the following error:

04/25/22 16:41:03 DC_AUTHENTICATE: message authenticator enabled with key id ldas-condor:2734:1650922863:33339.
04/25/22 16:41:03 DC_AUTHENTICATE: Success.
04/25/22 16:41:03 DC_AUTHENTICATE: authentication of <10.13.5.58:20190> was successful but resulted in a limited authorization which did not include this command (5 QUERY_STARTD_ADS), so aborting.

...which seems odd considering that I explicitly included QUERY_STARTD_ADS in the token request above.


As a user, it won't even authenticate:

$ condor_status
Error: communication error
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using IDTOKENS
AUTHENTICATE:1004:Failed to authenticate using FS

Any suggestions on where to look next?

--Mike
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx<mailto:htcondor-users-request@xxxxxxxxxxx> with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/