Re: [HTCondor-users] Upgrade from 8.8.5 to 9.0.17

[Todd L Miller <tlmiller@xxxxxxxxxxx>]

> We are planning to upgrade the LTS htcondor version in our infra.

	Regarding your subject, the LTS version is now 10.0.0.

> I have gone through the documentation specifically about the steps of
> upgrading and HTCondor security. We are using Host Based security and
> flocking in our setup.

	I'm glad to hear that someone's been reading it! ;)

> - Default security setting is "use security:recommended_v9_0" which
> translates everything to user based instead of host based. Is
> recommended_v9_0 using TOKENID for authentication underneath?

	It is using IDTOKEN.

> - after starting the central manager, /etc/condor/passwords.d/POOL
> credential file is automatically generated, I had to copy this file on
> submit and worker nodes to make them communicate with the central manager
> node. This looks related to password based authentication. I don't see
> anything generated in /etc/condor/tokens.d/ for TOKENID based
> authentication? However I am able to create user token, is
> /etc/condor/passwords.d/POOL signing the JWT tokens? if yes, then in which
> scenario tokes.d directory will be used?

	For others following along at home: using `get_htcondor` will 
avoid this issue entirely.

	The file `/etc/condor/passwords.d/POOL` is confusingly/misleading 
named and implemented.  HTCondor creates it so that every CM has a 
"signing key", which is required to produce individual IDTOKENs. 
Additionally, if a daemon has access to the same signing key as another 
daemon, those two daemons will trust each other; the effect is the same
as the "pool password" method (hence the name).

	The idea for an IDTOKENS pool is that each machine in the pool is 
issued an IDTOKEN by the CM.  (There's some machinery available to make
the resulting distribution problem easier to solve if you trust your 
network.  Your daemon logs, prior to copying the signing key around, 
should have mentioned that they requested an IDTOKEN, which you could
have approved at the CM.)  This allows the daemons on that machine to
securely communicate with the CM.  Note that users -- that is, the real 
people submitting jobs -- can almost always get away with being 
authenticated with FS at the submit node(s).

	Things are a little more annoying for the admin -- note that 
daemons with IDTOKENS but not the signing key can't authenticate 
_incoming_ connections.  Most of the time, that won't matter; the 
necessary authentication will have been securely forwarded through the 
collector.  But for some administrative commands, you'll either need the 
signing key or a different authentication method.

	The `tokens.d` directory is used by HTCondor to store IDTOKENS for 
the daemons.

> - Also as mentioned in [1] with flock based conf, password authentication
> will not work instead TOKENID, I didn't understand how it helps? Even with
> recommended_v9_0 I see only a POOL file is created. This will also become
> challenging for flock based conf, as I need to copy the same POOL file from
> two different pools on a single submit node?

	The idea for using IDTOKENS with flocking is that if schedd A is 
flocking to CM B, then the administrator of CM B issues schedd A an 
IDTOKEN and sends it (out of band) to the administrator of schedd A, who 
copies into it into `tokens.d/`.  (This allows A to securely flock to B 
without involving any external authentication of either's identity.)

- ToddM