Re: [HTCondor-users] Submitting different users jobs from the single remote user.

["Bockelman, Brian" <BBockelman@xxxxxxxxxxxxx>]

Hi Sergey,

Apologies - I was on vacation last week.  Answers inline.

> On Jun 25, 2019, at 11:08 AM, Sergey A. Komissarov <sergey.komissarov@xxxxxxxxxxxxxx> wrote:
> 
> Hello Brian,
> 
> Thank you, queue super user option works perfectly and seems exactly what wee need.
> Now I'm trying to add authentication for app-server user and stuck with token mechanism.

Do note the token mechanism was released in the latest developer release!  I am quite happy to have you try it out and apologize for any speed bumps in the meantime.

> 
> I have setup of 4 docker containers running in the same network: master, submit, worker and test container for remote submission.
> They all have htcondor 8.9.2 installed. Master, submit and worker uses pool password for authentication and the test container have token
> generated from those pool password (system in the test container does not have 'p7-server' user):
> 
>> user@459baa61ca5c:/$ condor_token_list 
>> Header: {"alg":"HS256","kid":"POOL"} Payload: {"iat":1561470288,"iss":"htcondor-manager","sub":"p7-server@xxxxxxxxxxxxxx"} File: /etc/condor/tokens.d/p7-server.token
> 
> All containers have the same condor-host and domain settings in config file (read, write, client and negotiate rights are allowed to '*'):
> 
>> CONDOR_HOST=htcondor-manager
>> DEFAULT_DOMAIN_NAME = htcondor.p7.da
>> UID_DOMAIN = htcondor.p7.da
>> TRUST_UID_DOMAIN = True
>> FILESYSTEM_DOMAIN = htcondor.p7.da
> 
> Since docker does not have any domain system by default I made two aliases for the master host: htcondor-manager and htcondor-manager.htcondor.p7.da . 
> Test system just have random hostname without domain name.  The problem is that test system can not find token in the token file:
> 
>> user@459baa61ca5c:/$ condor_submit -debug -file submit -remote htcondor-manager
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) SECMAN: new session, doing initial authentication.
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) SECMAN: Auth methods: TOKEN
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) AUTHENTICATE: setting timeout for <192.168.48.3:9618> to 20.
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: in handshake(my_methods = 'TOKEN')
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: handshake() - i am the client
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: sending (methods == 2048) to server
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_write(fd=3 collector at <192.168.48.3:9618>,,size=13,timeout=20,flags=0,non_blocking=0)
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_read(fd=3 collector at <192.168.48.3:9618>,,size=5,timeout=20,flags=0,non_blocking=0)
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_read(fd=3 collector at <192.168.48.3:9618>,,size=8,timeout=20,flags=0,non_blocking=0)
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: server replied (method = 2048)
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) PW.
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) PW: getting name.
>> 06/25/19 14:05:51 (fd:5) (pid:38) (D_SECURITY) TOKEN: Will use examine tokens found in /etc/condor/tokens.d/p7-server.token.
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_ALWAYS) TOKEN: No token found.
> 
> Full log is here: https://pastebin.com/AD1x6cd3
> 
> Test system user may read token file content and may ping both htcondor-manager and htcondor-manager.htcondor.p7.da .
> From the htcondor source code I guess that client is filed to match server hostname (or full hostname) with the certificate issuer.
> Could you point out how certificate issuer is related to condor-host and what can I do to debug this further?
> 

So, the token logic tries to match on two things.

1. "trust domain" for both the server and the client.  Verify it's the same for the client and server by examining the output of "condor_config_val TRUST_DOMAIN" on both sides.
2. The name of the key that signed the token on the client matches one key name in the server (in the output above, this is the value of "kid" -- currently "POOL").
  - Given that's the default key name, "POOL" (from the pool password), I doubt this is the problem.

Can you check to see if either is the problem?

Brian