Re: [HTCondor-users] Submitting different users jobs from the single remote user.

["Sergey A. Komissarov" <sergey.komissarov@xxxxxxxxxxxxxx>]

Hello Brian,

I added 'ALL_DEBUG = D_ALL D_FULLDEBUG' to the client config, here is the log: https://pastebin.com/BadKNXnL .

----------
Sergey Komissarov
Senior Software Developer
DATADVANCE

This message may contain confidential information
constituting a trade secret of DATADVANCE. Any distribution,
use or copying of the information contained in this
message is ineligible except under the internal
regulations of DATADVANCE and may entail liability in
accordance with the current legislation of the Russian
Federation. If you have received this message by mistake
please immediately inform me of it. Thank you!



----- Original Message -----
From: "Brian Bockelman" <BBockelman@xxxxxxxxxxxxx>
To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
Sent: Wednesday, July 3, 2019 2:34:51 PM
Subject: Re: [HTCondor-users] Submitting different	users	jobs	from	the	single	remote user.

Hi Sergey,

Your setup looks correct.  However, thereâs at least one detail missing in the log that might help debug.  Can you please post the exact same info but also adding the D_FULLDEBUG setting?

Sorry for the delay,

Brian

Sent from my iPhone

> On Jul 2, 2019, at 9:32 AM, Sergey A. Komissarov <sergey.komissarov@xxxxxxxxxxxxxx> wrote:
> 
> Hello Brian,
> 
> Settings from the condor master host (I'm skipping default values):
> 
> CONDOR_HOST = htcondor-manager.htcondor.p7.da
> FULL_HOSTNAME = htcondor-manager.htcondor.p7.da
> HOSTNAME = htcondor-manager
> 
> DEFAULT_DOMAIN_NAME = htcondor.p7.da
> TRUST_DOMAIN = htcondor.p7.da
> TRUST_UID_DOMAIN = True
> UID_DOMAIN = htcondor.p7.da
> FILESYSTEM_DOMAIN = htcondor.p7.da
> 
> The pool password and token generated with the following commands:
> 
> condor_store_cred -f /var/lock/condor/pool_password -p testpassword
> condor_token_create -identity p7-server@xxxxxxxxxxxxxx -token p7-server
> 
> condor_token_list outputs the same in the test and in the master containers:
> Header: {"alg":"HS256","kid":"POOL"} Payload: {"iat":1562073511,"iss":"htcondor.p7.da","sub":"p7-server@xxxxxxxxxxxxxx"} File: /etc/condor/tokens.d/p7-server
> 
> Test container settings:
> 
> CONDOR_HOST = htcondor-manager.htcondor.p7.da
> FULL_HOSTNAME = 34790c54e50c.htcondor.p7.da
> HOSTNAME = 34790c54e50c
> 
> DEFAULT_DOMAIN_NAME = htcondor.p7.da
> TRUST_DOMAIN = htcondor.p7.da
> TRUST_UID_DOMAIN = True
> UID_DOMAIN = htcondor.p7.da
> FILESYSTEM_DOMAIN = htcondor.p7.da
> 
> I added CLAIMTOBE as the second authentication method in the test container config and was able to get schedd data via python api, but it still fails to
> find suitable token in the token file:
> 
>> import htcondor
>> htcondor.enable_debug()
>> htcondor.Collector().locateAll(htcondor.DaemonTypes.Schedd)
> 07/02/19 13:53:04 (fd:12) (pid:263) (D_HOSTNAME) New Daemon obj (collector) name: "NULL", pool: "NULL", addr: "<192.168.96.5:9618>"
> 07/02/19 13:53:04 (fd:12) (pid:263) (D_HOSTNAME) Address "<192.168.96.5:9618>" specified but no name, looking up host info
> 07/02/19 13:53:04 (fd:12) (pid:263) (D_HOSTNAME) resolve_hostname_raw(): argument 'tests_htcondor-manager_1.p7-2k_devcontainer_devenv' is not a valid DNS name, returning no addresses.
> 07/02/19 13:53:04 (fd:12) (pid:263) (D_ALWAYS) WARNING: forward resolution of tests_htcondor-manager_1.p7-2k_devcontainer_devenv doesn't match 192.168.96.5!
> 07/02/19 13:53:04 (fd:13) (pid:263) (D_SECURITY) HANDSHAKE: server replied (method = 2048)
> 07/02/19 13:53:04 (fd:13) (pid:263) (D_SECURITY) PW.
> 07/02/19 13:53:04 (fd:13) (pid:263) (D_SECURITY) PW: getting name.
> 07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) TOKEN: Will use examine tokens found in /etc/condor/tokens.d/p7-server.
> 07/02/19 13:53:04 (fd:11) (pid:263) (D_ALWAYS) TOKEN: No token found.
> 07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) PW: Failed to fetch a login name
> 07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) AUTHENTICATE: method 2048 (TOKEN) failed.
> 07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) HANDSHAKE: server replied (method = 2)
> 07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) Authentication was a Success.
> 
> Full output is here: https://pastebin.com/YzJ55tCu . 
> For now I may use CLAIMTOBE authentication for further tests and integration development and hope token authentication will be a part of stable release before our own release =)
> 
> ----------
> Sergey Komissarov
> Senior Software Developer
> DATADVANCE
> 
> This message may contain confidential information
> constituting a trade secret of DATADVANCE. Any distribution,
> use or copying of the information contained in this
> message is ineligible except under the internal
> regulations of DATADVANCE and may entail liability in
> accordance with the current legislation of the Russian
> Federation. If you have received this message by mistake
> please immediately inform me of it. Thank you!
> 
> 
> 
> ----- Original Message -----
> From: "Brian Bockelman" <BBockelman@xxxxxxxxxxxxx>
> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
> Sent: Tuesday, July 2, 2019 5:53:53 AM
> Subject: Re: [HTCondor-users] Submitting different users jobs    from    the    single    remote user.
> 
> Hi Sergey,
> 
> Apologies - I was on vacation last week.  Answers inline.
> 
>> On Jun 25, 2019, at 11:08 AM, Sergey A. Komissarov <sergey.komissarov@xxxxxxxxxxxxxx> wrote:
>> 
>> Hello Brian,
>> 
>> Thank you, queue super user option works perfectly and seems exactly what wee need.
>> Now I'm trying to add authentication for app-server user and stuck with token mechanism.
> 
> Do note the token mechanism was released in the latest developer release!  I am quite happy to have you try it out and apologize for any speed bumps in the meantime.
> 
>> 
>> I have setup of 4 docker containers running in the same network: master, submit, worker and test container for remote submission.
>> They all have htcondor 8.9.2 installed. Master, submit and worker uses pool password for authentication and the test container have token
>> generated from those pool password (system in the test container does not have 'p7-server' user):
>> 
>>> user@459baa61ca5c:/$ condor_token_list 
>>> Header: {"alg":"HS256","kid":"POOL"} Payload: {"iat":1561470288,"iss":"htcondor-manager","sub":"p7-server@xxxxxxxxxxxxxx"} File: /etc/condor/tokens.d/p7-server.token
>> 
>> All containers have the same condor-host and domain settings in config file (read, write, client and negotiate rights are allowed to '*'):
>> 
>>> CONDOR_HOST=htcondor-manager
>>> DEFAULT_DOMAIN_NAME = htcondor.p7.da
>>> UID_DOMAIN = htcondor.p7.da
>>> TRUST_UID_DOMAIN = True
>>> FILESYSTEM_DOMAIN = htcondor.p7.da
>> 
>> Since docker does not have any domain system by default I made two aliases for the master host: htcondor-manager and htcondor-manager.htcondor.p7.da . 
>> Test system just have random hostname without domain name.  The problem is that test system can not find token in the token file:
>> 
>>> user@459baa61ca5c:/$ condor_submit -debug -file submit -remote htcondor-manager
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) SECMAN: new session, doing initial authentication.
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) SECMAN: Auth methods: TOKEN
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) AUTHENTICATE: setting timeout for <192.168.48.3:9618> to 20.
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: in handshake(my_methods = 'TOKEN')
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: handshake() - i am the client
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: sending (methods == 2048) to server
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_write(fd=3 collector at <192.168.48.3:9618>,,size=13,timeout=20,flags=0,non_blocking=0)
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_read(fd=3 collector at <192.168.48.3:9618>,,size=5,timeout=20,flags=0,non_blocking=0)
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_read(fd=3 collector at <192.168.48.3:9618>,,size=8,timeout=20,flags=0,non_blocking=0)
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: server replied (method = 2048)
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) PW.
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) PW: getting name.
>>> 06/25/19 14:05:51 (fd:5) (pid:38) (D_SECURITY) TOKEN: Will use examine tokens found in /etc/condor/tokens.d/p7-server.token.
>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_ALWAYS) TOKEN: No token found.
>> 
>> Full log is here: https://pastebin.com/AD1x6cd3
>> 
>> Test system user may read token file content and may ping both htcondor-manager and htcondor-manager.htcondor.p7.da .
>> From the htcondor source code I guess that client is filed to match server hostname (or full hostname) with the certificate issuer.
>> Could you point out how certificate issuer is related to condor-host and what can I do to debug this further?
>> 
> 
> So, the token logic tries to match on two things.
> 
> 1. "trust domain" for both the server and the client.  Verify it's the same for the client and server by examining the output of "condor_config_val TRUST_DOMAIN" on both sides.
> 2. The name of the key that signed the token on the client matches one key name in the server (in the output above, this is the value of "kid" -- currently "POOL").
>  - Given that's the default key name, "POOL" (from the pool password), I doubt this is the problem.
> 
> Can you check to see if either is the problem?
> 
> Brian
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/