Re: [HTCondor-users] Submitting different users jobs from the single remote user.

["Bockelman, Brian" <BBockelman@xxxxxxxxxxxxx>]

Ah - this is very useful.  It's potentially a bug.

The server advertises the list of keys it possesses (item 2 below) in the "IssuerKeys" attribute.  From you pastebin example (around line 239):

AuthMethods = "TOKEN"
AuthMethodsList = "TOKEN,CLAIMTOBE"
Authentication = "YES"
CryptoMethods = "3DES,BLOWFISH"
Enact = "YES"
Encryption = "NO"
Integrity = "NO"
RemoteVersion = "$CondorVersion: 8.9.2 Jun 04 2019 BuildID: Debian-8.9.2-1 PackageID: 8.9.2-1 Debian-8.9.2-1 $"
SessionDuration = "60"
SessionLease = 3600
TrustDomain = "htcondor.p7.da"

There's no "IssuerKeys" listed here!  That means the server isn't advertising its information correctly.

On the server side, try creating "/etc/condor/passwords.d" and copying your pool password into it as a file named "POOL".

Brian

> On Jul 3, 2019, at 7:11 AM, Sergey A. Komissarov <sergey.komissarov@xxxxxxxxxxxxxx> wrote:
> 
> Hello Brian,
> 
> I added 'ALL_DEBUG = D_ALL D_FULLDEBUG' to the client config, here is the log: https://pastebin.com/BadKNXnL .
> 
> ----------
> Sergey Komissarov
> Senior Software Developer
> DATADVANCE
> 
> This message may contain confidential information
> constituting a trade secret of DATADVANCE. Any distribution,
> use or copying of the information contained in this
> message is ineligible except under the internal
> regulations of DATADVANCE and may entail liability in
> accordance with the current legislation of the Russian
> Federation. If you have received this message by mistake
> please immediately inform me of it. Thank you!
> 
> 
> 
> ----- Original Message -----
> From: "Brian Bockelman" <BBockelman@xxxxxxxxxxxxx>
> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
> Sent: Wednesday, July 3, 2019 2:34:51 PM
> Subject: Re: [HTCondor-users] Submitting different	users	jobs	from	the	single	remote user.
> 
> Hi Sergey,
> 
> Your setup looks correct.  However, thereâs at least one detail missing in the log that might help debug.  Can you please post the exact same info but also adding the D_FULLDEBUG setting?
> 
> Sorry for the delay,
> 
> Brian
> 
> Sent from my iPhone
> 
>> On Jul 2, 2019, at 9:32 AM, Sergey A. Komissarov <sergey.komissarov@xxxxxxxxxxxxxx> wrote:
>> 
>> Hello Brian,
>> 
>> Settings from the condor master host (I'm skipping default values):
>> 
>> CONDOR_HOST = htcondor-manager.htcondor.p7.da
>> FULL_HOSTNAME = htcondor-manager.htcondor.p7.da
>> HOSTNAME = htcondor-manager
>> 
>> DEFAULT_DOMAIN_NAME = htcondor.p7.da
>> TRUST_DOMAIN = htcondor.p7.da
>> TRUST_UID_DOMAIN = True
>> UID_DOMAIN = htcondor.p7.da
>> FILESYSTEM_DOMAIN = htcondor.p7.da
>> 
>> The pool password and token generated with the following commands:
>> 
>> condor_store_cred -f /var/lock/condor/pool_password -p testpassword
>> condor_token_create -identity p7-server@xxxxxxxxxxxxxx -token p7-server
>> 
>> condor_token_list outputs the same in the test and in the master containers:
>> Header: {"alg":"HS256","kid":"POOL"} Payload: {"iat":1562073511,"iss":"htcondor.p7.da","sub":"p7-server@xxxxxxxxxxxxxx"} File: /etc/condor/tokens.d/p7-server
>> 
>> Test container settings:
>> 
>> CONDOR_HOST = htcondor-manager.htcondor.p7.da
>> FULL_HOSTNAME = 34790c54e50c.htcondor.p7.da
>> HOSTNAME = 34790c54e50c
>> 
>> DEFAULT_DOMAIN_NAME = htcondor.p7.da
>> TRUST_DOMAIN = htcondor.p7.da
>> TRUST_UID_DOMAIN = True
>> UID_DOMAIN = htcondor.p7.da
>> FILESYSTEM_DOMAIN = htcondor.p7.da
>> 
>> I added CLAIMTOBE as the second authentication method in the test container config and was able to get schedd data via python api, but it still fails to
>> find suitable token in the token file:
>> 
>>> import htcondor
>>> htcondor.enable_debug()
>>> htcondor.Collector().locateAll(htcondor.DaemonTypes.Schedd)
>> 07/02/19 13:53:04 (fd:12) (pid:263) (D_HOSTNAME) New Daemon obj (collector) name: "NULL", pool: "NULL", addr: "<192.168.96.5:9618>"
>> 07/02/19 13:53:04 (fd:12) (pid:263) (D_HOSTNAME) Address "<192.168.96.5:9618>" specified but no name, looking up host info
>> 07/02/19 13:53:04 (fd:12) (pid:263) (D_HOSTNAME) resolve_hostname_raw(): argument 'tests_htcondor-manager_1.p7-2k_devcontainer_devenv' is not a valid DNS name, returning no addresses.
>> 07/02/19 13:53:04 (fd:12) (pid:263) (D_ALWAYS) WARNING: forward resolution of tests_htcondor-manager_1.p7-2k_devcontainer_devenv doesn't match 192.168.96.5!
>> 07/02/19 13:53:04 (fd:13) (pid:263) (D_SECURITY) HANDSHAKE: server replied (method = 2048)
>> 07/02/19 13:53:04 (fd:13) (pid:263) (D_SECURITY) PW.
>> 07/02/19 13:53:04 (fd:13) (pid:263) (D_SECURITY) PW: getting name.
>> 07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) TOKEN: Will use examine tokens found in /etc/condor/tokens.d/p7-server.
>> 07/02/19 13:53:04 (fd:11) (pid:263) (D_ALWAYS) TOKEN: No token found.
>> 07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) PW: Failed to fetch a login name
>> 07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) AUTHENTICATE: method 2048 (TOKEN) failed.
>> 07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) HANDSHAKE: server replied (method = 2)
>> 07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) Authentication was a Success.
>> 
>> Full output is here: https://pastebin.com/YzJ55tCu . 
>> For now I may use CLAIMTOBE authentication for further tests and integration development and hope token authentication will be a part of stable release before our own release =)
>> 
>> ----------
>> Sergey Komissarov
>> Senior Software Developer
>> DATADVANCE
>> 
>> This message may contain confidential information
>> constituting a trade secret of DATADVANCE. Any distribution,
>> use or copying of the information contained in this
>> message is ineligible except under the internal
>> regulations of DATADVANCE and may entail liability in
>> accordance with the current legislation of the Russian
>> Federation. If you have received this message by mistake
>> please immediately inform me of it. Thank you!
>> 
>> 
>> 
>> ----- Original Message -----
>> From: "Brian Bockelman" <BBockelman@xxxxxxxxxxxxx>
>> To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
>> Sent: Tuesday, July 2, 2019 5:53:53 AM
>> Subject: Re: [HTCondor-users] Submitting different users jobs    from    the    single    remote user.
>> 
>> Hi Sergey,
>> 
>> Apologies - I was on vacation last week.  Answers inline.
>> 
>>> On Jun 25, 2019, at 11:08 AM, Sergey A. Komissarov <sergey.komissarov@xxxxxxxxxxxxxx> wrote:
>>> 
>>> Hello Brian,
>>> 
>>> Thank you, queue super user option works perfectly and seems exactly what wee need.
>>> Now I'm trying to add authentication for app-server user and stuck with token mechanism.
>> 
>> Do note the token mechanism was released in the latest developer release!  I am quite happy to have you try it out and apologize for any speed bumps in the meantime.
>> 
>>> 
>>> I have setup of 4 docker containers running in the same network: master, submit, worker and test container for remote submission.
>>> They all have htcondor 8.9.2 installed. Master, submit and worker uses pool password for authentication and the test container have token
>>> generated from those pool password (system in the test container does not have 'p7-server' user):
>>> 
>>>> user@459baa61ca5c:/$ condor_token_list 
>>>> Header: {"alg":"HS256","kid":"POOL"} Payload: {"iat":1561470288,"iss":"htcondor-manager","sub":"p7-server@xxxxxxxxxxxxxx"} File: /etc/condor/tokens.d/p7-server.token
>>> 
>>> All containers have the same condor-host and domain settings in config file (read, write, client and negotiate rights are allowed to '*'):
>>> 
>>>> CONDOR_HOST=htcondor-manager
>>>> DEFAULT_DOMAIN_NAME = htcondor.p7.da
>>>> UID_DOMAIN = htcondor.p7.da
>>>> TRUST_UID_DOMAIN = True
>>>> FILESYSTEM_DOMAIN = htcondor.p7.da
>>> 
>>> Since docker does not have any domain system by default I made two aliases for the master host: htcondor-manager and htcondor-manager.htcondor.p7.da . 
>>> Test system just have random hostname without domain name.  The problem is that test system can not find token in the token file:
>>> 
>>>> user@459baa61ca5c:/$ condor_submit -debug -file submit -remote htcondor-manager
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) SECMAN: new session, doing initial authentication.
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) SECMAN: Auth methods: TOKEN
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) AUTHENTICATE: setting timeout for <192.168.48.3:9618> to 20.
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: in handshake(my_methods = 'TOKEN')
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: handshake() - i am the client
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: sending (methods == 2048) to server
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_write(fd=3 collector at <192.168.48.3:9618>,,size=13,timeout=20,flags=0,non_blocking=0)
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_read(fd=3 collector at <192.168.48.3:9618>,,size=5,timeout=20,flags=0,non_blocking=0)
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_read(fd=3 collector at <192.168.48.3:9618>,,size=8,timeout=20,flags=0,non_blocking=0)
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: server replied (method = 2048)
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) PW.
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) PW: getting name.
>>>> 06/25/19 14:05:51 (fd:5) (pid:38) (D_SECURITY) TOKEN: Will use examine tokens found in /etc/condor/tokens.d/p7-server.token.
>>>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_ALWAYS) TOKEN: No token found.
>>> 
>>> Full log is here: https://pastebin.com/AD1x6cd3
>>> 
>>> Test system user may read token file content and may ping both htcondor-manager and htcondor-manager.htcondor.p7.da .
>>> From the htcondor source code I guess that client is filed to match server hostname (or full hostname) with the certificate issuer.
>>> Could you point out how certificate issuer is related to condor-host and what can I do to debug this further?
>>> 
>> 
>> So, the token logic tries to match on two things.
>> 
>> 1. "trust domain" for both the server and the client.  Verify it's the same for the client and server by examining the output of "condor_config_val TRUST_DOMAIN" on both sides.
>> 2. The name of the key that signed the token on the client matches one key name in the server (in the output above, this is the value of "kid" -- currently "POOL").
>> - Given that's the default key name, "POOL" (from the pool password), I doubt this is the problem.
>> 
>> Can you check to see if either is the problem?
>> 
>> Brian
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>> 
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>> 
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/