[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Submitting different users jobs from the single remote user.

Hello Brian,

Settings from the condor master host (I'm skipping default values):

CONDOR_HOST = htcondor-manager.htcondor.p7.da
FULL_HOSTNAME = htcondor-manager.htcondor.p7.da
HOSTNAME = htcondor-manager

DEFAULT_DOMAIN_NAME = htcondor.p7.da
TRUST_DOMAIN = htcondor.p7.da
UID_DOMAIN = htcondor.p7.da
FILESYSTEM_DOMAIN = htcondor.p7.da

The pool password and token generated with the following commands:

condor_store_cred -f /var/lock/condor/pool_password -p testpassword
condor_token_create -identity p7-server@xxxxxxxxxxxxxx -token p7-server

condor_token_list outputs the same in the test and in the master containers:
Header: {"alg":"HS256","kid":"POOL"} Payload: {"iat":1562073511,"iss":"htcondor.p7.da","sub":"p7-server@xxxxxxxxxxxxxx"} File: /etc/condor/tokens.d/p7-server

Test container settings:

CONDOR_HOST = htcondor-manager.htcondor.p7.da
FULL_HOSTNAME = 34790c54e50c.htcondor.p7.da
HOSTNAME = 34790c54e50c

DEFAULT_DOMAIN_NAME = htcondor.p7.da
TRUST_DOMAIN = htcondor.p7.da
UID_DOMAIN = htcondor.p7.da
FILESYSTEM_DOMAIN = htcondor.p7.da

I added CLAIMTOBE as the second authentication method in the test container config and was able to get schedd data via python api, but it still fails to
find suitable token in the token file:

> import htcondor
> htcondor.enable_debug()
> htcondor.Collector().locateAll(htcondor.DaemonTypes.Schedd)
07/02/19 13:53:04 (fd:12) (pid:263) (D_HOSTNAME) New Daemon obj (collector) name: "NULL", pool: "NULL", addr: "<>"
07/02/19 13:53:04 (fd:12) (pid:263) (D_HOSTNAME) Address "<>" specified but no name, looking up host info
07/02/19 13:53:04 (fd:12) (pid:263) (D_HOSTNAME) resolve_hostname_raw(): argument 'tests_htcondor-manager_1.p7-2k_devcontainer_devenv' is not a valid DNS name, returning no addresses.
07/02/19 13:53:04 (fd:12) (pid:263) (D_ALWAYS) WARNING: forward resolution of tests_htcondor-manager_1.p7-2k_devcontainer_devenv doesn't match!
07/02/19 13:53:04 (fd:13) (pid:263) (D_SECURITY) HANDSHAKE: server replied (method = 2048)
07/02/19 13:53:04 (fd:13) (pid:263) (D_SECURITY) PW.
07/02/19 13:53:04 (fd:13) (pid:263) (D_SECURITY) PW: getting name.
07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) TOKEN: Will use examine tokens found in /etc/condor/tokens.d/p7-server.
07/02/19 13:53:04 (fd:11) (pid:263) (D_ALWAYS) TOKEN: No token found.
07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) PW: Failed to fetch a login name
07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) AUTHENTICATE: method 2048 (TOKEN) failed.
07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) HANDSHAKE: server replied (method = 2)
07/02/19 13:53:04 (fd:11) (pid:263) (D_SECURITY) Authentication was a Success.

Full output is here: https://pastebin.com/YzJ55tCu . 
For now I may use CLAIMTOBE authentication for further tests and integration development and hope token authentication will be a part of stable release before our own release =)

Sergey Komissarov
Senior Software Developer

This message may contain confidential information
constituting a trade secret of DATADVANCE. Any distribution,
use or copying of the information contained in this
message is ineligible except under the internal
regulations of DATADVANCE and may entail liability in
accordance with the current legislation of the Russian
Federation. If you have received this message by mistake
please immediately inform me of it. Thank you!

----- Original Message -----
From: "Brian Bockelman" <BBockelman@xxxxxxxxxxxxx>
To: "HTCondor-Users Mail List" <htcondor-users@xxxxxxxxxxx>
Sent: Tuesday, July 2, 2019 5:53:53 AM
Subject: Re: [HTCondor-users] Submitting different users jobs	from	the	single	remote user.

Hi Sergey,

Apologies - I was on vacation last week.  Answers inline.

> On Jun 25, 2019, at 11:08 AM, Sergey A. Komissarov <sergey.komissarov@xxxxxxxxxxxxxx> wrote:
> Hello Brian,
> Thank you, queue super user option works perfectly and seems exactly what wee need.
> Now I'm trying to add authentication for app-server user and stuck with token mechanism.

Do note the token mechanism was released in the latest developer release!  I am quite happy to have you try it out and apologize for any speed bumps in the meantime.

> I have setup of 4 docker containers running in the same network: master, submit, worker and test container for remote submission.
> They all have htcondor 8.9.2 installed. Master, submit and worker uses pool password for authentication and the test container have token
> generated from those pool password (system in the test container does not have 'p7-server' user):
>> user@459baa61ca5c:/$ condor_token_list 
>> Header: {"alg":"HS256","kid":"POOL"} Payload: {"iat":1561470288,"iss":"htcondor-manager","sub":"p7-server@xxxxxxxxxxxxxx"} File: /etc/condor/tokens.d/p7-server.token
> All containers have the same condor-host and domain settings in config file (read, write, client and negotiate rights are allowed to '*'):
>> CONDOR_HOST=htcondor-manager
>> DEFAULT_DOMAIN_NAME = htcondor.p7.da
>> UID_DOMAIN = htcondor.p7.da
>> FILESYSTEM_DOMAIN = htcondor.p7.da
> Since docker does not have any domain system by default I made two aliases for the master host: htcondor-manager and htcondor-manager.htcondor.p7.da . 
> Test system just have random hostname without domain name.  The problem is that test system can not find token in the token file:
>> user@459baa61ca5c:/$ condor_submit -debug -file submit -remote htcondor-manager
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) SECMAN: new session, doing initial authentication.
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) SECMAN: Auth methods: TOKEN
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) AUTHENTICATE: setting timeout for <> to 20.
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: in handshake(my_methods = 'TOKEN')
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: handshake() - i am the client
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: sending (methods == 2048) to server
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_write(fd=3 collector at <>,,size=13,timeout=20,flags=0,non_blocking=0)
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_read(fd=3 collector at <>,,size=5,timeout=20,flags=0,non_blocking=0)
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_NETWORK) condor_read(fd=3 collector at <>,,size=8,timeout=20,flags=0,non_blocking=0)
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) HANDSHAKE: server replied (method = 2048)
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) PW.
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_SECURITY) PW: getting name.
>> 06/25/19 14:05:51 (fd:5) (pid:38) (D_SECURITY) TOKEN: Will use examine tokens found in /etc/condor/tokens.d/p7-server.token.
>> 06/25/19 14:05:51 (fd:4) (pid:38) (D_ALWAYS) TOKEN: No token found.
> Full log is here: https://pastebin.com/AD1x6cd3
> Test system user may read token file content and may ping both htcondor-manager and htcondor-manager.htcondor.p7.da .
> From the htcondor source code I guess that client is filed to match server hostname (or full hostname) with the certificate issuer.
> Could you point out how certificate issuer is related to condor-host and what can I do to debug this further?

So, the token logic tries to match on two things.

1. "trust domain" for both the server and the client.  Verify it's the same for the client and server by examining the output of "condor_config_val TRUST_DOMAIN" on both sides.
2. The name of the key that signed the token on the client matches one key name in the server (in the output above, this is the value of "kid" -- currently "POOL").
  - Given that's the default key name, "POOL" (from the pool password), I doubt this is the problem.

Can you check to see if either is the problem?

HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at: