[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Access Problems

Thank you for replying. I will try some of these and will report soon. Specific replies below.

On Mon, Aug 26, 2013 at 9:36 AM, Todd Tannenbaum <tannenba@xxxxxxxxxxx> wrote:
On 8/24/2013 7:39 PM, Andrey Kuznetsov wrote:

What is happening is that YYY talks to XXX over internal network because of
the hosts file, and machine XXX tries to authenticate machine YYY.
It first does a forward name resolution of YYY.ucsc.edu which turns out to
be because of the /etc/hosts file.
Then it does a reverse DNS lookup on YYY.ucsc.edu and returns an external
IP address of that machine.
When comparing the 2 addresses, they obviously don't match because one IP
is external, and the other is internal. This causes the following errors in
the XXX's SchedLog log file when a condor_q -g command was used on YYY:
PERMISSION DENIED to unauthenticated@unmapped from host 128.114.###.YYY for
command 1111 (QMGMT_READ_CMD), access level READ: reason: READ
authorization policy contains no matching ALLOW entry for this request;

How do you know this is what is happening? In other words, how did you observe that HTCondor forward/reverse lookups are coming from different places (/etc/hosts vs DNS) and that is the issue -- or is it just a guess?

If the above is what is happening, I would expect to see the following in warning in the logs:
 WARNING: forward resolution of XXX doesn't match YYY!
Is that warning present in the logs?
Sorry, I thought I posted that part of the log, yes, that is exactly what is in the logs.

I would expect HTCondor to always use either /etc/hosts or DNS in a consistent order (dictated by /etc/nsswitch.conf) regardless of forwards or backwards, as iirc all HTCondor is doing is calling libc calls to do the lookups in both cases.

Some other questions:

- what version of HTCondor are you running?
v7.8.7 soon to be upgraded to v8.X.X

- does a forward lookup of 128.114.###.YYY on machine xxx result in a hostname that matches *.ucsc.edu ?

machine is properly DNS registered, everything works except internal network with condor

How do I make XXX work with YYY over the internal network?
Is there an option to modify reverse DNS lookup to resolve YYY to an
internal IP address?
Is there an option to turn off reverse DNS lookup so that YYY is
authenticated as either YYY.ucsc.edu or instead of 128.114.###.YYY?

Some quick random thoughts/pointers:

- It may help to peruse

- On the machines that are connected to the private network (i.e. the machines with multiple interfaces), you could try setting something like
This tells HTCondor to use 128.114 interface for everything except for communication between machines that have the same private network name; for those machines it will use the 10.0.* interface.
Hmm, I didn't notice these, thanks will try.
- maybe add 128.114.* or similar settings to your ALLOW_READ / ALLOW_WRITE settings?
Will try.

- perhaps temporarily add "D_HOSTNAME D_SECURITY" to the SCHEDD_DEBUG for more insight/output into the scheddlog
Will try.
- it is a rather drastic step, but you can disable DNS lookups all-together in HTCondor via
   NO_DNS = True
this is probably not what you want as then HTCondor constructs host names using the host's IP address as a string... but could be an option of last resort, and maybe you do not care...

I saw the NO_DNS feature but would rather stay away from it for now.
Maybe this will point you in the right direction and/or others can chime in...


Andrey Kuznetsov <akuznet1@xxxxxxxx>