Re: [HTCondor-users] Access Problems

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 8/24/2013 7:39 PM, Andrey Kuznetsov wrote:
> What is happening is that YYY talks to XXX over internal network because of
> the hosts file, and machine XXX tries to authenticate machine YYY.
> It first does a forward name resolution of YYY.ucsc.edu which turns out to
> be 10.0.0.2 because of the /etc/hosts file.
> Then it does a reverse DNS lookup on YYY.ucsc.edu and returns an external
> IP address of that machine.
> When comparing the 2 addresses, they obviously don't match because one IP
> is external, and the other is internal. This causes the following errors in
> the XXX's SchedLog log file when a condor_q -g command was used on YYY:
> PERMISSION DENIED to unauthenticated@unmapped from host 128.114.###.YYY for
> command 1111 (QMGMT_READ_CMD), access level READ: reason: READ
> authorization policy contains no matching ALLOW entry for this request;

How do you know this is what is happening? In other words, how did you 
observe that HTCondor forward/reverse lookups are coming from different 
places (/etc/hosts vs DNS) and that is the issue -- or is it just a guess?

If the above is what is happening, I would expect to see the following 
in warning in the logs:
 WARNING: forward resolution of XXX doesn't match YYY!
Is that warning present in the logs?

I would expect HTCondor to always use either /etc/hosts or DNS in a 
consistent order (dictated by /etc/nsswitch.conf) regardless of forwards 
or backwards, as iirc all HTCondor is doing is calling libc calls to do 
the lookups in both cases.

Some other questions:

- what version of HTCondor are you running?

- does a forward lookup of 128.114.###.YYY on machine xxx result in a 
hostname that matches *.ucsc.edu ?

> Questions:
> How do I make XXX work with YYY over the internal network?
> Is there an option to modify reverse DNS lookup to resolve YYY to an
> internal IP address?
> Is there an option to turn off reverse DNS lookup so that YYY is
> authenticated as either YYY.ucsc.edu or 10.0.0.2 instead of 128.114.###.YYY?

Some quick random thoughts/pointers:

- It may help to peruse
http://research.cs.wisc.edu/htcondor/manual/v8.0/3_7Networking_includes.html 


- On the machines that are connected to the private network (i.e. the 
machines with multiple interfaces), you could try setting something like
  NETWORK_INTERFACE = 128.114.*
  PRIVATE_NETWORK_INTERFACE = 10.0.*
  PRIVATE_NETWORK_NAME = MyPrivNetworkName
This tells HTCondor to use 128.114 interface for everything except for 
communication between machines that have the same private network name; 
for those machines it will use the 10.0.* interface.

- maybe add 128.114.* or similar settings to your ALLOW_READ / 
ALLOW_WRITE settings?

- perhaps temporarily add "D_HOSTNAME D_SECURITY" to the SCHEDD_DEBUG 
for more insight/output into the scheddlog

- it is a rather drastic step, but you can disable DNS lookups 
all-together in HTCondor via
   NO_DNS = True
   DEFAULT_DOMAIN_NAME = ucsc.edu
this is probably not what you want as then HTCondor constructs host 
names using the host's IP address as a string... but could be an option 
of last resort, and maybe you do not care...

Maybe this will point you in the right direction and/or others can chime 
in...

Todd