[Condor-users] tips for setting up Condor on multiple(public/private) networks

[De-Wei Yin <dyin@xxxxxxxxxxxx>]

Oliver, Scott,

Since you're asking related questions, I'll try to answer them together.
We have compute nodes on a private network with the servers connected
also to the public network, and use the following configuration:

global configuration file for all nodes public and private:

    CONDOR_HOST = <public fully-qualified host name of CM>
    CM_IP_ADDR = <public IP addr of CONDOR_HOST>
    COLLECTOR_IP_ADDR = <public IP addr of CONDOR_HOST>
    HOSTALLOW_READ = *.public.net, 192.168.1.*
    HOSTALLOW_WRITE = *.public.net, 192.168.1.*

local configuration file for central manager:

    NETWORK_INTERFACE = <public IP addr of CONDOR_HOST>

local configuration file for submit machine:

    NETWORK_INTERFACE = <public IP addr of submit machine>

local configuration file for checkpoint server (we have one running):

    NETWORK_INTERFACE = <public IP addr of checkpoint server>

The private nodes must be able to reach the servers using the servers'
public IP addresses since the Condor services listen on those interfaces
only.  On our network each private node runs Red Hat 9 and has a file
/etc/sysconfig/network-scripts/route-eth0 to set up static routing;
this file contains:

    <public IP addr of CM> via <private IP addr of CM>
    <public IP addr of ckpt server> via <private IP addr of ckpt server>
    <public IP addr of whatever> via <private IP addr of whatever>

In case you have submit nodes or checkpoint servers running on machines
without internal network interfaces or you flock jobs to another network
that the private nodes cannot reach directly (this is your situation,
Oliver), you may need to set up IP masquerading on the gateway machine
for the private nodes.  To tell the privates nodes where the default
gateway is, specify in /etc/dhcpd.conf on the DHCP server the line

    option routers <private IP addr of gateway>;

or give this information explicitly to the node if static addressing
rather than DHCP server is used.  Then on the gateway machine you need
to turn on IP masquerading by executing the command

    echo "1" > /proc/sys/net/ipv4/ip_forward

This command needs to be executed everytime the gateway machine is
rebooted.  You can probably put this command in /etc/rc.d/rc.local to
have it run automatically.  To test that IP masquerading is working
correctly, log on to a private node and look at the output of the
"route" and "traceroute" commands, or you can simply try pinging some
far away but pingable machine from the private node and verify that it
responds.

Oliver:  In your case you will have two separate global configuration
files for the two pools;  both should have the lines

    HOSTALLOW_READ = *.public.net, 192.168.1.*, 192.168.2.*
    HOSTALLOW_WRITE = *.public.net, 192.168.1.*, 192.168.2.*

It might also be a good idea to set up actual checkpoint servers in each
pool at work and at home and set STARTER_CHOOSES_CKPT_SERVER to true.
This way if a job submitted at home flocks to the pool at your office
then it will checkpoint to the checkpoint server at the office rather
than to the submit machine at home.  But if you have only 10 machines in
a pool, maybe there isn't enough traffic to make you worry about this.

If there are firewalls in the way between the networks, configuring the
Condor pools is trickier but still possible.

Send me any question you still have about this.

Dewey


On Thu, 10 Jun 2004, Oliver Hotz wrote:

> Hey Guys,
>  
> I have a condor pool here at work, with Server1 on internal ip
> addresses (i.e. 192.168.1.1 and various submachines (clients),
> 192.168.1.10 - 20)
>  
> Then, I also have a condor pool at home, with the server there being
> on another internal IP address ((i.e. 192.168.2.1 and various
> submachines (clients), 192.168.2.10 - 20)
>  
>  
> Each of the servers, is also available to access as an outside IP. 
>  
> Is there any way to configure condor to see/use both pools ?... 
>  
> Thanks,
>  
> OLiver

On Fri, 18 Jun 2004 scott.brown@xxxxxxxxxx wrote:

> Hello,
> 
> I'm setting up a central manager on a machine that has two network 
> interfaces.  One is a private interface and the other is the outside
> world public interface.  I would like to set up my central manager to
> listen on both interfaces, because I want to make use of execution
> nodes that are located both inside and outside the private network.
> As far as I can tell NETWORK_INTERFACE assigns collector, etc to one
> and only one interface. Is there a way to simulataneously listen on
> more than one network interface? 
> 
> Thanks!
> 
> Scott

-- 
Mr. De-Wei Yin, MASc, PEng
Dept of Chemical & Biological Engineering tel: +1 608 262-3370
University of Wisconsin-Madison           fax: +1 608 262-5434
1415 Engineering Drive                    dyin at cae dot wisc dot edu
Madison WI 53706-1691 USA                 www.engr.wisc.edu/groups/mtsm/