Re: [condor-users] Condor shuts down our network

[Erik Paulson <epaulson@xxxxxxxxxxx>]

On Tue, Mar 30, 2004 at 09:18:28AM -0500, David Vestal wrote:
> Our IT dept. noticed that at random times, data would flood into our central manager, overwhelming our internet connection, and effectively shutting us down.  This tends to happen on the order of once per week or so, and the attacks have lasted between 5 and 30 minutes.  Through logging, we realized that the traffic is all going into or out of the central manager of our grid, and recently we were able to log some of the packets that came in.
> 
> All the packets in the most recent attack were from a single machine on our grid.  I'm not certain what they are, but it should be obvious to you guys; they begin with some binary data, and end with: Machine = "350-503-KA3DVP4" Name = "350-503-KA3DVP4" MasterIpAddr = "<10.83.1.5:1026>" CondorVersion = "$CondorVersion: 6.4.7 Jan 27 2003 $" CondorPlatform = "$CondorPlatform: INTEL-WINNT40 $" MASTER_Timestamp = 1043702192 MASTER_StartTime = 1080321908 STARTD_Timestamp = 1043702196 STARTD_StartTime = 1080321908 DaemonMaster
> 
> It appears to be an update of machine status.  Our logging software logged 57874 of them in just over 56 seconds.

That's certainly not right.


>  This seems like abnormal behavior.  Has anyone else seen it, or has anyone else a suggestion?  This particular machine is no longer appearing on the grid, and I don't have physical access to it.  I would also doubt that this machine alone is causing the problem.
> 

We've never seen it before. I'm certainly not willing to rule out a Condor
bug, but I'd expect to see it somewhere else.

The ad you're seeing is the Master Ad - the update that it gets sent at is
controlled by the MASTER_UPDATE_INTERVAL. It should default to 300 seconds.

The best thing to check would be to see the machine that's sending all of
the ads in. 

It's also worth looking in the CollectorLog on the central manager, and seeing
how many ads it's receiving. 

-Erik
Condor Support Information:
http://www.cs.wisc.edu/condor/condor-support/
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>