[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Fault Behaviour of Condor

Hi Todd,

Thank you very much for the explanation.
I understand the story.
I am glad if this problem will be solved in the future.


----- Original Message ----- From: "Todd Tannenbaum" <tannenba@xxxxxxxxxxx>
To: "Condor-Users Mail List" <condor-users@xxxxxxxxxxx>
Sent: Tuesday, August 15, 2006 7:31 AM
Subject: Re: [Condor-users] Fault Behaviour of Condor

At 10:57 AM 8/2/2006, Matt Hope wrote:
On 8/2/06, thomas.t.hoppe@xxxxxxxxxxxxxxxxxxx
<thomas.t.hoppe@xxxxxxxxxxxxxxxxxxx> wrote:

> 3.) Shutting down the NIC on the executor (I assume same as pulling the
> plug)
> Outcome: Condor hangs, a shadow process is existing all the time
> I even cannot remove the job with condor_rm!
> Maybe a bug? what can I do?

condor_rm -forcex may get rid of it (you may need to kill off the
shadow by hand, it should eventually timeout though, how long did you
give it?).

Here is the story:

Condor sends regular "pings" from the submit machine (schedd) to the
execute machine (startd).  Thus the execute machine knows relatively
quickly if a submit machine disappears (because it will not receive a
ping within the anticipated timeframe).  You can configure how often
these pings happen and how quickly the Condor execute machine will
throw off a current job by tweaking the job lease parameter in the submit file.

However, going the other way is a different story. There is currently
no way to configure how often "pings" happen from the execute machine
back to the submit machine.  Thus, there is no way to configure how
long it takes before a submit machine (the condor_shadow) notices
when a execute machine falls off of the network.  Have no fear,
however, as Condor most definitely *will* notice it eventually, but
you may need to be patient.  The socket created by the condor_shadow
is using TCP's KEEPALIVE option on the socket.  However, the standard
for TCP says it only needs to send a ping every two hours.   So in
the worst case, the shadow may take up to a max of two hours to
notice if the execute machine has fallen off the network without a
trace.    This is an issue we'd like to improve.

Hope this helps clarify things,

Todd Tannenbaum                       University of Wisconsin-Madison
Condor Project Research               Department of Computer Sciences
tannenba@xxxxxxxxxxx                  1210 W. Dayton St. Rm #4257
http://www.cs.wisc.edu/~tannenba      Madison, WI 53706-1685
Phone: (608) 263-7132  FAX: (608) 262-9777

Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at either