[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] Jobs die with signal 11
- Date: Wed, 29 Sep 2004 07:01:11 +0100
- From: Mark Calleja <mcal00@xxxxxxxxxxxxx>
- Subject: Re: [Condor-users] Jobs die with signal 11
It looks like that this was my goof, in that I was running an executable
built on a machine with glibc v2.3.x on a v2.2.x box. Having now built
the executable on a matching platform, I no longer get those pesky
signal 11's. However, other problems raise their head which I'll mention
in a separate thread.
Thx for the help,
Nick LeRoy wrote:
On Tue September 28 2004 12:25 pm, mcal00@xxxxxxxxxxxxx wrote:
Hi, we've got an old linux cluster (i286 processors running RH7.2) that
we've converted into a Condor pool and we constantly see jobs dying with
Shadow exceptions, with the only clue in the StarterLog files being of the
form (Condor v. 6.6.6 all round):
9/27 14:12:33 vm2: Got activate_claim request from shadow
9/27 14:12:33 vm2: Remote job ID is 352.0
9/27 14:12:33 vm2: Got universe "VANILLA" (5) from request classad
9/27 14:12:33 vm2: State change: claim-activation protocol successful
9/27 14:12:33 vm2: Changing activity: Idle -> Busy
9/27 14:21:22 Starter pid 21927 died on signal 11 (signal 11)
9/27 14:21:22 vm2: State change: starter exited
9/27 14:21:22 vm2: Changing activity: Busy -> Idle
What's that signal 11 mean? I notice that someone spotted something similar
under solaris last year (message 476), and Erik Paulson suggested that it
may have been a bug. Was it ever resolved?
Signal 11 means "segmentation fault" (SEGV); i.e. the program crashed. Mostly
likely, this is due to a buggy application being started by Condor.
These jobs are coming in from flocked pools across the campus, so the
network they have to traverse is slightly unfriendlier than your average
LAN. Could such a signal be due to a network glitch?
I really can't see how network topology could cause a job to get a SEGV,
except in some unusual circumstances.
One thing to try would be to run the job's executable directly on a machine
that you've seen it crash on (directly as in without Condor involved at all).
If it runs like that without crashing, then there's some unintended
interaction, but, most likely, you'll see the same behavior. In the above
log, the job had been running for ~11 minutes, so you shouldn't have to wait
There is one other thing that I can thing of looking into: environment
variables. User jobs may get started with a very different set of
environment variables than those that they were submitted with, which could
possibly cause a buggy application to crash as well.
Hope this helps
Dr Mark Calleja
Cambridge eScience Centre, University of Cambridge
Centre for Mathematical Sciences, Wilberforce Road, Cambridge CB3 0WA
Tel. (+44/0) 1223 765317, Fax (+44/0) 1223 765900