[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] condor-g job status unknown caused dagman to exit
- Date: Tue, 20 Apr 2010 16:14:42 -0400
- From: Peter Doherty <doherty@xxxxxxxxxxxxxxxxxxx>
- Subject: Re: [Condor-users] condor-g job status unknown caused dagman to exit
On Apr 20, 2010, at 12:59 , R. Kent Wenger wrote:
On Tue, 20 Apr 2010, Peter Doherty wrote:
To the condor experts,
I'm using Condor-G and this job (see logs below) caused a large dag
to exit unexpectedly this morning. I understand the concept of BAD
EVENTS ( in this case the job started executing after it was
supposed to be done)
But I want to know how to prevent this from happening.
From what I can see the job's state became unknown (What causes
this? I see it happen a lot actually)
and then Condor abandoned the job, but then the job called back
home and instead of just ignoring it, it heard the call, but then
got confused, and dagman exited.
It seems like a bug to me.
You could try working around it by setting a non-default value for
in the manual).
I'm actually not quite sure what DAGMan will do if it doesn't
consider the execute event "bad" -- it might still run into problems
farther along, but it seems worth trying.
I have changed the default.
DAGMAN_ALLOW_EVENTS = 1
The only other value I can use is 5, and the manual basically says
"don't do this"
"A value of 5 will never abort the DAG because of a bad event. But
this value should almost never be used, because the "job re-run after
terminated event" bug breaks the semantics of the DAG."
I can try setting it to 5, I just hope it doesn't create more problems
than it solves.
It seems like the real question is whether this is something that
should be dealt with at the DAGMan level or the Condor level --
we'll have to discuss that on our end. If we deal with it at the
DAGMan level, we'd have to allow a new job state transition -- from
completed/failed to executing to possibly completed/succeeded.
So it sounds like you're saying that the situation that I'm seeing is
within the design parameters, but can cause unexpected failures, so
not exactly a bug, but more of a design limitation?