[Condor-users] dagman jobs fail upredicatably

["Dr Ian C. Smith" <i.c.smith@xxxxxxxxxxxxxxx>]

Dear All,

We've recently been using DAGman to get long running
jobs working on our pool using the DAG recursion idea.
The submit host is a Solaris 9 box and all of the
execution PCs are Win XP/Intel. While the majority
of jobs work fine and run to completion, occasionally
some die. This error message appears in file.dagman.out:


> 2/4 02:33:39 Event: ULOG_EXECUTE for Condor Job A (13506.0.0)
> 2/4 02:33:49 Event: ULOG_IMAGE_SIZE for Condor Job A (13506.0.0)
> 2/4 02:53:47 Event: ULOG_IMAGE_SIZE for Condor Job A (13506.0.0)
> 2/4 03:13:49 Event: ULOG_IMAGE_SIZE for Condor Job A (13506.0.0)
> 2/4 03:33:47 Event: ULOG_IMAGE_SIZE for Condor Job A (13506.0.0)
> 2/4 04:33:56 read error on log
> /ffs/mat_alanca/condor/jobs/cl1/cmi600/mdr.log 2/4 04:33:56 ERROR:
> failure to read job log
>         A log event may be corrupt.  DAGMan will skip the event and try to
>         continue, but information may have been lost.  If DAGMan exits
>         unfinished, but reports no failed jobs, re-submit the rescue file
>         to complete the DAG

The log files are stored on an NFS mounted filesystem which I suppose
could cause problems but I can't understand why this would affect some
jobs and not others running concurrently. The actually dagaman process
still seems to be running happily on the submit host.

As a workaround can condor be set up to resubmit the rescue DAG 
automatically.

yours perplexed,

-ian.

-----------------------------------
Dr Ian C. Smith,
e-Science team,
University of Liverpool,
Computing Services Department.