Re: [Condor-users] recursive DAGs - how to implement ?

["Dr Ian C. Smith" <i.c.smith@xxxxxxxxxxxxxxx>]

Hi Mark,

That would be very useful.

many thanks,

-ian.


--On 03 November 2004 11:57 +0000 mcal00@xxxxxxxxxxxxx wrote:

> Hi Ian, that was me. We can take this offline and I'll email you my
> example implementation directly.
>
> Cheers,
> Mark
>
> Dr Ian C. Smith wrote:
>
> > Dear All,
> >
> > I was at the Condor Week in Edinburgh a while back and I
> > remember that a talk on the eMinerals project mentioned
> > something about using recursive DAGs to resubmit long
> > running jobs automatically. I think this would be really
> > useful here but I can't seem to find any info on how to
> > do it.
> >
> > To give you some background. Our Condor pool is only available
> > for around 16 hours each day so jobs needing to run for
> > several days need to have some way of getting the intermediate
> > results back and then using them as input on the next run.
> > At present we can look at the system clock to see if time is
> > about to run out then exit gracefully so that Condor stages the
> > results back. The submit host needs a cron style script to
> > re-submit the job which is messy to say the least.
> >
> > One idea is to have the job run only for a few hours only at a time.
> > If it's a time stepping code we can easy specify the number of
> > steps to be taken and the solution can be picked again later were we
> > left off. If it gets killed then a couple of hours of wallclock
> > have been wasted - no big deal. I imagine that this could easily
> > be achieved with a DAG but of course we don't know how many time steps
> > are needed in total a priori. This is were a recursion would be useful if
> > only I could work out how to do it.
> >
> > Anyone have any ideas ??
> >
> > cheers,
> >
> > -ian.
> >
> > PS pedantic point: doesn't a recursion introduce a cyclic dependency ?
> >   Why not just allow cycles in the DAG dependency graph ?
> >
> >
> >
> >
> >
> > _______________________________________________
> > Condor-users mailing list
> > Condor-users@xxxxxxxxxxx
> > http://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
>
>
>
> --
> Dr Mark Calleja
> Cambridge eScience Centre, University of Cambridge
> Centre for Mathematical Sciences, Wilberforce Road, Cambridge CB3 0WA
> Tel. (+44/0) 1223 765317, Fax  (+44/0) 1223 765900
> http://www.escience.cam.ac.uk/~mcal00
>
>
>
> ---------------------------------------------
> Department of Earth Sciences
> University of Cambridge
> Downing Street
> Cambridge CB2 3EQ
> Phone: ( +44 ) 1223 333400
> Fax: ( +44 ) 1223 333450
>
>
> _______________________________________________
> Condor-users mailing list
> Condor-users@xxxxxxxxxxx
> http://lists.cs.wisc.edu/mailman/listinfo/condor-users