[Condor-users] recursive DAGs - how to implement ?

[mcal00@xxxxxxxxxxxxx]

Hi Ian, that was me. We can take this offline and I'll email you my example
implementation directly.

Cheers,
Mark

Dr Ian C. Smith wrote:

> Dear All,
>
> I was at the Condor Week in Edinburgh a while back and I
> remember that a talk on the eMinerals project mentioned
> something about using recursive DAGs to resubmit long
> running jobs automatically. I think this would be really
> useful here but I can't seem to find any info on how to
> do it.
>
> To give you some background. Our Condor pool is only available
> for around 16 hours each day so jobs needing to run for
> several days need to have some way of getting the intermediate
> results back and then using them as input on the next run.
> At present we can look at the system clock to see if time is
> about to run out then exit gracefully so that Condor stages the
> results back. The submit host needs a cron style script to
> re-submit the job which is messy to say the least.
>
> One idea is to have the job run only for a few hours only at a time.
> If it's a time stepping code we can easy specify the number of
> steps to be taken and the solution can be picked again later were we
> left off. If it gets killed then a couple of hours of wallclock
> have been wasted - no big deal. I imagine that this could easily
> be achieved with a DAG but of course we don't know how many time steps
> are needed in total a priori. This is were a recursion would be useful if
> only I could work out how to do it.
>
> Anyone have any ideas ??
>
> cheers,
>
> -ian.
>
> PS pedantic point: doesn't a recursion introduce a cyclic dependency ?
>   Why not just allow cycles in the DAG dependency graph ?
>
>
>
>
>
> _______________________________________________
> Condor-users mailing list
> Condor-users@xxxxxxxxxxx
> http://lists.cs.wisc.edu/mailman/listinfo/condor-users




-- 
Dr Mark Calleja
Cambridge eScience Centre, University of Cambridge
Centre for Mathematical Sciences, Wilberforce Road, Cambridge CB3 0WA
Tel. (+44/0) 1223 765317, Fax  (+44/0) 1223 765900
http://www.escience.cam.ac.uk/~mcal00



---------------------------------------------
Department of Earth Sciences
University of Cambridge
Downing Street
Cambridge CB2 3EQ
Phone: ( +44 ) 1223 333400
Fax: ( +44 ) 1223 333450