Re: [Condor-users] [condor-users] Iterative Computations [was: Does stork avoid retransfering data?]

[Gabriel Mateescu <gabriel.mateescu@xxxxxxxxx>]

I am thinking about using Makeflow to
build a workflow engine that manages
a genetic algorithm which contains
two operations:

 1. next_gen tool: generates the next generation
     of candidates and partitions the candidates
     in two disjoint parts, part1 and part2;

  2. evaluate tool: computes the fitness of every
      candidate in either the part1 or part2 input.

A Makefile that encodes the workflow for a
Genetic Algorithm may look like this:


  part1 part2:  input.data
                    next_gen input.data

  part1_score:  part1
                     evaluate part1 > part1_score

  part2_score:  part2
                     evaluate part2 > part2_score

  stop-test:     part1 part2 part1_score part2_score
                    if ( max(part1_score, part2_score) < 13) {
                         update input.data part1 part2
                         make part1 part2
                    }

The intent of the rule for the target stop-test is to
specify that a new make cycle should be executed
if max(part1_score, part2_score) >= 13.

So it seems that in order to implement loops,
one needs to support a conditional statement
and the recursive invocation of make-flow.

Thank you.
Gabriel





On Thu, Mar 1, 2012 at 5:46 PM, Douglas Thain <dthain@xxxxxx> wrote:
> Gabriel -
>
> Makeflow does not currently do that; it is just a static DAG.
>
> However, this is an important workflow pattern that we have been thinking about,
> and we could work with you to get something going in a way that is
> Condor-compatible.
>
> Can you share some more detailed use cases?
> Is there anyone else on the list interested in such a capability?
>
> Doug
>
>
> On Thu, Mar 1, 2012 at 11:33 AM, Gabriel Mateescu
> <gabriel.mateescu@xxxxxxxxx> wrote:
>> Hi Doug,
>>
>> Is it possible to express with Makeflow
>> iterative computations that repeat a set
>> of steps until some stopping criterion
>> is met?
>>
>> Thanks.
>> Gabriel
>>
>>
>>
>>
>> On Tue, Jan 11, 2011 at 5:55 PM, Douglas Thain <dthain@xxxxxx> wrote:
>>> Thomas -
>>>
>>> You might consider using Makeflow for this task:
>>> http://www.nd.edu/~ccl/software/makeflow
>>>
>>> The idea is that you express your tasks in Makeflow, submit a bunch of
>>> 'worker' processes to Condor, and Makeflow will distribute tasks among
>>> the workers.  If they have some common executables and input files,
>>> they will be automatically cached at the workers, so you don't have to
>>> keep transmitting them.
>>>
>>> Cheers,
>>> Doug
>>>
>>> On Tue, Jan 11, 2011 at 8:27 AM, Rowe, Thomas <rowet@xxxxxxxxxx> wrote:
>>>> I have to run a simulation about a thousand times with different seeds.  The
>>>> simulation executable and data total about 100MB.  This sounds like a job
>>>> for DAGMan & Stork, because this 100MB collection of files needs to get
>>>> copied around reliably, and some large output files need to be transferred
>>>> back to the originating machine reliably.
>>>>
>>>>
>>>>
>>>> My question: Does Stork and/or DAGMan do anything intelligent about avoiding
>>>> recopying files?  The input files are identical for all thousand runs; only
>>>> the seed varies.  But I would like to have Condor manage each run
>>>> individually.  So does all the data and the executable get copied around a
>>>> thousand times, cleaned up after each run?  If the thousand reps are child
>>>> to the Stork job that transfers files in place, does everything just work
>>>> with no extraneous recopying of input data?
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Thomas Rowe
>>>>
>>>> _______________________________________________
>>>> Condor-users mailing list
>>>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>>>> subject: Unsubscribe
>>>> You can also unsubscribe by visiting
>>>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>>
>>>> The archives can be found at:
>>>> https://lists.cs.wisc.edu/archive/condor-users/
>>>>
>>>>
>>> _______________________________________________
>>> Condor-users mailing list
>>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/condor-users/
>> _______________________________________________
>> Condor-users mailing list
>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/condor-users/
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/