Re: [Condor-users] [condor-users] Iterative Computations [was: Does stork avoid retransfering data?]

[Douglas Thain <dthain@xxxxxx>]

Gabriele -

For that kind of application, I suggest that you go one level down in
our stack, and use the Work Queue library:
http://nd.edu/~ccl/software/workqueue/

Work Queue provides a fork-join programming interface that is Condor-compatible.
You write your app in C, Python, or Perl, link it against the WQ library,
then submit "workers" to Condor, which connect to the application and
start running components.

WQ has been used to implement things like genetic algorithms, replica
exchange, distributed
image processing -- anything that can be expressed in a fork-join style.

I suspect you would find it easier to write your application logic
(parsing files, checking conditions)
in a general purpose language, rather than trying to squeeze it into Makeflow.

Would that meet your needs?

Doug



On Thu, Mar 1, 2012 at 12:19 PM, Gabriel Mateescu
<gabriel.mateescu@xxxxxxxxx> wrote:
> I am thinking about using Makeflow to
> build a workflow engine that manages
> a genetic algorithm which contains
> two operations:
>
>  1. next_gen tool: generates the next generation
>     of candidates and partitions the candidates
>     in two disjoint parts, part1 and part2;
>
>  2. evaluate tool: computes the fitness of every
>      candidate in either the part1 or part2 input.
>
> A Makefile that encodes the workflow for a
> Genetic Algorithm may look like this:
>
>
>  part1 part2:  input.data
>                    next_gen input.data
>
>  part1_score:  part1
>                     evaluate part1 > part1_score
>
>  part2_score:  part2
>                     evaluate part2 > part2_score
>
>  stop-test:     part1 part2 part1_score part2_score
>                    if ( max(part1_score, part2_score) < 13) {
>                         update input.data part1 part2
>                         make part1 part2
>                    }
>
> The intent of the rule for the target stop-test is to
> specify that a new make cycle should be executed
> if max(part1_score, part2_score) >= 13.
>
> So it seems that in order to implement loops,
> one needs to support a conditional statement
> and the recursive invocation of make-flow.
>
> Thank you.
> Gabriel
>
>
>
>
>
> On Thu, Mar 1, 2012 at 5:46 PM, Douglas Thain <dthain@xxxxxx> wrote:
>> Gabriel -
>>
>> Makeflow does not currently do that; it is just a static DAG.
>>
>> However, this is an important workflow pattern that we have been thinking about,
>> and we could work with you to get something going in a way that is
>> Condor-compatible.
>>
>> Can you share some more detailed use cases?
>> Is there anyone else on the list interested in such a capability?
>>
>> Doug
>>
>>
>> On Thu, Mar 1, 2012 at 11:33 AM, Gabriel Mateescu
>> <gabriel.mateescu@xxxxxxxxx> wrote:
>>> Hi Doug,
>>>
>>> Is it possible to express with Makeflow
>>> iterative computations that repeat a set
>>> of steps until some stopping criterion
>>> is met?
>>>
>>> Thanks.
>>> Gabriel
>>>
>>>
>>>
>>>
>>> On Tue, Jan 11, 2011 at 5:55 PM, Douglas Thain <dthain@xxxxxx> wrote:
>>>> Thomas -
>>>>
>>>> You might consider using Makeflow for this task:
>>>> http://www.nd.edu/~ccl/software/makeflow
>>>>
>>>> The idea is that you express your tasks in Makeflow, submit a bunch of
>>>> 'worker' processes to Condor, and Makeflow will distribute tasks among
>>>> the workers.  If they have some common executables and input files,
>>>> they will be automatically cached at the workers, so you don't have to
>>>> keep transmitting them.
>>>>
>>>> Cheers,
>>>> Doug
>>>>
>>>> On Tue, Jan 11, 2011 at 8:27 AM, Rowe, Thomas <rowet@xxxxxxxxxx> wrote:
>>>>> I have to run a simulation about a thousand times with different seeds.  The
>>>>> simulation executable and data total about 100MB.  This sounds like a job
>>>>> for DAGMan & Stork, because this 100MB collection of files needs to get
>>>>> copied around reliably, and some large output files need to be transferred
>>>>> back to the originating machine reliably.
>>>>>
>>>>>
>>>>>
>>>>> My question: Does Stork and/or DAGMan do anything intelligent about avoiding
>>>>> recopying files?  The input files are identical for all thousand runs; only
>>>>> the seed varies.  But I would like to have Condor manage each run
>>>>> individually.  So does all the data and the executable get copied around a
>>>>> thousand times, cleaned up after each run?  If the thousand reps are child
>>>>> to the Stork job that transfers files in place, does everything just work
>>>>> with no extraneous recopying of input data?
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Thomas Rowe
>>>>>
>>>>> _______________________________________________
>>>>> Condor-users mailing list
>>>>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>>>>> subject: Unsubscribe
>>>>> You can also unsubscribe by visiting
>>>>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>>>
>>>>> The archives can be found at:
>>>>> https://lists.cs.wisc.edu/archive/condor-users/
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Condor-users mailing list
>>>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>>>> subject: Unsubscribe
>>>> You can also unsubscribe by visiting
>>>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>>
>>>> The archives can be found at:
>>>> https://lists.cs.wisc.edu/archive/condor-users/
>>> _______________________________________________
>>> Condor-users mailing list
>>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/condor-users/
>> _______________________________________________
>> Condor-users mailing list
>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/condor-users/
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/