Re: [Condor-users] Does stork avoid retransfering data?

[Douglas Thain <dthain@xxxxxx>]

Thomas -

You might consider using Makeflow for this task:
http://www.nd.edu/~ccl/software/makeflow

The idea is that you express your tasks in Makeflow, submit a bunch of
'worker' processes to Condor, and Makeflow will distribute tasks among
the workers.  If they have some common executables and input files,
they will be automatically cached at the workers, so you don't have to
keep transmitting them.

Cheers,
Doug

On Tue, Jan 11, 2011 at 8:27 AM, Rowe, Thomas <rowet@xxxxxxxxxx> wrote:
> I have to run a simulation about a thousand times with different seeds.  The
> simulation executable and data total about 100MB.  This sounds like a job
> for DAGMan & Stork, because this 100MB collection of files needs to get
> copied around reliably, and some large output files need to be transferred
> back to the originating machine reliably.
>
>
>
> My question: Does Stork and/or DAGMan do anything intelligent about avoiding
> recopying files?  The input files are identical for all thousand runs; only
> the seed varies.  But I would like to have Condor manage each run
> individually.  So does all the data and the executable get copied around a
> thousand times, cleaned up after each run?  If the thousand reps are child
> to the Stork job that transfers files in place, does everything just work
> with no extraneous recopying of input data?
>
>
>
> Thanks,
>
> Thomas Rowe
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
>