Mailing List Archives

Public Access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] condor-g data transfer mechanism

Hi Miron,


Yeah I know. My main concern was about dependent jobs on the same grid site. Does condor-g use the same transfer mechanism, meaning that when a job needs an input, does it check the submitter site directly to get the input data or does it check whether the input by any chance is available in the same grid site as the job is running and in case of not finding it ask the submitter for it?

Thanks,
Maryam





On Wed, Dec 10, 2008 at 3:44 AM, Miron Livny <miron@xxxxxxxxxxx> wrote:
Maryan,

Moving the data directly between the two execution nodes assumes that
the two sites are willing to store the data regardless of whether your
job is running at the site or is queued somewhere. This requires someone
to manage the storage on these sites. Not easy!

Miron



Maryam Khademi wrote:
> Hi Ewa,
>
>
> Yeah, I am pretty familiar with Pegasus. Had even long question&answer
> with your developers (Garang and the rest) before.
>
> You are right, it is of course better to transfer data just in case of
> need back to submitter. But my question was about condor-g mechanism
> itself and how it works!
>
> One of the guys mentioned in answer to this post, that using the
> transfer_input_files, and transfer_output_files parameter causes
> submitter to act as an intermediary. I am wondering using the pure
> Condor-G (i.e., not to add stage-in/out tasks to DAG), would it possible
> to transfer intermediate data between grid-sites without submitter
> involvement?! If so, then which kind of parameters should I use in my
> submit files?
>
> Many Thanks,
>
> Maryam
>
> On Sat, Dec 6, 2008 at 7:05 AM, Ewa Deelman <deelman@xxxxxxx
> <mailto:deelman@xxxxxxx>> wrote:
>
>     Hi Maryam,
>
>
>
>     I am not sure what you mean by Condor-G meta-scheduling.
>
>     In our work we use the Pegasus Workflow Management System
>     (http://pegasus.isi.edu) to map an abstract (resource-independent)
>     DAG onto the distributed resources. As part of the mapping, Pegasus
>     adds nodes to the DAG to stage the data to the execution sites and
>     move the results off.
>
>     Our workflow execution engine, DAGMan uses Condor-G to send the jobs
>     in the workflow to the grid sites. The results do not have to come
>     back to the submitter. If you don't want to save the intermediate
>     products, they will just exist on the remote sites for the duration
>     of the workflow and only the final data will be brought back to
>     where you want it to go.  If intermediate data needs to be moved
>     between sites, it will be done via 3^rd party, so that the submitter
>     is not involved.
>
>
>
>     For large data is it usually better to just transfer it from A to B
>     directly.  When jobs are mapped to the same site and there is shared
>     file system on that site, B can access the data directly.
>
>
>
>     Please let me know if you have any questions,
>
>
>
>     Thanks,
>
>     -Ewa
>
>
>
>
>
>     *From:* condor-users-bounces@xxxxxxxxxxx
>     <mailto:condor-users-bounces@xxxxxxxxxxx>
>     [mailto:condor-users-bounces@xxxxxxxxxxx
>     <mailto:condor-users-bounces@xxxxxxxxxxx>] *On Behalf Of *Maryam Khademi
>     *Sent:* Thursday, December 04, 2008 7:06 PM
>     *To:* Condor-Users Mail List
>     *Subject:* [Condor-users] condor-g data transfer mechanism
>
>
>
>     Hi,
>
>     I am using Condor-G to execute a DAG. As I understand its mechanism,
>     the meta-scheduler dispatches the jobs among different grid sites.
>     Meanwhile, all the intermediate data are transfered to the submitter
>     machine, so that if a job on grid site A needs the output of another
>     job on grid site B, it can have it through submitter. Is that
>     correct? Following this method, how about dependent jobs mapped to
>     the same grid sites?! Is the input data again being asked from
>     submitter?
>
>     Thanks,
>     Maryam
>
>
>     _______________________________________________
>     Condor-users mailing list
>     To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
>     <mailto:condor-users-request@xxxxxxxxxxx> with a
>     subject: Unsubscribe
>     You can also unsubscribe by visiting
>     https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
>     The archives can be found at:
>     https://lists.cs.wisc.edu/archive/condor-users/
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/