[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [condor-users] running globus jobs without a shared filesystem
- Date: Tue, 09 Dec 2003 17:18:38 -0600
- From: Dan Bradley <dan@xxxxxxxxxxxx>
- Subject: Re: [condor-users] running globus jobs without a shared filesystem
There's a line in the message you mention that says:
"This isn't much of a problem for output files, as Condor can be told transfer
all output files from the job (those created or modified after the job began
Well, I could never get it to do it.
I believe the confusion here is whether you are talking about stage-back
of files from the execution node to the gatekeeper node (which is what
Alain was talking about), or whether you are talking about stage-back of
files to the original point of submission, which is what I think you are
talking about, right Mark?
In any case, there is _no_ option in Condor-G to automatically stage
back output files to the submission site. I am right now updating the
manual in several places where the different behavior for Globus
universe is not made clear. The transfer-files options Alain was
talking about only handle copying back files from the execution machine
to the gatekeeper machine.
An additional transfer must usually take place between the gatekeeper
and the submit machine. If you know the files in advance, you can
specify them explicitly in the transfer_output_files list. This
mechanism (new in Condor 6.5) uses GRAM file staging. I personally do
not know if GRAM file staging is robust enough to handle large data flows.
Probably the most common technique is to submit a follow-up "fork" job
that sends files back via gsiftp. It sounds like you have a solution
similar to this which effectively spawns off a fork job automatically.
You may want to investigate how well this scales, if you are likely to
be submitting large numbers of jobs, because the gatekeeper tends to be
quite heavily burdened.
University of Wisconsin, Condor Project
Condor Support Information:
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>