[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [condor-users] running globus jobs without a shared filesystem

mcal00@xxxxxxxxxxxxx wrote:

There's a line in the message you mention that says:

"This isn't much of a problem for output files, as Condor can be told transfer
all output files from the job (those created or modified after the job began

Well, I could never get it to do it.

I believe the confusion here is whether you are talking about stage-back of files from the execution node to the gatekeeper node (which is what Alain was talking about), or whether you are talking about stage-back of files to the original point of submission, which is what I think you are talking about, right Mark?

In any case, there is _no_ option in Condor-G to automatically stage back output files to the submission site. I am right now updating the manual in several places where the different behavior for Globus universe is not made clear. The transfer-files options Alain was talking about only handle copying back files from the execution machine to the gatekeeper machine.

An additional transfer must usually take place between the gatekeeper and the submit machine. If you know the files in advance, you can specify them explicitly in the transfer_output_files list. This mechanism (new in Condor 6.5) uses GRAM file staging. I personally do not know if GRAM file staging is robust enough to handle large data flows.

Probably the most common technique is to submit a follow-up "fork" job that sends files back via gsiftp. It sounds like you have a solution similar to this which effectively spawns off a fork job automatically. You may want to investigate how well this scales, if you are likely to be submitting large numbers of jobs, because the gatekeeper tends to be quite heavily burdened.

Dan Bradley
University of Wisconsin, Condor Project

Condor Support Information: http://www.cs.wisc.edu/condor/condor-support/ To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with unsubscribe condor-users <your_email_address>