Re: [Condor-users] Problem in Getting Result Back

["Matt Hope" <matthew.hope@xxxxxxxxx>]

On 5/2/06, Natarajan, Senthil <senthil@xxxxxxxx> wrote:
> Hi,
>
> There is a job which produces lot of result files and creates a directory at
> the end and put the result files over.

you mean it writes the results to this sub dir or that, prior to
completing it copies them to the subdir?

> But condor doesn't get those directories from the execution node. The output
> file shows the list of files it created and put it in the new directory.

Condor only transfers files in the root of the initial working directory.

Thus simply not copying the results to a subdirectory or changing the
job so that it doesn't write them there in the first place (or just
moves them into the initial directory before exiting) will suffice.

> Is there any way we can get the results back from the execution node.

Potentially you may want to consider taring/zipping the files if they
are large prior to exiting, thus reducing the data transfer at the
expense of more time computing on the execute node. this is a tricky
balance and depends whther you are happy with your results being
tared/zipped up.

If you have many jobs running at once from one submit machine and the
amount of data transferred is large there is a fair chance expending
the additional effort of compressing the result data spread across the
execute nodes rather than heavy network load on the submit machine
will get you better throughput. As with everything in the performance
world Your Milage May Vary and Test It Yourself

Matt