[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] retrieving output subdirectories

Ian Chesal wrote:
This has been a very interesting thread but assumes Linux execute
computers. Has anyone done similar work in the Windows space?

Yup. Same approach except we use Perl (we actually use Perl for all
platforms) to write the wrapper. And Gnu Utils for Windows to get tar
and gzip.

I'd like to chime in that the tar/zip solution, while simple and general, does not suit all applications well. It becomes awkward when the output files (whose names may well depend on the job parameters and be unknown at job submission time) are several gigabytes in size. In such case, the tar/zip step puts an additional strain on the I/O and the disk capacity of the execution host.

This becomes particularly painful in Grid/cluster context where the same frontend host with one NFS file system normally supports multiple running jobs. Sure, it is possible to use third-party tools to recursively stage out the entire working directory tree, but it certainly would make life easier if Condor would simply accept a directory in the output_files list...

Best regards,
Jan Ploski