Re: [Condor-users] retrieving output subdirectories

[Jan Ploski <Jan.Ploski@xxxxxxxx>]

Ian Chesal wrote:
> > This has been a very interesting thread but assumes Linux execute
> > computers. Has anyone done similar work in the Windows space?
>
> Yup. Same approach except we use Perl (we actually use Perl for all
> platforms) to write the wrapper. And Gnu Utils for Windows to get tar
> and gzip.

I'd like to chime in that the tar/zip solution, while simple and 
general, does not suit all applications well. It becomes awkward when 
the output files (whose names may well depend on the job parameters and 
be unknown at job submission time) are several gigabytes in size. In 
such case, the tar/zip step puts an additional strain on the I/O and the 
disk capacity of the execution host.

This becomes particularly painful in Grid/cluster context where the same 
frontend host with one NFS file system normally supports multiple 
running jobs. Sure, it is possible to use third-party tools to 
recursively stage out the entire working directory tree, but it 
certainly would make life easier if Condor would simply accept a 
directory in the output_files list...

Best regards,
Jan Ploski