[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] retrieving output subdirectories
- Date: Mon, 04 Feb 2008 22:04:41 +0100
- From: Jan Ploski <Jan.Ploski@xxxxxxxx>
- Subject: Re: [Condor-users] retrieving output subdirectories
Ian Chesal wrote:
This has been a very interesting thread but assumes Linux execute
computers. Has anyone done similar work in the Windows space?
Yup. Same approach except we use Perl (we actually use Perl for all
platforms) to write the wrapper. And Gnu Utils for Windows to get tar
I'd like to chime in that the tar/zip solution, while simple and
general, does not suit all applications well. It becomes awkward when
the output files (whose names may well depend on the job parameters and
be unknown at job submission time) are several gigabytes in size. In
such case, the tar/zip step puts an additional strain on the I/O and the
disk capacity of the execution host.
This becomes particularly painful in Grid/cluster context where the same
frontend host with one NFS file system normally supports multiple
running jobs. Sure, it is possible to use third-party tools to
recursively stage out the entire working directory tree, but it
certainly would make life easier if Condor would simply accept a
directory in the output_files list...