[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Condor-users] Condor feature suggestion: automatically compressed output files
- Date: Thu, 27 Jul 2006 17:12:32 +0300
- From: Alex Gontmakher <gsasha@xxxxxxxxxxxxxxxxx>
- Subject: [Condor-users] Condor feature suggestion: automatically compressed output files
Recently I find myself using Condor in the following way: I have a plain
executable that produces large standard output, which I need stored for
further analysis. In each execution, I need to run that executable under lots
of different parameters, generating hundreds of jobs - and I would like to be
able to store these outputs for further reference.
The problem is, the outputs take a lot of place, and I need to use bzip2 to
compress them after they execute. However, I cannot just run bzip2 whenever I
want, since some of the jobs can be still running, and so compressing the
output file would result in broken output.
A solution would be to have Condor automatically compress the file contents if
the output (and .err) file name ends with ".bz2" or ".gz". This should be
easy to implement as bzip2 library is publicly available.
I think this feature would be useful not only to me, but to other users as