[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Condor feature suggestion: automatically compressed output files




Hi everybody,

I recently asked for a similar feature. For now I just wrote a wrapper script which compresses the output before it is returned to the submitting machine.
Regards, Matthias

Matthias Röhm, University Ulm,
DaimlerChrysler AG, Research Center Ulm,
Department for Inline Inspection and
Data Mining Solutions, REI/DM
P.O. Box 23 60, 89013 Ulm, Germany

Phone:  +49 (0)731 505 4864
Fax:      +49 (0)711 3052183085
Email:   mailto:Uni-Ulm.M.Roehm@xxxxxxxxxxxxxxxxxxx



> Hi all,
>
> Recently I find myself using Condor in the following way: I have a plain
> executable that produces large standard output, which I need stored for
> further analysis. In each execution, I need to run that executable under lots
> of different parameters, generating hundreds of jobs - and I would like to be
> able to store these outputs for further reference.
>
> The problem is, the outputs take a lot of place, and I need to use bzip2 to
> compress them after they execute. However, I cannot just run bzip2 whenever I
> want, since some of the jobs can be still running, and so compressing the
> output file would result in broken output.
>
> A solution would be to have Condor automatically compress the file
> contents if
> the output (and .err) file name ends with ".bz2" or ".gz". This should be
> easy to implement as bzip2 library is publicly available.
>
> I think this feature would be useful not only to me, but to other users as
> well.
>
> Regards,
> Alex
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at either
> https://lists.cs.wisc.edu/archive/condor-users/
> http://www.opencondor.org/spaces/viewmailarchive.action?key=CONDOR