Re: [Condor-users] Condor feature suggestion: automatically compressed output files

[Alex Gontmakher <gsasha@xxxxxxxxxxxxxxxxx>]

Oh, checkpoint is a problem indeed, didn't think of that (but can't the state 
of the compression algorithm be checkpointed as well?)
> What it should do in the standard universe on checkpoint is not so
> clear. Recommencing output to the stream might be a best problematic
> and at worst require new protocol level functionality.
> Streaming output may also need changes. No idea about Globus, GAHP, GCB
> etc.. What compression to use is also something that is best left in the
> hands of the app writer (who would know whether the significant
> additional cpu cost of bzip2 or 7z was worth it against using a
> straight deflate).
Oh, my suggestion was to analyze the extension of the output file and use a 
corresponding engine, i.e., use bzip2 if the file name ends with ".bz2" etc. 
This way, the user gets a say on what engine to use, and the decompression 
programs automatically will recognize the file.
>
> Providing something like deflate as a useful default does sound nice -
> but would likely require considerable restrictions on use to prevent
> it causing a lot more recoding that initially expected. Due to this
> the ever present "wrap it in a script" option is always likely to be
> the default response since it allows so much more flexibility and
> power.
Er??? I can't comment on the amount of recoding necessary - or at least, my 
estimate is guesswork as I don't really know the architecture of Condor, but 
I don't think my proposed solution is that unflexible...
> Admittedly this is at the cost of  one additional line/ entry in the
> submit script to transfer the 'real' exe as well as the script') and
> of course the lack of standard universe (but as we mentioned before
> this has complications if you do that anyways)
> For the standard universe there is a reasonable likelihood that, if
> you can relink the app you can prob also change it to output
> differently.
Well then, how do you propose changing a C program to compress its standard 
output?
>
> The only really big saving you would get in complexity is if the job
> is cross platform, then wrapping in a script means creating multiple
> scripts each doing it differently. I admit this sounds nice but I
Well, our cluster happens to be cross platform indeed - a mix of Intel 
machines and PowerPC blades... and the code I run on it is compiled and works 
on both.
> don't know how many people use this functionality to make the
> additional code/maintenance complexity worth it (that's a question for
> the cs.wisc guys and gals :)