Re: [HTCondor-users] Compress output/error/log files at the end of the job

[Greg Thain <gthain@xxxxxxxxxxx>]

On 4/18/22 05:03, Stuart Walker wrote:
> Hello,
>
> I have a slight problem that I wondered if someone could help me with.
>
> My error, log and output files are set like this in my job submission script:
>
> output              = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/out/bdsim-$(Cluster)_$(Process).out
> error               = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/error/bdsim-$(Cluster)_$(Process).error
> log                 = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-$(Cluster)_$(Process).log


Hi Stuart:

Looking at the log file first, there shouldn't be too many bytes written 
to it per job, so I hope that's not a problem for your quota. HTCondor 
does support safely sharing one big log file for many jobs -- perhaps 
that would help with minimizing the number of files or inodes. You 
could set

log = 
/afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-some-workflow-name.log

Then all the jobs in some workflow that share that log file will safely 
append to it.


Now, for the output and error files, I don't think we have a good way in 
HTCondor proper to compress them. If these jobs are part of a DAG, then 
a DAG postscript (which runs on the submit machine) could compress the 
files. However, I suspect it is better overall to explicitly compress 
them as part of the job on the execute side.


-greg