[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Compress output/error/log files at the end of the job

On 4/18/22 05:03, Stuart Walker wrote:

I have a slight problem that I wondered if someone could help me with.

My error, log and output files are set like this in my job submission script:

output              = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/out/bdsim-$(Cluster)_$(Process).out
error               = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/error/bdsim-$(Cluster)_$(Process).error
log                 = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-$(Cluster)_$(Process).log

Hi Stuart:

Looking at the log file first, there shouldn't be too many bytes written to it per job, so I hope that's not a problem for your quota. HTCondor does support safely sharing one big log file for many jobs -- perhaps that would help with minimizing the number of files or inodes. You could set

log = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-some-workflow-name.log

Then all the jobs in some workflow that share that log file will safely append to it.

Now, for the output and error files, I don't think we have a good way in HTCondor proper to compress them. If these jobs are part of a DAG, then a DAG postscript (which runs on the submit machine) could compress the files. However, I suspect it is better overall to explicitly compress them as part of the job on the execute side.