[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Compress output/error/log files at the end of the job
- Date: Mon, 18 Apr 2022 13:57:42 -0500
- From: Greg Thain <gthain@xxxxxxxxxxx>
- Subject: Re: [HTCondor-users] Compress output/error/log files at the end of the job
On 4/18/22 05:03, Stuart Walker wrote:
I have a slight problem that I wondered if someone could help me with.
My error, log and output files are set like this in my job submission script:
output = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/out/bdsim-$(Cluster)_$(Process).out
error = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/error/bdsim-$(Cluster)_$(Process).error
log = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-$(Cluster)_$(Process).log
Looking at the log file first, there shouldn't be too many bytes written
to it per job, so I hope that's not a problem for your quota.Â HTCondor
does support safely sharing one big log file for many jobs -- perhaps
that would help with minimizing the number of files or inodes.Â You
Then all the jobs in some workflow that share that log file will safely
append to it.
Now, for the output and error files, I don't think we have a good way in
HTCondor proper to compress them.Â If these jobs are part of a DAG, then
a DAG postscript (which runs on the submit machine) could compress the
files.Â However, I suspect it is better overall to explicitly compress
them as part of the job on the execute side.