[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[HTCondor-users] Compress output/error/log files at the end of the job
- Date: Mon, 18 Apr 2022 12:03:09 +0200
- From: Stuart Walker <stuart.walker@xxxxxxx>
- Subject: [HTCondor-users] Compress output/error/log files at the end of the job
I have a slight problem that I wondered if someone could help me with.
My error, log and output files are set like this in my job submission script:
output = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/out/bdsim-$(Cluster)_$(Process).out
error = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/error/bdsim-$(Cluster)_$(Process).error
log = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-$(Cluster)_$(Process).log
However sadly for my simulations that I am running there are a lot of these files, which are also very big, and my quota is really quite small. So I want to compress them at the end of each job. How is this possible? I tried compress_files but it does not work. I can't compress at the end of the executable because it would presumably end in a malformed log file as the gzip could end up appended to by stdout/err/log. So what is the solution here?
Thank you for your help.