Hi Jon
We archive all the raw history files (and rotated files) from all of our submit nodes (21) onto a separate machine
and store them in a hierarchical manner based on separate folders for each submit node. The archiving script only
copies history files it doesnât already have. This means we never miss history files that are rotated out of existence.
We then have scripts on this separate machine that can query (using condor_history) this archive for various types
of information, e.g. all jobs run by a particular user between 2 different dates, all jobs run on one particular submit
node on a certain day or month or year, etc.
With the raw history files archived in one location you do not need to query multiple schedds to get this
sort of information. It also means that you can access more/different job info rather than just what you
happen to request by running condor_history directly and archiving that subset of the full job info.
Just my 2c worth.
😊
Cheers
Greg
P.S. Our current history file configs:
ENABLE_HISTORY_ROTATION = True
MAX_HISTORY_LOG = 100000000
MAX_HISTORY_ROTATIONS = 1000
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx>
On Behalf Of Jonathan Martin
Sent: Thursday, 18 November 2021 4:58 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] History file rotation - reading from backups
Thanks Jaime, I actually didnât know condor_history looked through rotated files as well, I had just made an assumption based on the info in the
docs:
If no history files are specified with the -file option, the local history file
as specified in HTCondorâs configuration file
($(SPOOL)/history
by default) is read.
I will go ahead and confirm that it is actually reading through backups as well, but it sounds like this is a non-issue.
Thanks,
Jon
Are you able to adjust the history rotation configuration on these schedds? You can set the rotation trigger based on size or time, and the number of rotated files. condor_history (whether remote or local) should look
through all of the rotated files automatically. If your remote condor_history queries arenât reading all of the rotated files, that sounds like a bug we should investigate.
- Jaime
Iâm building a data archiving system to warehouse historical job data and machine metrics, through various commands like condor_history and condor_status.
One issue Iâm facing is job history file rotation on remote condor schedds â by default, history files are rotated into backup files once they hit 20 MB. The system
Iâm building is polling all schedds in the pool with a condor_history command every few minutes, but some jobs will slip through the cracks (history file gets rotated since last poll). Additionally, Iâm unable to retrieve the names of the backup files since
the HTCondor folder on each schedd is not exposed via a network share.
Curious if there are any built-in config options that can help address this, without turning off rotation. Examples:
- Overlap
the last N jobs in the backup file and the new history file
- Have
condor_history additionally read through successive backup files until HISTORY_HELPER_MAX_HISTORY is hit (or reach end of all files)
This communication (both the message and any attachments or links) is confidential and only
intended for the use of the person or persons to whom it is addressed unless we have expressly authorized otherwise. It also may contain information that is protected by solicitor-client privilege. If you are reading this communication and are not an addressee
or authorized representative of an addressee, we hereby notify you that any distribution, copying or other use of it without our express authorization is strictly prohibited. If you have received this communication in error, please delete both the message
and any attachments from your system and notify us immediately by e-mail or phone. In addition, we note that this communication and its transmission of data have not been secured by encryption. Therefore, we are not able to confirm or guarantee that the communication
has not been intercepted, amended, or read by an unintended third party.
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with
a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
This communication (both the message and any attachments or links) is confidential and only intended for the use of the person or persons to whom it is addressed unless
we have expressly authorized otherwise. It also may contain information that is protected by solicitor-client privilege. If you are reading this communication and are not an addressee or authorized representative of an addressee, we hereby notify you that
any distribution, copying or other use of it without our express authorization is strictly prohibited. If you have received this communication in error, please delete both the message and any attachments from your system and notify us immediately by e-mail
or phone. In addition, we note that this communication and its transmission of data have not been secured by encryption. Therefore, we are not able to confirm or guarantee that the communication has not been intercepted, amended, or read by an unintended third
party.
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif){kind=logo}