I’m building a data archiving system to warehouse historical job data and machine metrics, through various commands like condor_history and condor_status.
One issue I’m facing is job history file rotation on remote condor schedds – by default, history files are rotated into backup files once they hit 20 MB. The system I’m building is polling all schedds in the pool with a condor_history command every few minutes, but some jobs will slip through the cracks (history file gets rotated since last poll). Additionally, I’m unable to retrieve the names of the backup files since the HTCondor folder on each schedd is not exposed via a network share.
Curious if there are any built-in config options that can help address this, without turning off rotation. Examples:
- Overlap the last N jobs in the backup file and the new history file
- Have condor_history additionally read through successive backup files until HISTORY_HELPER_MAX_HISTORY is hit (or reach end of all files)
This communication (both the message and any attachments or links) is confidential and only intended for the use of the person or persons to whom it is addressed unless we have expressly authorized otherwise. It also may contain information that is protected by solicitor-client privilege. If you are reading this communication and are not an addressee or authorized representative of an addressee, we hereby notify you that any distribution, copying or other use of it without our express authorization is strictly prohibited. If you have received this communication in error, please delete both the message and any attachments from your system and notify us immediately by e-mail or phone. In addition, we note that this communication and its transmission of data have not been secured by encryption. Therefore, we are not able to confirm or guarantee that the communication has not been intercepted, amended, or read by an unintended third party.