[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] centrally force removal after some time even if leave_in_queue is true?
- Date: Wed, 31 Oct 2018 18:57:16 +0100
- From: Andrea Sartirana <sartiran@xxxxxxxxxxxx>
- Subject: Re: [HTCondor-users] centrally force removal after some time even if leave_in_queue is true?
thanks! I'll give it a try on our test cluster.
On 31/10/2018 16:32, Todd Tannenbaum wrote:
On 10/31/2018 5:49 AM, Andrea Sartirana wrote:
much is in the title.
I was wondering if there is a way to force removal from the queue of the
X state jobs after some centrally defined time even if the
leave_in_queue expression given by the user at submission still
evaluates to true. I'm running 8.6.0, vanilla universe, direct submission.
I've tried to include garbage collecting of the remove jobs in the
SYSTEM_PERIODIC_REMOVE but this does not seem to have the desired effect.
There may be an easier way, but a quick thought is you could use Job Transforms to accomplish the above. Job Transforms allow you, the administrator, to edit job classads upon submission --- see this section of the v8.6 manual:
So the idea here is to configure your schedd to edit the user's leave_in_queue expression (which ends up in the job classad as attribute LeaveJobInQueue) so that it will always evaluate to False for X state jobs after a specified amount of time, else fall back to whatever the user wanted.
Try appending the below to the HTCondor configuration (it will be used by your submit machines, and ignored on machines not running a schedd) to allow jobs in X state to leave the queue after 120 seconds regardless of what the user's submit file says:
JOB_TRANSFORM_NAMES = $(JOB_TRANSFORM_NAMES) LeaveInQueue
copy_LeaveJobInQueue = "SubmitterLeaveJobInQueue";
set_LeaveJobInQueue = (JobStatus == 3 && (time() - EnteredCurrentStatus) > 120) ? False : SubmitterLeaveJobInQueue
Warning - the above is off the top of my head, I did not test it.
Seems like HTCondor would benefit from a SYSTEM_LEAVE_IN_QUEUE knob to make doing the above simpler. But Job Transforms are a pretty powerful generic tool.
Hope the above helps.