Re: [HTCondor-users] high rate of killed jobs

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 3/28/2018 2:50 AM, Almudena Montiel wrote:
> Hello,
> 
> I am trying to understand this behaviour: I find very often that jobs 
> are exited with status 102. In the configuration we have defined not to 
> preempt neither kill jobs, these variables:
> 
>  ÂSUSPEND = FALSE
>  ÂPREEMPT = FALSE
>  ÂPREEMPTION_REQUIREMENTS = FALSE
>  ÂKILL = FALSE
> 
> One example:
> 

>From the example logs, it looks to me like HTCondor killed running job 698148.0 because back on the submit machine it was explicitly removed from the queue.  Ie, someone ran "condor_rm" on the job, or the job's PeriodicRemove expression became True.

Here is the telling line:

> 
> ShadowLog 
> 03/27/18 19:43:34 (698148.0) (2185496): Requesting graceful removal of job.

I assume job 698148.0 disappeared from the queue after this happened?  If yes, what does:

  condor_history 698148.0 -limit 1 

show?  condor_history is like condor_q, but for completed/removed jobs.  Does it show the job was removed (removed job will have Status = "X")?  The job classad likely also will contain a RemoveReason attribute stating why the job was removed. Some examples from my windows laptop (same idea on Linux):

C:\condor\log>condor_history 424.0 -limit 1
 ID     OWNER          SUBMITTED   RUN_TIME     ST COMPLETED   CMD
 424.0   tannenba        3/28 09:41   0+00:00:12 X         ???  c:\utils\sleep.exe 300

C:\condor\log>condor_history 424.0 -limit 1 -l | grep -i remove
OnExitRemove = true
PeriodicRemove = false
RemoveReason = "via condor_rm (by user tannenba)"

C:\condor\log>condor_history 424.0 -limit 1 -af JobStatus RemoveReason
3 via condor_rm (by user tannenba)

 
Hope the above helps,
Todd