[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] [CondorLIGO] Preempted jobs not carrying their ImageSize for the next match?



Forgot the attachment :(

On Wed, Mar 04, 2015 at 05:09:53PM +0100, Steffen Grunewald wrote:
> On Wed, Mar 04, 2015 at 09:37:04AM +0100, Steffen Grunewald wrote:
> > On Tue, Mar 03, 2015 at 10:36:30AM -0600, Greg Thain wrote:
> > > On 03/03/2015 05:31 AM, Steffen Grunewald wrote:
> > > >I have a couple of users who underestimate the memory their jobs
> > > >would attempt to allocate, and as a result some worker nodes end
> > > >up swapping heavily.
> > > >I tried to get those jobs preempted, and sent back into the queue
> > > >with their updated (ImageSize) request_memory:
> > > >
> > > ># Let job use its declared amount of memory and some more
> > > >MEMORY_EXTRA            = 2048
> > > >MEMORY_ALLOWED          = (Memory + $(MEMORY_EXTRA)*Cpus)
> > > ># Get the current footprint
> > > >MEMORY_CURRENT          = (ImageSize/1024)
> > > ># Exceeds expectations?
> > > >MEMORY_EXCEEDED         = $(MEMORY_CURRENT) > $(MEMORY_ALLOWED)
> > > ># If exceeding, preempt
> > > >#[preset]PREEMPT        = False
> > > >PREEMPT                 = ($(PREEMPT)) || ($(MEMORY_EXCEEDED))
> > > >WANT_SUSPEND            = False
> > > >
> > > >
> > > This should all work.
> 
> And indeed it does, see below. No need to debug right away.
> 
> > For the "exclude parallel universe from preemption" part, I will now use
> > PREEMPT                 = ($(PREEMPT)) || ($(MEMORY_EXCEEDED) && (JobUniverse =!= 11))
> 
> I had no opportunity to test this part yet...
> 
> > (and I'm afraid "PREEMPT_VANILLA = False" was the cause for preemption not
> > happening to vanilla universe jobs... removed that one from the config now)
> 
> This one seems to have made the real difference.
> Now preemption *does* happen, as one of the users quickly learned:
> 
> > Hi Steffen,
> > 
> [...]
> > This is a plotting job that looks like it's trying to plot a vast
> > number of points .... and the job does seem to get kicked pretty
> > quickly. However, the memory requirement doesn't get above 15GB (even
> > though the reported condor size is bigger) and so it keeps matching,
> > evicting, matching, evicting.
> > 
> > 15GB is the original memory request of this job.
> 
> As the job was requesting more than the 15000 MB asked for in
> request_memory, at 17090 MB the additional margin of 2 GB was
> also reached, and the job was evicted from the slot it was running in.
> 
> Now I would have expected that the job would go back to the queue,
> with its updated ImageSize as new RequestMemory setting, but this
> apparently didn't happen.
> Well, it got re-scheduled, but kept its request size at 15000, 
> resulting in a never-ending loop, visiting more and more worker nodes.
> I have extracted the corresponding messages from the overall log file.
> 
> Condor version is 8.3.3 (this isn't supposed to behave different from 
> 8.2.7 in this particular respect, is it?)
> 
> Log file extract attached.
> 
> Any ideas how to help the user (except suggesting to specify 50000,
> and wait for a machine big enough to take this)?
> I'm afraid this is only the tip of a bigger iceberg...
> 
> Thanks,
>  Steffen
> 
> -- 
> Steffen Grunewald * Cluster Admin * steffen.grunewald(*)aei.mpg.de
> MPI f. Gravitationsphysik (AEI) * Am Mühlenberg 1, D-14476 Potsdam
> http://www.aei.mpg.de/ * ------- * +49-331-567-{fon:7274,fax:7298}
> _______________________________________________
> Condorligo mailing list
> Condorligo@xxxxxxxxxx
> http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/condorligo

-- 
Steffen Grunewald * Cluster Admin * steffen.grunewald(*)aei.mpg.de
MPI f. Gravitationsphysik (AEI) * Am Mühlenberg 1, D-14476 Potsdam
http://www.aei.mpg.de/ * ------- * +49-331-567-{fon:7274,fax:7298}
000 (1172546.000.000) 03/04 14:37:36 Job submitted from host: <10.150.100.40:52088>
    DAG Node: plot_snrchi-_FULL_DATA-H1_ID32_ID0011928
    pool:local

001 (1172546.000.000) 03/04 14:37:39 Job executing on host: <10.150.85.6:40762>

006 (1172546.000.000) 03/04 14:37:47 Image size of job updated: 75
	3  -  MemoryUsage of job (MB)
	2936  -  ResidentSetSize of job (KB)

006 (1172546.000.000) 03/04 14:40:24 Image size of job updated: 19267108
	15533  -  MemoryUsage of job (MB)
	15905696  -  ResidentSetSize of job (KB)

004 (1172546.000.000) 03/04 14:40:24 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:47, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338416
	   Memory (MB)          :    15533    15000     15000

001 (1172546.000.000) 03/04 14:40:42 Job executing on host: <10.150.85.6:40762>

004 (1172546.000.000) 03/04 14:42:56 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338415
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 14:43:03 Job executing on host: <10.150.85.6:40762>

004 (1172546.000.000) 03/04 14:45:12 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338415
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 14:45:22 Job executing on host: <10.150.85.6:40762>

004 (1172546.000.000) 03/04 14:47:43 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:45, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338415
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 14:48:05 Job executing on host: <10.150.85.6:40762>

004 (1172546.000.000) 03/04 14:50:18 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338414
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 14:50:44 Job executing on host: <10.150.85.6:40762>

004 (1172546.000.000) 03/04 14:52:58 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338414
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 14:53:03 Job executing on host: <10.150.87.5:49979>

004 (1172546.000.000) 03/04 14:55:03 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:41, Sys 0 00:00:06  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338415
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 14:55:25 Job executing on host: <10.150.85.6:40762>

004 (1172546.000.000) 03/04 14:57:39 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338414
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 14:58:05 Job executing on host: <10.150.85.6:40762>

004 (1172546.000.000) 03/04 15:00:21 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:45, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338413
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:00:26 Job executing on host: <10.150.86.5:36515>

004 (1172546.000.000) 03/04 15:02:40 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:45, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338412
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:02:47 Job executing on host: <10.150.86.5:36515>

004 (1172546.000.000) 03/04 15:04:55 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:45, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338412
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:05:08 Job executing on host: <10.150.86.5:36515>

004 (1172546.000.000) 03/04 15:07:12 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:45, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338412
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:07:35 Job executing on host: <10.150.86.5:36515>

004 (1172546.000.000) 03/04 15:09:49 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:43, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338412
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:10:28 Job executing on host: <10.150.85.4:54745>

004 (1172546.000.000) 03/04 15:12:28 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:41, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338393
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:12:49 Job executing on host: <10.150.86.1:43439>

004 (1172546.000.000) 03/04 15:14:48 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:43, Sys 0 00:00:06  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338407
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:15:18 Job executing on host: <10.150.86.1:43439>

004 (1172546.000.000) 03/04 15:17:32 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:43, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338406
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:18:03 Job executing on host: <10.150.86.1:43439>

004 (1172546.000.000) 03/04 15:20:01 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:06  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338407
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:20:25 Job executing on host: <10.150.86.1:43439>

004 (1172546.000.000) 03/04 15:22:24 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:42, Sys 0 00:00:06  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338407
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:23:23 Job executing on host: <10.150.86.1:43439>

004 (1172546.000.000) 03/04 15:25:36 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:45, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338406
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:26:36 Job executing on host: <10.150.86.1:43439>

004 (1172546.000.000) 03/04 15:28:35 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:42, Sys 0 00:00:06  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338407
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:29:32 Job executing on host: <10.150.86.1:43439>

004 (1172546.000.000) 03/04 15:31:46 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338404
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:32:01 Job executing on host: <10.150.86.1:43439>

004 (1172546.000.000) 03/04 15:34:05 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:42, Sys 0 00:00:06  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338405
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:34:33 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 15:36:49 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:45, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338413
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:37:21 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 15:39:36 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:43, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338413
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:39:53 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 15:42:07 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:46, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338415
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:42:25 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 15:44:39 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338415
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:45:37 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 15:47:51 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:47, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338414
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:48:31 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 15:50:46 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338413
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:50:53 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 15:53:02 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:46, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338414
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:53:09 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 15:55:22 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338413
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:55:29 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 15:57:39 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:45, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338413
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 15:57:40 Job executing on host: <10.150.91.1:33832>

004 (1172546.000.000) 03/04 15:59:55 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338412
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 16:00:13 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 16:02:29 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:47, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338413
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 16:03:11 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 16:05:25 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:43, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338413
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 16:05:47 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 16:08:01 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:45, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338412
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 16:08:05 Job executing on host: <10.150.91.2:41583>

004 (1172546.000.000) 03/04 16:10:20 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:46, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338416
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 16:10:23 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 16:12:37 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:45, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338412
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 16:12:51 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 16:14:54 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:43, Sys 0 00:00:06  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338412
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 16:15:55 Job executing on host: <10.150.90.1:36625>

004 (1172546.000.000) 03/04 16:18:09 Job was evicted.
	(0) Job was not checkpointed.
		Usr 0 00:00:44, Sys 0 00:00:07  -  Run Remote Usage
		Usr 0 00:00:00, Sys 0 00:00:00  -  Run Local Usage
	0  -  Run Bytes Sent By Job
	0  -  Run Bytes Received By Job
	Partitionable Resources :    Usage  Request Allocated
	   Cpus                 :                 1         1
	   Disk (KB)            :       75        0   1338411
	   Memory (MB)          :    17090    15000     15000

001 (1172546.000.000) 03/04 16:18:55 Job executing on host: <10.150.90.3:34770>