Re: [HTCondor-users] [CondorLIGO] Preempted jobs not carrying their ImageSize for the next match?

["Peter F. Couvares" <pfcouvar@xxxxxxx>]

Is the ImageSize issue you describe below the same as <https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=3766>?

-Peter


On Mar 4, 2015, at 11:09 AM, Steffen Grunewald <Steffen.Grunewald@xxxxxxxxxx> wrote:
> 
> On Wed, Mar 04, 2015 at 09:37:04AM +0100, Steffen Grunewald wrote:
>> On Tue, Mar 03, 2015 at 10:36:30AM -0600, Greg Thain wrote:
>>> On 03/03/2015 05:31 AM, Steffen Grunewald wrote:
>>>> I have a couple of users who underestimate the memory their jobs
>>>> would attempt to allocate, and as a result some worker nodes end
>>>> up swapping heavily.
>>>> I tried to get those jobs preempted, and sent back into the queue
>>>> with their updated (ImageSize) request_memory:
>>>> 
>>>> # Let job use its declared amount of memory and some more
>>>> MEMORY_EXTRA            = 2048
>>>> MEMORY_ALLOWED          = (Memory + $(MEMORY_EXTRA)*Cpus)
>>>> # Get the current footprint
>>>> MEMORY_CURRENT          = (ImageSize/1024)
>>>> # Exceeds expectations?
>>>> MEMORY_EXCEEDED         = $(MEMORY_CURRENT) > $(MEMORY_ALLOWED)
>>>> # If exceeding, preempt
>>>> #[preset]PREEMPT        = False
>>>> PREEMPT                 = ($(PREEMPT)) || ($(MEMORY_EXCEEDED))
>>>> WANT_SUSPEND            = False
>>>> 
>>>> 
>>> This should all work.
> 
> And indeed it does, see below. No need to debug right away.
> 
>> For the "exclude parallel universe from preemption" part, I will now use
>> PREEMPT                 = ($(PREEMPT)) || ($(MEMORY_EXCEEDED) && (JobUniverse =!= 11))
> 
> I had no opportunity to test this part yet...
> 
>> (and I'm afraid "PREEMPT_VANILLA = False" was the cause for preemption not
>> happening to vanilla universe jobs... removed that one from the config now)
> 
> This one seems to have made the real difference.
> Now preemption *does* happen, as one of the users quickly learned:
> 
>> Hi Steffen,
>> 
> [...]
>> This is a plotting job that looks like it's trying to plot a vast
>> number of points .... and the job does seem to get kicked pretty
>> quickly. However, the memory requirement doesn't get above 15GB (even
>> though the reported condor size is bigger) and so it keeps matching,
>> evicting, matching, evicting.
>> 
>> 15GB is the original memory request of this job.
> 
> As the job was requesting more than the 15000 MB asked for in
> request_memory, at 17090 MB the additional margin of 2 GB was
> also reached, and the job was evicted from the slot it was running in.
> 
> Now I would have expected that the job would go back to the queue,
> with its updated ImageSize as new RequestMemory setting, but this
> apparently didn't happen.
> Well, it got re-scheduled, but kept its request size at 15000, 
> resulting in a never-ending loop, visiting more and more worker nodes.
> I have extracted the corresponding messages from the overall log file.
> 
> Condor version is 8.3.3 (this isn't supposed to behave different from 
> 8.2.7 in this particular respect, is it?)
> 
> Log file extract attached.
> 
> Any ideas how to help the user (except suggesting to specify 50000,
> and wait for a machine big enough to take this)?
> I'm afraid this is only the tip of a bigger iceberg...
> 
> Thanks,
> Steffen
> 
> -- 
> Steffen Grunewald * Cluster Admin * steffen.grunewald(*)aei.mpg.de
> MPI f. Gravitationsphysik (AEI) * Am Mühlenberg 1, D-14476 Potsdam
> http://www.aei.mpg.de/ * ------- * +49-331-567-{fon:7274,fax:7298}
> _______________________________________________
> Condorligo mailing list
> Condorligo@xxxxxxxxxx
> http://lists.aei.mpg.de/cgi-bin/mailman/listinfo/condorligo