[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] out-of-memory issues in parallel universe
- Date: Wed, 19 Mar 2008 12:44:23 -0500
- From: Dan Bradley <dan@xxxxxxxxxxxx>
- Subject: Re: [Condor-users] out-of-memory issues in parallel universe
Robert E. Parrott wrote:
I'm also/instead looking for a solution to enforce memory limits at
It looks as if a USER_JOB_WRAPPER with a ulimit line is the solution
here. Does that jibe with what others have done?
That is one option. Here are two others:
1. Have Condor preempt jobs from the machine when their virtual image
size exceeds some amount. Example:
MEMORY_EXCEEDED = ( ImageSize > 1.5*Memory*1024 )
MEMORY_NOT_EXCEEDED = ($(MEMORY_EXCEEDED) =!= TRUE)
WANT_SUSPEND = ($(WANT_SUSPEND)) && $(MEMORY_NOT_EXCEEDED)
PREEMPT = ($(PREEMPT)) && $(MEMORY_EXCEEDED)
2. Have Condor (on the submit side) put jobs on hold when their virtual
image size exceeds some amount. It is a little more awkward to set the
amount based on the size of the machine's memory in this case, but it is
# When a job matches, insert the machine memory into the
# job ClassAd so periodic_remove can refer to it.
MachineMemory = "$$(Memory)"
SUBMIT_EXPRS = $(SUBMIT_EXPRS) MachineMemory
SYSTEM_PERIODIC_HOLD = (MATCH_EXP_MachineMemory =!= UNDEFINED &&
ImageSize > 1.5*int(MATCH_EXP_MachineMemory))
Both of these techniques suffer from the shortcoming that they are based
off of the virtual memory size of the job, which may not be an accurate
measure of the job's actual demand on physical memory.