[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] out-of-memory issues in parallel universe

Thanks for the pointers.

The point you make about virtual memory size is relevant. On quad core, 4 GB machines (so 1 GB per "slot"), I'm seeing jobs with virtual memory reported to be 1.5 GB, but resident size of 850 MB.

As such, using the virtual memory value would make a significant impact on usability.

Too bad that the kernel OOM killer isn't smarter.


On Mar 19, 2008, at 1:44 PM, Dan Bradley wrote:

Robert E. Parrott wrote:

I'm also/instead looking for a solution to enforce memory limits at

It looks as if a USER_JOB_WRAPPER with a ulimit line is the solution
here. Does that jibe with what others have done?

That is one option.  Here are two others:

1. Have Condor preempt jobs from the machine when their virtual image
size exceeds some amount.  Example:

MEMORY_EXCEEDED = ( ImageSize > 1.5*Memory*1024 )


2. Have Condor (on the submit side) put jobs on hold when their virtual image size exceeds some amount. It is a little more awkward to set the amount based on the size of the machine's memory in this case, but it is
possible.  Example:

# When a job matches, insert the machine memory into the
# job ClassAd so periodic_remove can refer to it.
MachineMemory = "$$(Memory)"

ImageSize > 1.5*int(MATCH_EXP_MachineMemory))

Both of these techniques suffer from the shortcoming that they are based off of the virtual memory size of the job, which may not be an accurate
measure of the job's actual demand on physical memory.


Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at:

Robert E. Parrott, Ph.D. (Phys. '06)
Associate Director, Grid and
       Supercomputing Platforms
Project Manager, CrimsonGrid Initiative
Harvard University Sch. of Eng. and App. Sci.
Maxwell-Dworkin  211,
33 Oxford St.
Cambridge, MA 02138