[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] out-of-memory issues in parallel universe

Robert E. Parrott wrote:

I'm also/instead looking for a solution to enforce memory limits at runtime.

It looks as if a USER_JOB_WRAPPER with a ulimit line is the solution here. Does that jibe with what others have done?

That is one option.  Here are two others:

1. Have Condor preempt jobs from the machine when their virtual image size exceeds some amount. Example:

MEMORY_EXCEEDED = ( ImageSize > 1.5*Memory*1024 )


2. Have Condor (on the submit side) put jobs on hold when their virtual image size exceeds some amount. It is a little more awkward to set the amount based on the size of the machine's memory in this case, but it is possible. Example:

# When a job matches, insert the machine memory into the
# job ClassAd so periodic_remove can refer to it.
MachineMemory = "$$(Memory)"

SYSTEM_PERIODIC_HOLD = (MATCH_EXP_MachineMemory =!= UNDEFINED && ImageSize > 1.5*int(MATCH_EXP_MachineMemory))

Both of these techniques suffer from the shortcoming that they are based off of the virtual memory size of the job, which may not be an accurate measure of the job's actual demand on physical memory.