[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Partitionable Slot Starvation

On 8/16/12 12:51 PM, William Strecker-Kellogg wrote:
Hi Dan,

On 08/16/2012 10:42 AM, Dan Bradley wrote:
If the problem was caused by DEFRAG_REQUIREMENTS and/or
DEFRAG_WHOLE_MACHINE_EXPR, the defrag log would indicate so with a
message like the following:

"Drained 0 machines (wanted to drain X machines)."

"Doing nothing, because DEFRAG_MAX_WHOLE_MACHINES=X and there are Y
whole machines."
Right, I'm not seeing that message.

As a sanity check, what numbers do you see in the following line in the
log when defrag starts up or is reconfigured?

"polling interval %ds, DEFRAG_DRAINING_MACHINES_PER_HOUR = %f/hour =
%d/interval + %d/hour + %d/day"

08/15/12 15:07:13 polling interval 90s,
DEFRAG_DRAINING_MACHINES_PER_HOUR = 12.000000/hour = 0/interval +
12/hour + 0

Based on that, I would expect defrag to attempt to drain 12 machines every hour. You should see a message in the logs something like this:

"Looking for 12 machines to drain."

If your DEFRAG_INTERVAL were larger so that DEFRAG_DRAINING_MACHINES_PER_HOUR*DEFRAG_INTERVAL/3600 >= 1, then the draining attempts would be distributed more evenly throughout the hour rather than all landing at the beginning of the hour. For example, with the default DEFRAG_INTERVAL=600, it should attempt to drain 2 machines every 10 minutes.

Anyway, if it is never attempting to drain, then that is unexpected. If it _is_ attempting to drain, then perhaps the attempt is being rejected for some reason. The log should contain an error message in this case. Do you see anything?

And what numbers do you see in the most recent log line of the following

"There are currently %d draining and %d whole machines."

08/16/12 12:09:31 There are currently 0 draining and 0 whole machines.

One word of warning: defrag drains the whole startd, partitionable slots
and static slots alike.  If you only want it to drain some slots and not
others, you need to run multiple startds and set DEFRAG_REQUIREMENTS to
only match the slots of the startd to be drained and not the slots of
the other startd.
OK, so do I infer that the defrag will only work on machines where there
is only one whole-machine slot? Or just that it will drain single-core
slots in addition to the partitionable ones?

The latter. When it chooses to drain a machine, it drains all slots on the machine, because that is all that is supported by the current draining operation in Condor.