[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] Partitionable Slot Starvation
- Date: Thu, 16 Aug 2012 17:28:41 -0500
- From: Dan Bradley <dan@xxxxxxxxxxxx>
- Subject: Re: [Condor-users] Partitionable Slot Starvation
On 8/16/12 2:46 PM, Dan Bradley wrote:
On 8/16/12 12:51 PM, William Strecker-Kellogg wrote:
On 08/16/2012 10:42 AM, Dan Bradley wrote:
If the problem was caused by DEFRAG_REQUIREMENTS and/or
DEFRAG_WHOLE_MACHINE_EXPR, the defrag log would indicate so with a
message like the following:
"Drained 0 machines (wanted to drain X machines)."
"Doing nothing, because DEFRAG_MAX_WHOLE_MACHINES=X and there are Y
Right, I'm not seeing that message.
As a sanity check, what numbers do you see in the following line in the
log when defrag starts up or is reconfigured?
"polling interval %ds, DEFRAG_DRAINING_MACHINES_PER_HOUR = %f/hour =
%d/interval + %d/hour + %d/day"
08/15/12 15:07:13 polling interval 90s,
DEFRAG_DRAINING_MACHINES_PER_HOUR = 12.000000/hour = 0/interval +
12/hour + 0
Based on that, I would expect defrag to attempt to drain 12 machines
every hour. You should see a message in the logs something like this:
"Looking for 12 machines to drain."
I just found a bug in this code :(
The workaround is to set DEFRAG_INTERVAL large enough to avoid the
hourly and daily draining rate corrections. If you set it back to the
default 600 seconds, I would expect the scheduling to work correctly.
Sorry about that!
Here's the bug ticket:
If your DEFRAG_INTERVAL were larger so that
DEFRAG_DRAINING_MACHINES_PER_HOUR*DEFRAG_INTERVAL/3600 >= 1, then the
draining attempts would be distributed more evenly throughout the hour
rather than all landing at the beginning of the hour. For example,
with the default DEFRAG_INTERVAL=600, it should attempt to drain 2
machines every 10 minutes.
Anyway, if it is never attempting to drain, then that is unexpected.
If it _is_ attempting to drain, then perhaps the attempt is being
rejected for some reason. The log should contain an error message in
this case. Do you see anything?
And what numbers do you see in the most recent log line of the
"There are currently %d draining and %d whole machines."
08/16/12 12:09:31 There are currently 0 draining and 0 whole machines.
One word of warning: defrag drains the whole startd, partitionable
and static slots alike. If you only want it to drain some slots and
others, you need to run multiple startds and set DEFRAG_REQUIREMENTS to
only match the slots of the startd to be drained and not the slots of
the other startd.
OK, so do I infer that the defrag will only work on machines where there
is only one whole-machine slot? Or just that it will drain single-core
slots in addition to the partitionable ones?
The latter. When it chooses to drain a machine, it drains all slots
on the machine, because that is all that is supported by the current
draining operation in Condor.
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
You can also unsubscribe by visiting
The archives can be found at: