Re: [Condor-users] Partitionable Slot Starvation

[Dan Bradley <dan@xxxxxxxxxxxx>]

On 8/16/12 12:51 PM, William Strecker-Kellogg wrote:
> Hi Dan,
>
> On 08/16/2012 10:42 AM, Dan Bradley wrote:
> > If the problem was caused by DEFRAG_REQUIREMENTS and/or
> > DEFRAG_WHOLE_MACHINE_EXPR, the defrag log would indicate so with a
> > message like the following:
> >
> > "Drained 0 machines (wanted to drain X machines)."
> >
> > "Doing nothing, because DEFRAG_MAX_WHOLE_MACHINES=X and there are Y
> > whole machines."
> Right, I'm not seeing that message.
>
> > As a sanity check, what numbers do you see in the following line in the
> > log when defrag starts up or is reconfigured?
> >
> > "polling interval %ds, DEFRAG_DRAINING_MACHINES_PER_HOUR = %f/hour =
> > %d/interval + %d/hour + %d/day"
>
> 08/15/12 15:07:13 polling interval 90s,
> DEFRAG_DRAINING_MACHINES_PER_HOUR = 12.000000/hour = 0/interval +
> 12/hour + 0

Based on that, I would expect defrag to attempt to drain 12 machines 
every hour.  You should see a message in the logs something like this:

"Looking for 12 machines to drain."

If your DEFRAG_INTERVAL were larger so that 
DEFRAG_DRAINING_MACHINES_PER_HOUR*DEFRAG_INTERVAL/3600 >= 1, then the 
draining attempts would be distributed more evenly throughout the hour 
rather than all landing at the beginning of the hour.  For example, with 
the default DEFRAG_INTERVAL=600, it should attempt to drain 2 machines 
every 10 minutes.

Anyway, if it is never attempting to drain, then that is unexpected.  If 
it _is_ attempting to drain, then perhaps the attempt is being rejected 
for some reason.  The log should contain an error message in this case.  
Do you see anything?

> > And what numbers do you see in the most recent log line of the following
> > form:
> >
> > "There are currently %d draining and %d whole machines."
> 08/16/12 12:09:31 There are currently 0 draining and 0 whole machines.
>
>
> > One word of warning: defrag drains the whole startd, partitionable slots
> > and static slots alike.  If you only want it to drain some slots and not
> > others, you need to run multiple startds and set DEFRAG_REQUIREMENTS to
> > only match the slots of the startd to be drained and not the slots of
> > the other startd.
> OK, so do I infer that the defrag will only work on machines where there
> is only one whole-machine slot? Or just that it will drain single-core
> slots in addition to the partitionable ones?

The latter.  When it chooses to drain a machine, it drains all slots on 
the machine, because that is all that is supported by the current 
draining operation in Condor.

--Dan