Re: [HTCondor-users] [CondorLIGO] Problem with parallel universe jobs and dynamic partitioning

[Steffen Grunewald <Steffen.Grunewald@xxxxxxxxxx>]

On Mon, Sep 15, 2014 at 10:39:36AM -0500, Todd L Miller wrote:
> >Why does it need a central defrag process to re-join the resource
> >fragments that remain after dynamic slots have become idle (for long
> >enough)? Shouldn't the local startd take care of these fragments?
> 
> 	As I understand it, we can't assume the dynamic slots become idle.

In that case, it should be ensured that parallel jobs may use already
existing dynamic slots if they match the requirements. Something that
apparently doesn't happen now.

> The goal of the defrag daemon is to minimize the pool-wide cost of
> forcing fragments to stay idle so they can be rejoined and not
> starve the larger jobs.

I understand that, but I don't understand why it has be done by a central
task, not a daemon running locally on the machine (and knowing what's
going on - if splitting off dynamic jobs from a partitionable slot does
work, I'd expect the opposite way, fusing the parts together again, should
also work).

> >If eviction is the universal solution to make resources available
> >to other (parallel) jobs, what if there are only parallel jobs
> >running? Evicting one thread of a multi-core parallel task has a
> >very high cost, and I can imagine a dead-lock scenario very easily.
> 
> 	According to our manual (section 2.9.2), we never preempt parallel
> jobs, probably for this reason.

Okay, so we've reached a catch-22 if there are only parallel jobs
on a pool?

> >I'm still somewhat unable to see where the DD addresses the specifics
> >of parallel jobs (as those seem to make the difference). Got to reread
> >the whole thing, at least twice...
> 
> 	I think the idea is that parallel jobs have a higher startd rank
> than serial ones.  Thus, a parallel job can boot a serial job off a
> (or some) dynamic slot(s) -- if startd rank worked with dynamic
> slots.

I've been trying to do this using NEGOTIATOR_PRE_JOB_RANK, but obviously
failed (at least in the ParU case).

We need existing dynamic slots to be matched against new parallel jobs,
or/and to be rejoined into their parent slot (to be split off again,
what a waste...). TBH I cannot see the difference of a dynamically 
created slot and a static one when it comes to matching (should there
be any?)...

- S