[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: [Condor-users] When do machine RANK settings apply?



Thanks for all your help! A few more questions below.

> Answers inline
> 
> On Tue, 4 Jan 2005 21:45:11 -0500, Ian Chesal 
> <ICHESAL@xxxxxxxxxx> wrote:
> > Thinking more about this now. See below.
> > 
> > > I'm not answering my own question here. I'm adding to it. 
> See below...
> > >
> > > > Do machine RANK settings apply before or after resources are 
> > > > partitioned based on effective priority? I'm not certain I
> > > completely
> > > > understand a negotation cycle in condor.
> 
> Me either though it is safe to say that it is opaque...
> 
> > > > If the RANK on the machine was:
> > > >
> > > > RANK = TARGET.JobPrio * 1000
> > > >
> > > > Then should the machine not go to the job with the highest job 
> > > > priority value? I'm seeing it go to User A's job.
> 
> I believe the negotiator deals first with user priority 
> considerations then with machine considerations. Since 
> getting both optimal with one taking priority (machine rank) 
> is going to be NP so I'm not surprised it needs some tweaking 
> to try to het it to achieve the desired effect.
> 
> > > Sitting and thinking some more. If I have 100 vm's in my 
> system. And 
> > > 100 users, each with infinitly long queues of jobs. Over 
> time will 
> > > each user's EUP move to 1? Will each user end up using exactly 1 
> > > machine in the system as steady-state is reached? Have I 
> understood 
> > > this correctly?
> 
> That's the basic idea...
> 
> > > If I artificially reset all RUPs to 0.50 every minute (basically 
> > > cron a call to "condor_userprio -resetall") -- would this 
> get me to 
> > > the priority-based queueing utopia I desire? If everyone's EUP is 
> > > always as low as it can be, so all resources are equally 
> likely to 
> > > be assigned to each user, will RANK statements not take over to 
> > > decide which jobs take up to a free machine? Am I way off 
> here with 
> > > my understanding of how this works?
> 
> Correct idea, but a more pleasant solution is to alter the 
> PRIORITY_HALFLIFE setting to be 1 (not sure if a setting 0 
> will play nicely 1 sec is sufficiently close).
> 
> This will ensure all *users* priority remains at whatever you 
> set it to (leave everyone at 0.5). This will then let the 
> negotiator have a simple job and seems to force machine rank 
> to the fore
>  
> > A little more thought a re-read of section 3.5 of the 
> manual and now 
> > I'm wondering if I could eliminate preemption based on user 
> priorities 
> > by simply changing the PREEMPTION_REQUIREMENTS. Currently I have:
> 
> > PREEMPTION_REQUIREMENTS = (CurrentTime - 
> EnteredCurrentState) > (1 * 
> > (60
> > * 60)) && RemoteUserPrio > SubmittorPrio * 1.2
> 
> Make it false... it's the only way to make the thing remain 
> stable - any thing not false seems to periodically get into a 
> thrashing state if more than 3 users have jobs. (this is on a 
> farm with >100 execute
> nodes)
> 
> > We run vanilla jobs. Would losing this portion of the 
> > PREEMPTION_REQUIREMENTS setting ensure that preemption would never 
> > occur because of condor-calculated priorities?
> 
> Yes - though machine based rank will *always* cause 
> preemption The retirementtime will however be respected in these cases
> 
> >I'm looking
> > back over the mailing list archives and based on a thread 
> started by  
> >Matthew Hope I think this might be the answer to my trouble. 
> For the  
> >record I am trying to make Condor behave in a priority-like queuing  
> >manner.
> 
> Whilst I don't (ab)use the job +-20 priorities and instead 
> use the 'Tier' approach discussed previously onthe list there 
> are some advantages to your approach.
> 
> If a user has two jobs A and B A being higher 'Tier' than B 
> but B has a high job priority then B will run first - 
> confusing :) (and I guess this answers your question over 
> when machine based Ranks apply).
> Reusing the job prio will prevent this from happening
> 
> In summary:
> 
> So long as PREEMPT and PREEMPTION_REQUIREMENTS both evaluate 
> to False then no preemption will occur except based on 
> machine rank or a user altering the job priorities (oh or 
> someone running condor_vacate :)

So PREEMPTION_REQUIREMENTS only comes into play when condor wants to
preempt for user priority reasons? 

> To make this setup function properly it appears a requirement 
> (at least on 6.6 series without the luxury of retirement) 
> that the user priorities are maintained at equal, or close to 
> equal levels - thus effectively disabling this feature.

My experiment last night consisted of setting:

PREEMPTION_REQUIREMENTS = (CurrentTime - EnteredCurrentState) > (1 * (60
* 60)) && MY.RANK < TARGET.RANK
RANK = (TARGET.JobPrio * 2880)

This got close the desired behaviour but lower priority users were still
preempting higher priority users on occasion. And users new to the
system were getting more resources than users with long-time running
jobs if they were all the same (highest) priority level.

I'm now experiments with your suggestion of:

PREEMPTION_REQUIREMENTS = False
PRIORITY_HALFLIFE = 1
RANK = (TARGET.JobPrio * 2880)

With our very long retirement time (enough for our jobs to finish
normally) this should be okay. I'll let you know how it works out.

Matthew, thanks for all the help -- it was your archive post that led me
in the right direction!

Cheers!

- Ian