Re: [Condor-users] PASSWD_CACHE_REFRESH in 6.9.4?

[Dan Bradley <dan@xxxxxxxxxxxx>]

Ian Chesal wrote:
> > This seems counter intuitive to me. Why would _not_ nice'ing 
> > the shadow
> > processes on a busy submit machine be a good thing?
> >     
>
> Ditto. Is this a Windows scheduler only thing? I'm almost certain Alan
> De Smet's talk every year at Condor Week talks about using higher nice
> levels on the shadows to help out a starved-for-CPU schedd process.
>   

If you want to increase the priority of the schedd, that is possibly a 
good idea.  However, using SHADOW_RENICE_INCREMENT=10 to decrease the 
priority of the shadows below all other normal processes on the system 
degrades throughput in every case we have observed or tested in the 6.9 
branch.  Part of the problem is that the schedd and the shadow need to 
communicate.  During this communication, it is actually possible for the 
schedd to be slowed down because it is stuck waiting for a response from 
a low priority shadow.  More common is to see connection failures in the 
shadow logs due to the shadow being so cpu starved that it cannot form a 
connection to the schedd, even with very generous timeouts.

Another thing that has changed is that the 6.9.4 schedd is much less cpu 
hungry than 6.8.  Having 10s of thousands of jobs in the queue and a few 
thousand jobs running should not severely tax the 6.9.4 schedd on 
reasonable server-class hardware unless the jobs are so fast that the 
completion rate is greater than ~10-15 jobs per second.

I'll admit that our tests of this have all been under linux and have 
been focussed on vanilla universe.  We're certainly hoping for feedback 
on all the other possible usage cases.

Cheers,
--Dan