[Condor-users] Only allow 40 of my runs to run at once

[Rob Stevenson <r.stevenson@xxxxxxxxxxxxxxxxxxx>]

Hi all,

One of our Condor users has a particular type of job which puts a lot of pressure on a database server. We've worked out that any more than 40 runs at once and the db server will fall over and all jobs fail completely.

 

So far I've got a partial solution of adding a ClassAd to 40 machine's config files. Eg:

 

>

STARTD_EXPRS = $(STARTD_EXPRS) , ONLY40

 

And the user adds a requirement ( True) to their submit file.

 

This works great while the same 40 CPUs remain in the pool and there's not much else going on. But if many other users jobs happen to be running on these 40 CPUs or I have to remove some of these from the pool, then these jobs will not be able to run on anywhere near 40 CPUs at once.

 

 

Of course, the ideal solution is to look at the code and optimise the sql queries - I'm sure there's scope for this.. But for now:

 

Does anyone have any ideas about how I might be able to modify the limit so we're limiting to ANY 40 CPUs rather than a SPECIFIC set of 40?

 

Many thanks!

 

Rob

 

---

HR Wallingford uses faxes and emails for confidential and legally privileged business communications. They do not of themselves create legal commitments. Disclosure to parties other than addressees requires our specific consent. We are not liable for unauthorised disclosures nor reliance upon them.
If you have received this message in error please advise us immediately and destroy all copies of it.

HR Wallingford Limited
Howbery Park, Wallingford, Oxfordshire, OX10 8BA, United Kingdom
Registered in England No. 02562099

---