Ahh...it's fun to dream. In all honesty though, while it's a bit cumbersome to setup once, Startd cron jobs can handle a lot of what we're dreaming about here when it comes to slot-level concurrency controls. And once you get one set up, the rest follow pretty easily.
I haven't but only because I haven't found a problem that would be solved well with the use crond's for the reasons you point. The cron nature means you can only use to solve things were absolutely correctness isn't a requirement. Where you can handle a fuzzy area when things might be oversubscribed as the crons catch up to reality.
For instance, say I have a SAN-attached host and want to enable no more than 3 concurrent SAN IO jobs while also enabling other job types. Today I'd have to set up a special partitionable slot with a SAN attribute and start _expression_ to only allow SAN jobs and do something like dedicating 3 CPU's and some amount of RAM towards that partitionable slot. Or make 3 SAN slots with dedicated memory resources. And then add a partitionable slot for all remaining CPU, memory, and local disk resourcse. There's no way to apply a "SAN" resource counter, and no way to alter the number "3" live based on actual SAN link utilization or storage subsystem latency. Right?
Right.
I'd generally approach a problem like this by setting a few slots to prefer my special job. So if it was "I can only run 3 SAN jobs per machine" I'd have the first three jobs set to preempt non-SAN jobs in favour of SAN jobs when SAN jobs appear in the queue. Depending on the rest of my workloads I might just set it so SAN jobs can only preempt if non-SAN jobs have been running for only a small amount of time. That way there's less SAN job starvation and less non-SAN job interruption. Not ideal, but it gets the job done and it maintains absolute correctness when it comes to enforcing the "no more than 3 SAN jobs per machine" rule.
Honestly: I can't see how being able to reference an a third entity classad would help the SAN case. You can always have a race condition updating the ClassAd that'll mean you could do the wrong thing enforcing rules in the system. Taking your SAN example: if I had a SAN type ad I could reference that had a counter attribute that listed total connections, and I wanted to enforce no more than N connections; I could have two jobs start, see TotalConnections < N, both decide it's okay to run, but once they were in the running state they could cause TotalConnections to end up at N+1 because access to querying and setting TotalConnections can't be wrapped in a mutex or transaction block like you can do with SQL-based data stores.
It's a tough problem to solve. It's tempting to say "use Job Hooks" but then you lose all the other wonderful semantics of matchmaking that Condor offers and I'm not sure the benefits outweigh the costs. In that case.
But maybe I'm not seeing the solution?
If you have a startd cron for that class of use case, we'd be interested in seeing it. Other competing resource schedulers have host-based resource counters for this reason.
Interesting -- I'm not overly familiar with how other systems do this sort of thing? Can you elaborate a bit? I'd love to learn about it.
Regards,
- Ian
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif){kind=logo}