On 02/29/2016 03:42 PM, Mary Romelfanger wrote:
In the case where all the jobs are coming from the same user, you can use HTCondor's job priority to influence which idle job goes next when a machine becomes available. Note this is completely different than the similarly-named user priority. HTCondor wlil not preempt lower job-priority jobs in favor of better job-priority, so this seems like the mechanism that would best suit your requirements.
You can set the job priority globally in a dag via the dagman PRIORITY directive, or with the condor_submit_dag -prio command line option.
TotalSchedulerJobsRunning constrains the number of dags runnable themselves. Unfortunately, if you hit this limit, Condor does not keep the highest priority dags running. How many dags do you have running and idle? Probably the best approach is to raise the TotalSchedulerJobsRunning constraint, unless it is so high the the machine is swapping.