[HTCondor-users] Doubt regarding job priorities on distributed scheduling

[Sud <sudhamsh.kukkadapu2@xxxxxxxxx>]

I am interested in knowing more about the job priorities, So let me first explain the architectureÂwe have in the current setup for HTCondor in brief.

We have one collector, one negotiator on one same machine.

We have multiple submit nodes and execute nodes (i.e. we use distributed scheduling to reduce load on schedulers).ÂAnd my negotiator cycle frequency is kept at 60sec.

Now consider this scenario:

Consider 30 jobs that I submit will occupy entire grid (for simplicity let's assume I am the only user using as well)

Step 1:ÂLet's say I submitted 100 jobs using SchedulerÂA, and 30 jobs started running (whose runtime is lets say 10 mins). (Here I gave 0 as job prioritiesÂfor all jobs, these jobs are independent and running as a DAG submission but as these nodes are not connected hence they all run in parallel without any dependency)

Step 2:ÂNow I submit lets say 100 new jobs from Scheduler B with jobÂpriorities as 10 for all and having sameÂruntime and memory/cpu requirements as Step 1 jobs. Basically you can think that I am submitting the sameÂjobs again with higher job priority on a different scheduler. Also at this point 30 jobs from Scheduler A are running and remaining 70 are in Idle state as the grid is full.

So now I want my Step 2 jobs to be running once the Step 1 first set of jobs (i.e. 30 jobs are done). i.e. the execution flow I want is this:

Step 1, 30 jobs (0 priority, as they are already running before Step 2 submission and I don't want to preempt these jobs) â Step 2, 100 jobs (10 priority) â Step 1, remaining 70 jobs

I have read that job priorities are constrained per scheduler, So If I have distributed scheduling how can I ensure that the job priorities are still obeyed at user level? i.e. Is there any way to control the queue that the negotiator maintains (or) any way to convey the negotiator to pick jobs from the Scheduler which has higher priority jobs with any macro?