Brian Candler wrote;
> I would like to raise something which is unclear in the documentation, and that's around the idea of "priority".
> At first glance it seems that there are two distinct and unrelated priorities.
> (1) The job priority, which is set in the submit file with "priority = XXX" or using the condor_prio command line tool, and is reflected in the JobPrio ClassAd attribute.
> (2) The DAG node priority, which is set using the "PRIORITY <node> <value>" statement in the DAG file.
> Now, the documentation doesn't say at this point if the DAG node priority is related to the job priority. However, testing shows that it is (see end of message). What appears to happen is that the DAG node priority is also being used as the job priority. Not only that, but the DAG node priority *overrides* any job priority set explicitly in the submit file.
> On reading the documentation again, there is a hint of this behaviour where it says later on:
> "There is no way to change the priority in the submit description file for a job, as DAGMan will override any priority command placed in a submit description file."
> But until that point, I couldn't find any indication that the DAG node priority was in any way related to the job priority.
Well, there is this: "Priorities are propagated to children, to SUBDAGs, and to the HTCondor job itself, via the JobPrio attribute in the job's ClassAd."
At any rate, though, I'll change the manual to make this clearer. I think that part of the problem is that the DAG node priorities originally *were* separate from the HTCondor job priorities, and then when that got changed maybe the manual didn't get updated as clearly as it should have.
> Where this becomes important is where I've made a DAG with explicit node priorities - influencing the order in which things are run to optimise throughput - but then I decide that I want one DAG to take priority over another. It seems now that I have to apply offsets to all the DAG node priorities.
Well, at this point if you want to change things once the DAGs have been submitted, there's not really a good way to do it. Doing a condor_prio on a DAGMan job won't affect the priorities of the node jobs; and if you do condor_prio on the node jobs, that only affects jobs that are already in the queue; newly-submitted jobs will have their original priority.
We've thought about having condor_prio on the DAGMan job affect the priorities of the node jobs (there's a ticket for this: https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=3389) but before we do that we also want to simplify how priorities are propagated between nodes: https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=4024.
> I don't really have a problem with this behaviour - indeed it would probably be confusing to have two distinct priorities - but I think it could be made a lot clearer in the documentation for condor_dagman. The following would do the trick:
> "The PRIORITY keyword assigns a priority to a DAG node. The syntax for PRIORITY is
> PRIORITY JobName PriorityValue
> The node priority affects the order in which nodes that are ready at the same time will be submitted, and it also sets the job's priority in the queue, i.e. its JobPrio attribute. Note that node priority does not override the DAG dependencies."
I'll add something like this to the wording in the manual.