[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Documentation of DAG priorities

On 05/10/2015 22:37, Kent WENGER wrote:
> But until that point, I couldn't find any indication that the DAG node priority was in any way related to the job priority.

Well, there is this: "Priorities are propagated to children, to SUBDAGs, and to the HTCondor job itself, via the JobPrio attribute in the job's ClassAd."
Ah, now I have it. Thank you :-)

> > Where this becomes important is where I've made a DAG with explicit node priorities - influencing > > the order in which things are run to optimise throughput - but then I decide that I want one DAG > > to take priority over another. It seems now that I have to apply offsets to all the DAG node priorities.
> Well, at this point if you want to change things once the DAGs have been submitted, there's not really a good way to do it.

In our case we are submitting a new DAG: it's a "rush job" which needs to take priority over the existing work, without pre-emption. So we just boost the priority of nodes in the new DAG at the time it's created/submitted. We don't (currently) have a need to update the priority of an already-running DAG.

Now, one thing I find odd is how dagman propagates priorities using the PARENT/CHILD relationship between nodes. Why is it expected that if one particular node of a DAG is run at a high priority, all its children must run at least at that priority too? Imagine I have a DAG which contains a number of subgraphs like this:


If I have a preference that all the 'A' jobs run before the 'B' jobs, and I set a higher priority on the 'A' jobs, then I have no way to avoid the high priority being inherited by the 'C' jobs. But I might not want ready 'C' jobs to be submitted in preference to the 'B' jobs in other subgraphs. It seems pretty inflexible, but no doubt there is a good historical use case for it.

Also: where a DAG is included within another DAG, I think it would make sense to *add* the priority of the enclosing subdag node to the nodes of the subdag.

As a real example: suppose I have a DAG with node priorities 1,2,3,4. If I submit it with "condor_submit_dag -priority 100 foo.dag" then all the nodes end up with the same priority (100). What I wanted was 101,102,103,104. In our case we had to change the script which writes out the DAG to take a priority base value.