[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Avoiding combinatorial explosion in dependencies between spliced DAGS

On Thu, 30 Jul 2015, John N Calley wrote:

What I'd really like to do is to reach 'into' each sub-dag and insert dependencies between specific final nodes and specific initial nodes. I've considered hacking this solution together, but the ways of doing it that I can think of seem inelegant. I wonder if anyone has thoughts on how to do this kind of thing cleanly? To expand a bit, this comes up when I want to do Analysis A on samples 1-2000 and then I want to do Analysis B on the same samples. Analysis B for sample 1 depends on Analysis A for the same sample, but not on Analysis A for any other samples. It's a shame to require that Analysis A finish for all samples before I start Analysis B for any samples, but that is what I feel stuck with at the moment.

I thought that the following would work:

  * top.dag
  splice A A.dag
  splice B B.dag
  parent A+A1 child B+B1

  * A.dag
  job A1 ...

  * B.dag
  job B1...

Unfortunately, it doesn't. I think that the nodes from the splices are not fully incorporated into the top-level DAG at the time that the parent...child line is parsed.

At any rate, I've created a ticket for this request:


and I'll be coming up with a design document soon.