[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Condor-users] DAG pipelines running on machines they started from?


I'm running an experiment which has DAG nodes that look roughly like this:

  |--> E1 -> F1 -> G1 -> H1
S -|--> E2 -> F2 -> G2 -> H2
  |--> E3 -> F3 -> G3 -> H3
  |--> E4 -> F4 -> G4 -> H4

Here, I have a parent submit file that causes several DAGs to be spawned simultaneously (from state S). Is there a way I can enforce each (or part of) one of these pipelines to run on one machine (i.e. the machine it started running on)? Of course, I could lump E1, F1, G1, and H1 together into a shell script. However, I dynamically generate my Condor job/DAG configurations, and I think it would be easier and neater from a configuration standpoint to keep the complexity in my Condor job/DAG configuration rather than an external script, because I'd like to pass arbitrary arguments to each of E1, F1, G1, and H1. Plus, I could use other nice Condor features like retrying nodes that fail (at least, nodes at a lower level).

Any ideas?  Thanks!

 - Armen

Armen Babikyan
MIT Lincoln Laboratory
armenb@xxxxxxxxxx . 781-981-1796