[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Image processing with HTCondor

From: Sean Milligan <Sean.Milligan@xxxxxxxxxxxxx>
Date: 01/18/2016 11:41 AM

> We need to run these seven algorithms over each image, these images are
> all in separate directories so Iâm trying to figure out how others
> approach this. I thought Iâd be able to use something like the flexible
> queue command to iterate over each image but my reading through the
> mailing list archive explained why this isnât support with DAGman. At this
> point the only thing Iâve figured is to write a script to create unique
> DAG files for each image and then either submit each DAG file individually
> or wrap all of the individual DAGs into a âmasterâ DAG as SUBDAGs.

> I guess Iâm ultimately asking for pointers or what approaches have others
> used in situations like this?

You've got the right idea. At HTCondor Week 2015, Miron described DAGman as
something akin to "assembly language," and that's a good thing to keep in
mind when dealing with it.

My DAGman app has a little submission script I wrote in Perl, which accepts
the name of the input file to be run and builds out the DAG accordingly
through each of the processing steps, with a node for splitting up the
input file, children for running the code against each one, then a child
of all the run nodes for combining the output again, and finally a node
for cleaning up the DAG files upon successful completion of the run.

Either way, you'll need a script - or Pegasus - to write out the DAG
"assembly language."

        -Michael Pelletier.