[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Image processing with HTCondor
From: Sean Milligan <Sean.Milligan@xxxxxxxxxxxxx>
Date: 01/18/2016 11:41 AM
> We need to run these seven algorithms over each
image, these images are
> all in separate directories so Iâm trying to figure out how others
> approach this. I thought Iâd be able to use something like the flexible
> queue command to iterate over each image but my reading through the
> mailing list archive explained why this isnât support with DAGman.
> point the only thing Iâve figured is to write a script to create
> DAG files for each image and then either submit each DAG file individually
> or wrap all of the individual DAGs into a âmasterâ DAG as SUBDAGs.
> I guess Iâm ultimately asking for pointers or
what approaches have others
> used in situations like this?
You've got the right idea. At HTCondor Week 2015,
Miron described DAGman as
something akin to "assembly language," and
that's a good thing to keep in
mind when dealing with it.
My DAGman app has a little submission script I wrote
in Perl, which accepts
the name of the input file to be run and builds out
the DAG accordingly
through each of the processing steps, with a node
for splitting up the
input file, children for running the code against
each one, then a child
of all the run nodes for combining the output again,
and finally a node
for cleaning up the DAG files upon successful completion
of the run.
Either way, you'll need a script - or Pegasus - to
write out the DAG