[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[HTCondor-users] Question about new foreach condor_submit syntax and dagman



Hello,

I want to submit 200 jobs, each to process a different file. I also
want to throttle their execution so that I can set the maximum number
of running jobs at any given time. So far, pretty much a bread and
butter use case for every day data processing business.

Until now, I have been doing this with dagman, which provides the
maxjobs throttling functionality.

I learned in the htcondor week about the new features of
condor_submit, in particular the one that allows me to submit my 200
jobs with one submit file using the syntax like:

Queue <vars> from <filename>

Nice. I would like to use the new condor_submit feature, but then
still using dagman to throttle the maxjobs. How should I do this? Now
I have one submit file that will generate 200 jobs, so I tried a dummy
dagman file like:

JOB Job1 my200jobs.submit

this way I see my 200 jobs get submitted, but dagman does not seem to
apply the maxjobs constraints to them.

What is the way to get dagman maxjobs throttling to work with the new
"queue from" syntax in condor_submit?

thanks a lot,
Gonzalo