Re: [Condor-users] Dagman with a variable number of jobs

["R. Kent Wenger" <wenger@xxxxxxxxxxx>]

On Tue, 1 Jun 2010, Robert Sandilands wrote:

> I am just fishing for some ideas on how to better handle the following 
> scenario:
>
> Job 1 collects a list of data objects to process from a database. This list 
> can be of variable size and this size is unknown until the first job 
> finishes.
>
> Job 2 .. N then processes the data objects in batches of up to 1,000 items 
> per job and updates the database.
>
> As additional fun this needs to run once a day or on a continuous basis 
> depending on the specific data objects.
>
> My current attempt to solve this involves running a script that submits a dag 
> (right terminology?) and waits for it to finish. It then will sleep for the 
> required amount of time and resubmit the dag. Inside the .dag file I use PRE 
> scripts to determine which individual jobs needs to be submitted and which 
> not.
>
> This works fine if there is a reasonable upper limit to the number of data 
> objects. The number of items in the list can be anything from 1,000 to 
> 1,000,000.
>
> If we assume that no single job should process more than 1,000 items then it 
> implies that there can be between 2 and 1,001 jobs in the dag.
>
> Is it even possible to write a dag with that number of dependencies 
> especially since there is only 1 parent? I have tested up to 51 jobs and that 
> seems to run without any issues.
>
> And what do you do if the list suddenly grows to have 1,000,001 entries?
>
> Any ideas would be appreciated.

We have users running DAGs with up to around 500k nodes, so having 1000
children of one parent should be no problem.

I'd recommend slightly changing your approach, though -- I think this is a 
great case for using nested DAGs.  If you take this approach, your 
top-level DAG would have two nodes -- one is the node that figures out how 
much processing has to be done, and writes the lower-level DAG that does 
the processing, and one is the node that actually runs the lower-level 
DAG.

You can find more information about nested DAGs in the DAGMan section of 
the Condor manual:

http://www.cs.wisc.edu/condor/manual/v7.5/2_10DAGMan_Applications.html#SECTION003106700000000000000

Also, this will be easier to implement if you are running the latest 
DAGMan (7.5.2).  If you're running an older DAGMan, you have to have a 
"dummy" lower-level DAG in place when you submit the upper-level DAG, etc.

If you're not running Condor 7.5.2, you can run the latest DAGMan without 
upgrading your overall Condor version -- just grab the 7.5.2 condor_dagman 
and condor_submit_dag binaries, and put them somewhere that's closer to 
the beginning of your $path list than the "regular" Condor binaries -- 
maybe ~/bin, for example.

Kent Wenger
Condor Team