[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] dynamic dagman



On Mon, 29 Mar 2010, Jason Reilly wrote:

I have a job that has 2 parts. The output of the first job returns a zip file with N entries. For the second job, I would like to dynamically create a dagman based on the number of entries in the zip file generated by the output of the first job. Is there such a thing as a dynamic dag that can handle this kind of task, where I don't need to programmatically generate the Condor submit file?

If I'm understanding things correctly, you should be able to do what you want with a little trickery. As of 7.5.2, the trickery will no longer be
necessary.

I'm assuming that you want to do the whole thing in a DAG. Because of the dynamic aspect, you'll have to use nested DAGs.

* Top-level DAG:
Job A getzip.sub
Subdag external B unzip.dag
Script Pre B generate_dag
Parent A child B

The idea is that the generate_dag script would look at the zip file and write the "unzip.dag" file.

The trickery is this: In the current versions of DAGMan, a valid unzip.dag file must exist at the time when you submit the top-level DAG. You can overwrite the unzip.dag file, but it must exist. In current versions of DAGMan (since 7.3-something, I think), condor_submit_dag will automatically run itself on the lower-level DAG file (with the -no_submit option) to generate the .condor.sub file.

In 7.5.2, by default, the running of condor_submit_dag -no_submit will
be done by DAGMan itself, just before the node job will be submitted,
so you won't have to have a "dummy" unzip.dag file in place when you
submit the top-level DAG.

Let me know if this isn't clear...

Kent Wenger
Condor Team