[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Including data in a submit file

On Fri, 13 Dec 2013, Szabolcs Horvátth wrote:

I have a DAGMan job graph, but some of the later running jobs' submit settings should depend on the result of the first job. For example the first job generates some datastructure and based on that result just a limited number of the computers are able to run the other jobs.

If I could include the content of a text file into the submit files or read it into a variable in the submit files than the first job could simply write its requirements (or concurrency attributes) into this text file and that could be used automatically in all other jobs without the need to modify *all* submit files. I checked the documentation for such feature or solution but could not find any. Can it be done somehow? Or can you suggest an alternate solution that does not require
editing all submit files or submitting a new DAG?

If I'm correctly understanding what you want to do, it shouldn't be hard.
Remember that submit files are evaluated lazily, so the PRE script of a node can write the submit file for that node.

So if your first node writes out requirements.txt, the PRE script of your
later node can do something like this:

  cat submit_template requirements.txt > real_submit

You probably don't want to do this:

  cat requirements.txt >> real_submit

because that won't work right if the node get retried, etc.

There are a few more details, like removing real_submit if it already exists, but that's the basic idea. (To summarize, the PRE script can create the submit file however it wants to -- DAGMan doesn't actually read the submit file until it's time to submit the job.)

Kent Wenger