[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[HTCondor-users] Scratch space and input subdirectories

Hi folks,

So we've got a fairly simple setup - a PreCmd populates an input template file and places it into a numbered directory it creates named with the $(Process), and then the job runs against that template, the PostCmd compresses the various output files which may differ from run to run, and then HTCondor will transfer back the numbered directory and all its contents, so we end up with a directory named for the process ID number containing all the outputs for that run, across all the runs. By transferring the directory, we don't have to worry about figuring out which files will be in there based on what version or scenario we're using.

However we've got an executable that apparently doesn't have an option to specify a run directory on the command line, so it just works in whatever directory it's started in. This means that it needs to be in the numbered directory, not in the root of the scratch directory, when it starts under our current approach. And doing a "cd" in the PreCmd doesn't work, because that doesn't carry over to the executable, naturally.

Now I realize that we can just do a wrapper script to change into the directory created by the PreCmd and then launch the run, but I'd really prefer to strip off the layers of wrappers and have the tool itself be what's run by the starter, rather than a wrapper which runs the tool. I'd also prefer to have the final numbered output directory not exist on the submitter's output path until the run is complete, in an effort to keep things tidy and self-contained.

If we run it as-is without the numbered subdirectory, HTCondor can transfer all output files it finds in the scratch root without having to enumerate them, but can we get those into a subdirectory on the submitter which won't exist before the job is complete?

Is there some trick for this I'm missing, or do I want to have my cake and eat it too and should just go with a simple cd/exec wrapper?


Michael V. Pelletier
IT Program Execution
Principal Engineer
978.858.9681 (5-9681)
339.293.9149 cell