[Condor-users] Changing a working directory for a DAG within a DAG?

[Masakatsu Ito <m-ito@xxxxxxxxxxxxxx>]

Hi, all.

I am trying to use a DAG within a DAG to perform
a number of simulations which is easier to construct
hierarchically. I have to put these simulation outputs
into hierarchical directory structure, which a post (analysis)
script requires. But the problem is that I've failed to put
them into the directory structure, because I cannot get
condor_dagman to change a working directory.

Basically I've tried to follow the way in the thread
https://lists.cs.wisc.edu/archive/condor-users/2006-March/msg00004.shtml
which suggested using -UseDagDir . But this thread
is not completely clear for me to follow.


My goal is to sort the simulation outputs into the directories
param1, param2, ... . The whole directory tree should be
like :

--+-- outer.dag
|
+-- param1 -+-- inner.dag
| +-- preparation.submit
| +-- simulation.submit
| +-- preparation.inp
| +-- ( simulation.out )
|
+-- param2 -+-- inner.dag
| +-- preparation.submit
| +-- simulation.submit
| +-- preparation.inp
| +-- ( simulation.out )
;

I used two nested DAGs. The inner DAG describes the
dependency between preparation and simulation programs.

# inner.dag
#
Job Pre preparation.submit
Job Sim simulation.submit

PARENT Pre CHILD Sim

I wrote the outer DAG description file to sort
the simulation results into the directories.

# outer.dag
#
Job Param1 param1/inner.dag.condor.sub
Job Param2 param2/inner.dag.condor.sub

I created these "inner.dag.condor.sub" using :

% condor_submit_dag -no_submit -UseDagDir param1/inner.dag

as the thread
https://lists.cs.wisc.edu/archive/condor-users/2006-March/msg00004.shtml
suggested.

>> It would be cool if there was a way to get condor_dagman to chdir() into
>> a directory before executing.  I looked at -UseDagDir, but this will put
>> output/log files in the parent directory - something I am trying to avoid.
>
>The way you have things right now, -UseDagDir has no effect because
>there is no path in the DAG file specification in your submit file.
>Basically, -UseDagDir does cause DAGMan to chdir() to the DAG file's
>directory before doing a condor_submit on a node job, or whatever.
>
>Okay, I *think* this will solve your problem:  when you run
>condor_submit_dag for your inner DAGs, run it in the top-level
>directory, like this:
>
>    condor_submit_dag -no_submit -UseDagDir 111/maindag_111.dag
>  
>
But, for my case, -UseDagDir does not cause DAGMan to change
a working directory when invoking condor_submit_dag.
After I submitted the outer DAG like

% condor_submit_dag outer.dag

the job of the inner DAG, Pre, failed because
the preparation program was not able to find
its input, preparation.inp . Moreover, the log file of
this job was put into the parent directory.
It seems that despite of using -UseDagDir, DAGMan kept
using the parent directory, which I'm trying to avoid.

Probably I miss some points in the thread.
So I'd be very grateful if anyone gives me any hint,
advice, or comments.

Thanks in advance.

-- 
Masakatsu Ito 
Nanotechnology Research Center
FUJITSU LABORATORIES LTD.