[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] Create dir hierarchy for job output files?


Do your users need access to the /scratch space after their jobs have run? If not, you could just put your EXECUTE (config param) directory on /scratch, and let Condor create&destroy temporary directories in it for each job.

If you need the output for the user to access later you could use the new Job Hooks' Job Exit hook [1] to copy your user's data out of a Condor controlled execute directory into the scratch space. You may be able to use the Prepare Hook [2], but it'll probably have the same problem as the wrapper script.



[1] http://www.cs.wisc.edu/condor/manual/v7.1/4_4Job_Hooks.html#SECTION00542600000000000000 [2] http://www.cs.wisc.edu/condor/manual/v7.1/4_4Job_Hooks.html#SECTION00542400000000000000

Steffen Grunewald wrote:
Hi everyone,

as recommended, we're using local disk storage for job output, error, and
log files, and specify corresponding paths in the submit file, using
/scratch/<user>/output.$(Process) (etc., with <user> substituted with the
real account name) expressions.
Every now and then, a scratch disk has to be replaced after a disk failure,
which means that the user cannot rely on the directory structure being there
on all nodes anymore.
Is it feasible to create the underlying directory hierarchy if necessary?
Since the job would fail in the starter stage, there's no opportunity to
do this in a wrapper script (which wouldn't run).
Perhaps a new command for the submit file? (along the lines of 'pre-run="mkdir -p /scratch/me"', or 'createpath=True')
Perhaps something that's already there (and for some reason, invisible to