[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] Create dir hierarchy for job output files?
- Date: Tue, 24 Jun 2008 07:44:26 -0500
- From: Matthew Farrellee <mfarrellee@xxxxxxxxxx>
- Subject: Re: [Condor-users] Create dir hierarchy for job output files?
Do your users need access to the /scratch space after their jobs have
run? If not, you could just put your EXECUTE (config param) directory on
/scratch, and let Condor create&destroy temporary directories in it for
If you need the output for the user to access later you could use the
new Job Hooks' Job Exit hook  to copy your user's data out of a
Condor controlled execute directory into the scratch space. You may be
able to use the Prepare Hook , but it'll probably have the same
problem as the wrapper script.
Steffen Grunewald wrote:
as recommended, we're using local disk storage for job output, error, and
log files, and specify corresponding paths in the submit file, using
/scratch/<user>/output.$(Process) (etc., with <user> substituted with the
real account name) expressions.
Every now and then, a scratch disk has to be replaced after a disk failure,
which means that the user cannot rely on the directory structure being there
on all nodes anymore.
Is it feasible to create the underlying directory hierarchy if necessary?
Since the job would fail in the starter stage, there's no opportunity to
do this in a wrapper script (which wouldn't run).
Perhaps a new command for the submit file? (along the lines of
'pre-run="mkdir -p /scratch/me"', or 'createpath=True')
Perhaps something that's already there (and for some reason, invisible to