Re: [HTCondor-users] Renaming input files on transfer

[Ben Cotton <ben.cotton@xxxxxxxxxxxxxxxxxx>]

On Tue, May 27, 2014 at 1:05 PM, Brian Candler <b.candler@xxxxxxxxx> wrote:
> Question: is there a way to rename input files when transferring them?

Not that I'm aware of, which isn't the same as "no".

> (1) Potential name clashes. e.g.
> transfer_input_files = /foo/file1,/bar/file1
>
One option is to transfer the entire directory, e.g.:

    transfer_input_files = /foo, /bar

>From the condor_submit man page:
"When there is no trailing path separator, the directory is
transferred, its contents are transferred, and these contents are
placed inside the transferred directory."

> Or consider if a job writes "file1" as a temporary file, but this happens to
> be the same as the name of an input file.
>
That's bad. Assuming you can't make the job behave differently, you
should give the input file a different name before submission.

> (2) Scripts need to parse the basename of files. e.g. when you write a
> submit file you may have to do
>
> executable = myscript
> transfer_input_files = $1,$2
> arguments = "'$(basename "$1")' '$(basename "$2")'
>
If $1 is '/foo/file1', then it will end up in the execute directory as
file1. If you're writing the submit file from a program, I can see how
you'd need something like that in the script, but even if
transfer_input_remaps existed, you'd still probably want to do that, I
think. I'm just not sure what the real-world use case is here.

> Any option I've overlooked?
>
If your job has very specific and inflexible needs in regards to where
files are stored, you might try setting up the file structure with a
PreCmd:

+PreCmd = "executable"
A vanilla universe job may specify that a script is to be run on the
execute machine before the job, and this is called a prescript.
Definition of this specifically-named attribute causes the script,
identified by path and file name, to be executed. The prescript could
prepare or initialize the job. Note that this definition of a
prescript is different from the PRE script described in DAGMan. The
prescript is not automatically transferred with the job, as the main
executable is, so it must be entered into the transfer_input_files
list, when file transfer is enabled.


Hope this helps!
BC

-- 
Ben Cotton
main: 888.292.5320

Cycle Computing
Leader in Utility HPC Software

http://www.cyclecomputing.com
twitter: @cyclecomputing