Re: [HTCondor-users] passing --pwd to singularity

[Michael Pelletier <michael.v.pelletier@xxxxxxxxxxxx>]

Regarding using values set at runtime in HTCondor job arguments, the important thing to remember is that the starter daemon does not use a shell to launch the executable and its arguments, so anything that is provided by the shell, including environment and other variable expansion, flow control statements, etc, is not available for the command line...

... unless ...

You change your "executable" to "/bin/sh" and use the "arguments" to lay out a "-c" command line to allow the shell to interpret the command line before launching the task at hand. This way the shell will expand all the shell variables, run a sequence of commands or an inline script, etc. For example:

Executable = /bin/sh
Transfer_executable = false
Arguments = "-c 'the_command -home $(DOLLAR)HOME -pwd $(DOLLAR)HOME' "

Now I'll grant that "$HOME" may not be a great example, since the submit description $ENV(HOME) value provides the $HOME environment variable - however, it provides it from the job submission environment, not from the runtime environment after starting up on a remote machine, so if those are different, you'll want to choose one or the other.

A more salient example would be something like this, if you have some decrepit tool that hardcodes /tmp as a default instead of using $TMPDIR:

Arguments = "-c 'decrepit_tool.exe -scratch $(DOLLAR)_CONDOR_SCRATCH_DIR' "

I use an approach like this in a Caffe submit description file, which adds a command-line argument containing the most recent available snapshot file found in the job's working directory by using an "ls" command. I think an example of it is in my most recent HTCondor Week presentation from 2018.

You can also combine the submission's "environment" directive to deliver details from the submit file into  the job's runtime environment without the job needing to query the .job.ad file:

Environment = " HTCONDOR_JOB_ID=$(Cluster).$(Process) JOB_STARTS=$$([NumJobStarts])"

I hope this is helpful for you!

Michael V. Pelletier
Information Technology
Digital Transformation & Innovation
Integrated Defense Systems
Raytheon Company