Re: [HTCondor-users] _CONDOR_SCRATCH_DIR in submit file?

[Michael Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>]

The _CONDOR_SCRATCH_DIR is not set in the submitter's environment, which is where the $ENV() macro gets its data. It is only set when the job is matched to a resource and the starter fires up. This is why you saw the results you did.

You can enable file transfer so that the job's initial directory is in scratch, and set the directory to "./.mcrCache" (dot-slash), or for Linux you can set "mount_under_scratch = /tmp, /var/tmp" in the configuration so that it can use the default "/tmp/.mcrCacheXX" directory.

Another alternative is to use a startd cron job to insure that all of your exec nodes have a world-writable (chmod 1777) /tmp/.mcrCacheXX directory, named appropriately for the version, so that multiple users can share it.

One important thing to remember when using mount_under_scratch is that the PreCmd and PostCmd don't use the scratch-mounted directories, so if you're using compiled MATLAB as a PreCmd or PostCmd you'll need to set the cache root or use a shared /tmp cache.

On a related note, you can eliminate the need for the wrapper script that MCC generates by setting the LD_LIBRARY_PATH and the other environment variables set in the wrapper via the submit's environment directive.

	-Michael Pelletier.

-----Original Message-----
From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Dimitri Maziuk
Sent: Monday, October 24, 2016 7:28 PM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] _CONDOR_SCRATCH_DIR in submit file?

Hi all,

can I use _CONDOR_SCRATCH_DIR is a submit file?

I could swear I had it working without a wrapper script at one point but now I can't remember how (if) I did it.

This is a (compiled) matlab job that hits the usual "Could not access the MATLAB Runtime component cache."

With the submit file specifying:
environment = "MCR_CACHE_VERBOSE=1
MCR_CACHE_ROOT=$ENV(_CONDOR_SCRATCH_DIR) ...

The output is

> Creating MATLAB Runtime Cache at location: UNDEFINED/.mcrCache9.0.1 
> ...
> File open failed for 
> /var/lib/condor/execute/dir_1400101/UNDEFINED/.mcrCache9.0.1/.mcr_cach
> e_index
> ...
> Checking whether index file /var/lib/condor/execute/dir_1400101/UNDEFINED/.mcrCache9.0.1/.mcr_cache_index needs to be written...
> Write is needed.
> Writing cache index file: 
> /var/lib/condor/execute/dir_1400101/UNDEFINED/.mcrCache9.0.1/.mcr_cach
> e_index
> ...
> Unable to start MATLAB. Relative path for preferences directory 
> ('UNDEFINED/.mcrCache9.0.1/main_F0/.matlab') is not supported

Two questions: is it supposed to be "UNDEFINED" and how can I get the absolute path? -- it seems matlab writes to UNDEFINED just fine and only cares about relative path.

TIA
--
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu