[HTCondor-users] singularity : user-job-wrapper not found

[Philippe Grassia <philippe.grassia@xxxxxxxx>]

When I run a simple job using a singularity container, it fails claiming 
a user-job-wrapper script is nowhere to be found and yet the script is 
present on both the submit and execute nodes, so I assume it is not 
found inside the singularity container. What is the canonical way to fix 
that ?

Note: my current workaround is to have a shell script as my executable 
that takes care of properly invoking singularity and it works.

submit file :

```
universe = vanilla
executable = pwd
should_transfer_files = YES
transfer_output_files = $(macrooutputDir)
+SingularityBindCVMFS = True
requirements = (HAS_SINGULARITY=?=TRUE)
+SingularityImage = 
"/cvmfs/ligo-containers.opensciencegrid.org/lscsoft/bayeswave/master"
transfer_executable = False
+PreArgs = "$(macrooutputDir)"
when_to_transfer_output = ON_EXIT
```

Result of the submitting that file

```
::::::::::::::
68.0.err
::::::::::::::
WARNING: Not mounting /tmp (already mounted in container)
WARNING: Not mounting /var/tmp (already mounted in container)
/.singularity.d/actions/exec: 9: exec: 
/etc/condor/modules/user-job-wrapper: not found
::::::::::::::
68.0.log
::::::::::::::
000 (068.000.000) 11/07 14:52:44 Job submitted from host: 
<1********************************+[--1]-9618&noUDP&sock=3304_d900_5>
...
001 (068.000.000) 11/07 14:54:09 Job executing on host: 
<1*********************************+[--1]-9618&noUDP&sock=3157_0494_3>
...
006 (068.000.000) 11/07 14:54:09 Image size of job updated: 0
ÂÂÂ 0Â -Â MemoryUsage of job (MB)
ÂÂÂ 0Â -Â ResidentSetSize of job (KB)
...
005 (068.000.000) 11/07 14:54:10 Job terminated.
ÂÂÂ (1) Normal termination (return value 127)
ÂÂÂÂÂÂÂ Usr 0 00:00:00, Sys 0 00:00:00Â -Â Run Remote Usage
ÂÂÂÂÂÂÂ Usr 0 00:00:00, Sys 0 00:00:00Â -Â Run Local Usage
ÂÂÂÂÂÂÂ Usr 0 00:00:00, Sys 0 00:00:00Â -Â Total Remote Usage
ÂÂÂÂÂÂÂ Usr 0 00:00:00, Sys 0 00:00:00Â -Â Total Local Usage
ÂÂÂ 225Â -Â Run Bytes Sent By Job
ÂÂÂ 0Â -Â Run Bytes Received By Job
ÂÂÂ 225Â -Â Total Bytes Sent By Job
ÂÂÂ 0Â -Â Total Bytes Received By Job
 Partitionable Resources : Usage Request Allocated
ÂÂÂÂÂÂ CpusÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 1ÂÂÂÂÂÂÂÂ 1
ÂÂÂÂÂÂ Disk (KB)ÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂ 15ÂÂÂÂÂÂÂ 0ÂÂ 2046334
ÂÂÂÂÂÂ GpusÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0
ÂÂÂÂÂÂ Memory (MB)ÂÂÂÂÂÂÂÂÂ :ÂÂÂÂÂÂÂ 0ÂÂÂÂ 2048ÂÂÂÂÂ 2048
...
::::::::::::::
68.0.out
::::::::::::::
```