[HTCondor-users] Singularity fails after upgrading from 2.6 to 3.1 with dirs not mountable to scratch

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi all,

as a warning: after updating Singularity from
  v2.6.1
to
  v3.1
jobs requesting to be started in a Singularity container fail with an
error message, that dirs cannot be created under the scratch space [1]
(update was more or less accidental with repo priorities needing
adjustment...)
Singularity extra arguments etc. are as [2].

Thing seems to be, that I can only bind one directory into scratch but
not two (which worked with 2.6).

I.e.,

* fails with 3.1 but works for 2.6
> singularity shell --scratch /tmp,/var/tmp  --contain --cleanenv --bind
/cvmfs:/cvmfs,/pnfs:/pnfs /path/to/container/...
smae with two
> singularity shell --scratch /tmp --scratch /var/tmp  --contain
--cleanenv --bind /cvmfs:/cvmfs,/pnfs:/pnfs /path/to/container/...

* works for 3.1 but only with one scratch dir bind
> singularity shell --scratch /tmp  --contain --cleanenv --bind
/cvmfs:/cvmfs,/pnfs:/pnfs /path/to/container/...

Does somebody else observes such a behaviour?

btw: Condor is on 8.7 [3]

Cheers,
  Thomas

[1]
FATAL:   container creation failed: could not create scratch
working directory /var/singularity/mnt/session: /scratch/tmp already
exists in layout


[2]
MOUNT_UNDER_SCRATCH = /tmp, /var/tmp,
SINGULARITY_JOB = !isUndefined(TARGET.SingularityImage)
SINGULARITY_IMAGE_EXPR = TARGET.SingularityImage
SINGULARITY_EXTRA_ARGUMENTS = "  --contain --cleanenv --bind
/cvmfs:/cvmfs --bind /afs:/afs --bind /pnfs:/pnfs --bind
/etc/batch:/etc/batch  "

[3]
condor-external-libs-8.7.10-1.el7.x86_64
condor-classads-8.7.10-1.el7.x86_64
condor-python-8.7.10-1.el7.x86_64
condor-8.7.10-1.el7.x86_64
condor-procd-8.7.10-1.el7.x86_64

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature