[HTCondor-users] Problems with singularity version 3.8.1

[Matthias Schnepf <matthias.schnepf@xxxxxxx>]

Hi all,

Since we updated singularity to version 3.8.1, HTCondor (version 9.0.4) fails to start jobs in singularity container.

09/03/21 15:54:00 (pid:697043) (D_ALWAYS) Running job via singularity. 09/03/21 15:54:00 (pid:697043) (D_ALWAYS) singularity test returns 36096 09/03/21 15:54:00 (pid:697043) (D_ALWAYS) Non zero return code 36096 from singularity test of /usr/bin/singularity test --pwd /srv -B /var/lib/condor/execute/dir_695935/execute/dir_697043:/srv -B /cvmfs:/cvmfs -B /etc/cvmfs/SITECONF/KIT-T3:/etc/cvmfs/SITECONF -B /etc/cvmfs/default.local -B /etc/hosts -B /etc/localtime --no-home -C --env SINGULARITY_BIND= /cvmfs/unpacked.cern.ch/registry.hub.docker.com/matterminers/wlcg-wn:latest09/03/21 15:54:00 (pid:697043) (D_ALWAYS) singularity output was WARNING: While bind mounting '/etc/cvmfs/default.local:/etc/cvmfs/default.local': destination is already in the mount point list 09/03/21 15:54:00 (pid:697043) (D_ALWAYS|D_FAILURE) ERROR "Singularity test failed:WARNING: While bind mounting '/etc/cvmfs/default.local:/etc/cvmfs/default.local': destination is already in the mount point list" at line 530 in file /var/lib/condor/execute/slot1/dir_51204/userdir/.tmpqXU8eJ/BUILD/condor-9.0.4/src/condor_starter.V6.1/os_proc.cpp09/03/21 15:54:00 (pid:697043) (D_ALWAYS) ShutdownFast all jobs.

With singularity version 3.8.0 it run fine.

09/03/21 15:46:05 (pid:1661577) (D_ALWAYS) Running job via singularity.
09/03/21 15:46:06 (pid:1661577) (D_ALWAYS) singularity test returns 0

The only difference between 3.8.0 and 3.8.1 is an additional warning printout:

$ singularity --version
singularity version 3.8.0-1.el7

$ /usr/bin/singularity test --pwd /srv -B /cvmfs:/cvmfs -B /etc/cvmfs/SITECONF/KIT-T3:/etc/cvmfs/SITECONF -B /etc/cvmfs/default.local -B /etc/hosts -B /etc/localtime --no-home -C --env SINGULARITY_BIND= /cvmfs/unpacked.cern.ch/registry.hub.docker.com/matterminers/wlcg-wn:latest
INFO: No test script found in container, exiting No test found in container, executing /bin/sh -c true


$ echo $?
0

$ singularity --version
singularity version 3.8.1-1.el7

$ /usr/bin/singularity test --pwd /srv -B /cvmfs:/cvmfs -B /etc/cvmfs/SITECONF/KIT-T3:/etc/cvmfs/SITECONF -B /etc/cvmfs/default.local -B /etc/hosts -B /etc/localtime --no-home -C --env SINGULARITY_BIND= /cvmfs/unpacked.cern.ch/registry.hub.docker.com/matterminers/wlcg-wn:latest
WARNING: While bind mounting '/etc/cvmfs/default.local:/etc/cvmfs/default.local': destination is already in the mount point list
WARNING: While bind mounting '/etc/hosts:/etc/hosts': destination is already in the mount point list
WARNING: While bind mounting '/etc/localtime:/etc/localtime': destination is already in the mount point list
INFO: No test script found in container, exiting No test found in container, executing /bin/sh -c true

$ echo $?
0

It seems that condor has problems with the longer output.

Regards,

Matthias

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature