Re: [HTCondor-users] Singularity not mounting the home directory for a submitted job

[Tom Downes <downes@xxxxxxx>]

Michael:

What I'd wager is happening is that you have

mount home = yes

in your singularity.conf file. This has the effect of bind mounting $HOME onto /home/user-running-singularity in the container. Which means you can't bind mount host:/home onto the non-empty container:/home. If you have a situation where a job needs BOTH /home/A and /home/B, you should probably set "mount home = no" and then 

bind path = /home

or the corresponding  SINGULARITY_BIND_EXPR.

Tom

ïOn 10/15/18, 4:50 PM, "HTCondor-users on behalf of Todd Tannenbaum" <htcondor-users-bounces@xxxxxxxxxxx on behalf of tannenba@xxxxxxxxxxx> wrote:

    On 10/15/2018 1:42 PM, Michael Pelletier wrote:
    > Hi folks,
    > 
    > Does anyone have some insight into how Singularity decides when to bind-mount the user's home directory?
    > 
    > I've got a situation where when I run the command on the exec machine directly, the container spins up successfully with the home directory in place, but when started via condor_run -a 'MY.SingularityImage = "imagefile"' or via a submit description, the container doesn't get the home directory and thus the launch of the application fails.
    > 
    > The bind mount arguments include the /home/condor directory since that's where the global libexec lives in my config, but omitting that didn't change the behavior. An error message I see is "/home/user already mounted in container" - so it's apparently detecting something that isn't actually there.
    > 
    > I have a user_job_wrapper in place, but omitting that didn't change this behavior. I wrapped the Singularity binary as well in order to add the "--nv" option for jobs where it's appropriate until SINGULARITY_EXTRA_ARGUMENTS is in a production release, but that was another dead end investigation. I tried inserting a "-H ${HOME}:$HOME" argument along with --nv, but that seems to be getting ignored with the "already mounted in container" message.
    > 
    > What's even more peculiar is that it works for some user accounts, but not others. I checked the environment variables for any differences, but nothing jumped out at me, and condor_run does a getenv=True in any case.
    > 
    > Any suggestions? Thanks
    > 
    
    I am not the Singularity support expert, but maybe HTCondor config knob 
    SINGULARITY_BIND_EXPR could help?  Ie something like:
    
       SINGULARITY_BIND_EXPR = strcat("/home/",Owner)
    
    regards,
    Todd
    
    _______________________________________________
    HTCondor-users mailing list
    To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
    subject: Unsubscribe
    You can also unsubscribe by visiting
    https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
    
    The archives can be found at:
    https://lists.cs.wisc.edu/archive/htcondor-users/