[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] could not make temporary directory: `/var/lib/condor/execute/dir_ Re: Fw: Xvfb | Docker Universe | Can't open display | (but works with docker run)



This morning I did a final test:

I recreated the DOcker container so that it would run inside as a non-privileged user, adding this to the Docker file:

# https://stackoverflow.com/questions/39855304/how-to-add-user-with-dockerfile
RUN useradd -ms /bin/bash  vault
USER vault

If I run the container locally on the Mac the answer of whoami is the username vault which is the expected answer:

$ whoami
vault

If I run the container on HTcondor, the error messages are:

$ cat err_test3.0
whoami: cannot find name for user ID 390870428
/usr/bin/id: cannot find name for group ID 390800513
/usr/bin/id: cannot find name for user ID 390870428
could not make temporary directory: `/var/lib/condor/execute/dir_40174/Graph_Xmgrace_30' at /usr/local/share/perl5/Chart/Graph/Xmgrace.pm line 136.

Looking back the error reported by /usr/bin/id were also present.
The 2 values are always the same.

I also tried to use:
export CONDOR_IDS="390870428.390800513"
but this never had any effect.

Could this be the source of why SELINUX does not trust the container process i.e. not being able to ascertain user credentials?

I tried to find is there is a way to add credentials within the .sub file but I have not found that info in manual/v8.6.13/condor_submit.html (for the version of HTcondor that is installed. I was wondering if there had been something like this that could be added in the .sub file:
user = jsgro
for example so that the system know who is running the command(s).

Is this something that makes sense?

Jean-Yves


From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Sent: Sunday, October 31, 2021 12:43 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Cc: JEAN-YVES SGRO <jsgro@xxxxxxxx>
Subject: Re: [HTCondor-users] could not make temporary directory: `/var/lib/condor/execute/dir_ Re: Fw: Xvfb | Docker Universe | Can't open display | (but works with docker run)
 
Hello Dima,

Final thoughts: it is very likely that Selinux is the culprit. (and thus I cannot fix it.)

I tried many more things, including changing the location of TEMP and TMPDIR to a /scratch directory "visible" to HTCondor with all permissions. However, this only added more errors from other software (ghostscript) that could not find some temp files.

This forum discussion was useful to read: [Solved] selinux - at wits end
From there it seemed that perhaps selinux does not allow to write within /var but that is where the Condor Temp directories are located.

This perl page dont-use-posix-to-create-temporary-files has a one-line script to check where the temporary folder is (and also shows temp file names on the fly. (Module File::Temp is installed by default.)

perl -MFile::Temp=tempfile -le 'print +((tempfile())[1])'

At the very beginning of the run.sh script that I send to HTcondor, the answer is:
/var/lib/condor/execute/dir_8341/jgrY5QWLsa
And the condor temporary directory is:
pwd:
/var/lib/condor/execute/dir_8341

But further down I change the location of TMP and TEMPDIR to /tmp and the answer is after the exportâ command is now: (hence no longer with /var in the address)

/tmp/Uj33_8mmOe

But the error persisted:
could not make temporary directory: `/tmp/Graph_Xmgrace_30' at /usr/local/share/perl5/Chart/Graph/Xmgrace.pm line 136.

So, it seems that if SELINUX is the culprit, it is not the location of the temporary folder, but rather simply the fact that the Xmgrace.pm program wants to create a temporary directory at all.

On another test... I was able to run the HTcondor Docker container "htcondor/mini" and it worked... it is in fact a nice tool to learn. But it does not have the Docker Universe installed (requires to install Docker engine etc.)

Thank you for the SElinux suggestion, it seems that the implication is that there is no readily solution to address this issue other than changing the Selinux rules. (but as an end-user I can't.)

Another solution will arise: not using the Docker container if possible and try to install everything natively on the cluster itself.

Thanks again,

Jean-Yves



From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Sent: Saturday, October 30, 2021 10:47 PM
To: htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Cc: JEAN-YVES SGRO <jsgro@xxxxxxxx>
Subject: Re: [HTCondor-users] could not make temporary directory: `/var/lib/condor/execute/dir_ Re: Fw: Xvfb | Docker Universe | Can't open display | (but works with docker run)
 
Hello Dima,

Thank you for your message.
I was not aware of Selinux, and I have read about it online.
It seems that a few years ago it caused problems with HTcondor.

The CentOS 7 within the container does not have this installed.

However, I finally found out that it is installed on the CentOS 7 running on the cluster that hosts HTcondor. It has a file called /etc/selinux/config which contains 2 uncommented lines that define the rules:

SELINUX=enforcing
SELINUXTYPE=targeted


Since this is part of the cluster installation, I can't disable it.
I am just an "end - user."

I discovered a Docker image that has a HTcondor installed. I will try to run that on my Mac and submit the job this way. But I might not have enough RAM to do all that... CHecking the documentation this seems like a dauting task and may not prove that Selinux is the culprit...

I had also thought of running a Docker image on the DOcker Universe to have "Double Docker" layers but it seems that in all cases the Kernel is shared in some way and that will probably not work.

I am trying to find an easier way to handle this, by perhaps creating an environment variable that provides a pre-defined temporary folder, pre-created, but I have not been successful in finding such customization on any of the xmgrace documentation on how to tell the program to use that one.... I just found perldoc.perl.org/File::Temp , but I am not sure that this would help in any way.

I am now trying with a directory created by the script.sh called Graph_Xmgrace_26 but 26 is not always the number used so it may not work. Indeed this time it is called... 27.
I discovered command to make many... I made 99 of them and made them all permissions:
mkdir -p Graph_Xmgrace_{1..99}
chmod a+rwx Graph_Xmgrace_*

But still could not even use #27 again even though it clearly existed.
could not make temporary directory: `/var/lib/condor/execute/dir_5652/Graph_Xmgrace_27'

Thank you for your suggestions and that of Selinux...
I'll keep digging!

Jean-Yves


From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of dmitri.maziuk@xxxxxxxxx <dmitri.maziuk@xxxxxxxxx>
Sent: Friday, October 29, 2021 9:21 PM
To: htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] could not make temporary directory: `/var/lib/condor/execute/dir_ Re: Fw: Xvfb | Docker Universe | Can't open display | (but works with docker run)
 
On 2021-10-29 6:02 PM, JEAN-YVES SGRO via HTCondor-users wrote:
...
> Any idea?
Did you disable selinux?

Dima
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/