[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Docker and Condor Versions



-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I've run into this a bit too, and it's not as straightforward as the
error suggests. I haven't put much effort into debugging/fixing this
myself because we don't presently care if HasDocker is set for our use,
but condor's definitely doing something funny.

StarterLog has lines as mentioned: 

04/18/17 13:13:30 (pid:2427) Read more than one line (or a very long
line) from '/usr/bin/docker -v', which we think means it's not Docker.
The (first line of the) trailing text was 'WARNING: Error loading config
file:/root/.docker/config.json - stat /root/.docker/config.json:
permission denied'.
04/18/17 13:13:30 (pid:2427) DockerAPI::detect() failed to detect the
Docker version; assuming absent.

However, running the same command separately does not produce messages
under any user I can find:

[mian@cnode15 condor]$ /usr/bin/docker -v
Docker version 17.03.1-ce, build c6d412e
[mian@cnode15 condor]$ sudo -u condor -s /bin/bash
bash-4.2$ /usr/bin/docker -v
Docker version 17.03.1-ce, build c6d412e
bash-4.2$ exit
[mian@cnode15 condor]$ sudo -i
[root@cnode15 ~]# /usr/bin/docker -v
Docker version 17.03.1-ce, build c6d412e

There's no such directory `/root/.docker` in the first place -- it's not
a required configuration file or anything like that. I can't figure out
how to reproduce the log line condor reports nor why it's erroring at
all on a totally optional configuration file. My docker version seems to
be different than the initial report; I can say it seems to work fine on
docker 1.11.2, which is what we're using on our other execute nodes, so
perhaps it's something introduced with docker 1.12. I'm also running
condor 8.6.1, not 8.4.

I can also confirm that docker works fine from any user with the
appropriate group access on the host. It's just Condor that thinks
docker's broken, as far as I can tell.

- --
Ian

On Fri, Apr 21, 2017 at 05:34:07PM +0000, John M Knoeller wrote:
> this error message from HTCondorâs docker detection code seems pretty clear.
> 
> 'WARNING: Error loading config file:/root/.docker/config.json - stat /root/.docker/config.json: permission denied'
> 
> Your problem isnât the version of docker.  itâs that docker doesnât have the right to read its own configuration file.
> 
> -tj
> 
> 
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Knudson, Jonathan
> Sent: Friday, April 21, 2017 12:19 PM
> To: htcondor-users@xxxxxxxxxxx
> Subject: [HTCondor-users] Docker and Condor Versions
> 
> So here is my dilemma.  A while back, 2-3 months, we started upgrading our pool and during the upgrade I noticed that Docker jobs stopped running on the machines we were upgrading.
> 
> The machines that work are on the following versions.
> 
> Condor = 8.4.10
> Docker = 1.12.6
> 
> Here are the versions on one of the machine that won't run these Docker jobs
> 
> Docker version = 1.13.0
> Condor version = 8.6.0 and 8.6.1
> 
> I read in the release notes that there are version issues and this version combination should work.
> 
> Here is the only thing in the logs that point/say anything about Docker.
> 
> 04/18/17 15:54:35 (pid:28577) Read more than one line (or a very long line) from '/usr/bin/docker -v', which we think means it's not Docker.  The (first line of the) trailing text was 'WARNING: Error loading config file:/root/.docker/config.json - stat /root/.docker/config.json: permission denied'.
> 04/18/17 15:54:35 (pid:28577) DockerAPI::detect() failed to detect the Docker version; assuming absent.
> 04/18/17 15:54:35 (pid:28577) '/usr/bin/singularity --version' did not exit successfully (code 32750); the first line of output was ''.
> 
> # docker version
> Client:
>  Version:      1.13.1
>  API version:  1.26
>  Go version:   go1.7.5
>  Git commit:   092cba3
>  Built:        Wed Feb  8 06:38:28 2017
>  OS/Arch:      linux/amd64
> 
> Server:
>  Version:      1.13.1
>  API version:  1.26 (minimum version 1.12)
>  Go version:   go1.7.5
>  Git commit:   092cba3
>  Built:        Wed Feb  8 06:38:28 2017
>  OS/Arch:      linux/amd64
>  Experimental: false
> 
> I don't want to upgrade the older versions because they are running properly... but if it would help get everything running, then that's what will need to be done.  Does anyone have any thoughts on how to get this running the way it's supposed to?
> 
> Oh and I also tried downgrading and that didn't work either.
> 
> 
> Thanks
> â for your help.â
> 
> Jon
> 

> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCAAGBQJY+mnTAAoJEBhBn2xUAkHRa2cH/jqt/yOlb2ub+PEM1dOGvIBs
Ug6t8uX2qQ1RzX7xG95SmxN8+BvOhKYpWcd+4MIME7F6j2LKZKGqAheVhSs71sEy
NFmOKyQjv29JI76cOe3x2eWaK9toJ5r+k0KN98hZ8XGXBYXoee+j19zU9ge/LZjq
9YQ5SP3jjx3n3LFh1Gi+wTOzWoucrIB7QYysvE+7QHM00Mf/Bx1v1EUb4prMLkmc
zsXJAyFCAKoeH4URfsQmLwKhs36qxjkNdktd6Ugg6obraGB3ww5RgiamYwV34+jL
pcKO/GykcnGwMrQ3JEYYgQ4nrAH+1Dyv7lE27ejysMpTW1zpibc37NUI+7+Ax0s=
=uS5P
-----END PGP SIGNATURE-----