Re: [HTCondor-users] Propagate DOCKER_CONFIG to docker universe

[Markus Stabrin <Markus.Stabrin@xxxxxxxxxxxxxxxxxxx>]

Hello Thomas,

I am also looking into Apptainer and it actually appears to be the better fit for my use case since I mainly need the OCI semantics.

However, in my first thought, I also planned to fetch the apptainer SIFs from the registry...

But now that you mention it, it might put a heavy strain on the endpoint.

Would you suggest to rely on a shared filesystem to cache the apptainer files?

I think that should be possible.

Best,

Markus

---

From: Thomas Hartmann <thomas.hartmann@xxxxxxx>
Sent: Thursday, May 11, 2023 3:05:38 PM
To: HTCondor-Users Mail List; Markus Stabrin
Subject: Re: [HTCondor-users] Propagate DOCKER_CONFIG to docker universe

 

Hi Markus,

is there a strong need to use Docker as container runtime engine? I am
always a bit wary due to needing the powerful daemon and container
images not necessarily isolated from different users. Pulling and
deleting image tarballs each time in a job sounds also like putting
heavy strain on the endpoint (not that you hit a rate limit....

Would something like Apptainer in the the executing user's space also
work for you? (if you need just OCI semantics and not networking and so on).
If you hit scaling issues, you could deploy your container images into a
scalable caching tier system like CVMFS.

Cheers,
   Thomas


On 11/05/2023 14.20, Dr. Markus Stabrin wrote:
> Hello everybody,
>
>
> actually while writing my previous mail I had an idea about how to solve
> it and apparently it works, so I will share my finding:
>
>
> I use the PreCmd to pull the image into the local cache using the
> DOCKER_CONFIG variable.
>
> The docker universe then uses the pre-fetched image to execute the job.
>
> The PostCmd then deletes the image.
>
>
> I think this could lead to some issues when the image gets removed while
> being used by other jobs, so something more sophisticated might be more
> suitable, but it works for now:
>
>
> ```
> universe                = docker
> docker_image            = <private-image>
> RunAsOwner              = True
> output                  = stdout.log
> error                   = stderr.log
> log                     = stdlog.log
>
> +PreCmd = "/usr/bin/docker"
> +PreArguments = "pull <private-image>"
> +PreEnvironment = "DOCKER_CONFIG=/etc/docker/userconf"
> +PostCmd = "/usr/bin/docker"
> +PostArguments = "image rm -f <private-image>"
> queue
>
> ```
>
>
> If there are other best-practice solutions, I am still open for
> suggestions ð
>
>
> Best,
>
> Markus
>
>
>
> ------------------------------------------------------------------------
> *From:* Dr. Markus Stabrin
> *Sent:* Thursday, May 11, 2023 1:35:00 PM
> *To:* htcondor-users@xxxxxxxxxxx
> *Subject:* Propagate DOCKER_CONFIG to docker universe
>
> Hello everybody,
>
>
> I am new to Condor and I am struggling to connect to a private docker
> registry, which I have no control over, so a nginx proxy does not work.
>
> I also checked this:
> https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml <https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml>
>
> But unfortunately it does not seem to work for me and I get an `access
> forbidden` and I do not find much more information about the topic.
>
>
> I am running a setup where the actual owner is executing the job and the
> credentials are stored in `${HOME}/.docker/config.json`.
>
> This fails:
> ```
> universe                = docker
> docker_image            = <private-image>
> RunAsOwner              = True
> output                  = stdout.log
> error                   = stderr.log
> log                     = stdlog.log
> queue
> ```
>
> But this works:
> ```
> executable              = /usr/bin/docker
> arguments               = "pull <private-image>"
> output                  = stdout.log
> error                   = stderr.log
> log                     = stdlog.log
> RunAsOwner              = True
> queue
> ```
> as well as if there is no home directory at all and the credentials are
> stored in `/etc/docker/userconf/config.json`:
> ```
> executable              = /usr/bin/docker
> arguments               = "pull <private-image>"
> output                  = stdout.log
> error                   = stderr.log
> log                     = stdlog.log
> RunAsOwner              = True
> environment             = "DOCKER_CONFIG=/etc/docker/userconf"
> queue
> ```
>
> So apparently, the docker universe does not read the `config.json` for
> the user or from the DOCKER_CONFIG variable?
> I tried to put
> ```
> DOCKER_CONFIG = /etc/docker/userconf
> ```
> in `/etc/condor/config.d/01-execute.config` , but this does not seem to
> be applied.
>
> I also tried to set the variable in a PreEnvironment, but I read that it
> is not propagated to the execution environment.
>
> Does anyone have further ideas how to use the docker universe with a
> private docker registry?
>
> Thank you very much,
> Markus
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/