Re: [HTCondor-users] Propagate DOCKER_CONFIG to docker universe

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi Markus,

is there a strong need to use Docker as container runtime engine? I am 
always a bit wary due to needing the powerful daemon and container 
images not necessarily isolated from different users. Pulling and 
deleting image tarballs each time in a job sounds also like putting 
heavy strain on the endpoint (not that you hit a rate limit....

Would something like Apptainer in the the executing user's space also 
work for you? (if you need just OCI semantics and not networking and so on).
If you hit scaling issues, you could deploy your container images into a 
scalable caching tier system like CVMFS.

Cheers,
  Thomas


On 11/05/2023 14.20, Dr. Markus Stabrin wrote:
> Hello everybody,
>
>
> actually while writing my previous mail I had an idea about how to solve 
> it and apparently it works, so I will share my finding:
>
>
> I use the PreCmd to pull the image into the local cache using the 
> DOCKER_CONFIG variable.
>
> The docker universe then uses the pre-fetched image to execute the job.
>
> The PostCmd then deletes the image.
>
>
> I think this could lead to some issues when the image gets removed while 
> being used by other jobs, so something more sophisticated might be more 
> suitable, but it works for now:
>
>
> ```
> universe ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= docker
> docker_image ÂÂÂÂÂÂÂÂÂÂÂ= <private-image>
> RunAsOwner ÂÂÂÂÂÂÂÂÂÂÂÂÂ= True
> output ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdout.log
> error ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stderr.log
> log ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdlog.log
>
> +PreCmd = "/usr/bin/docker"
> +PreArguments = "pull <private-image>"
> +PreEnvironment = "DOCKER_CONFIG=/etc/docker/userconf"
> +PostCmd = "/usr/bin/docker"
> +PostArguments = "image rm -f <private-image>"
> queue
>
> ```
>
>
> If there are other best-practice solutions, I am still open for 
> suggestions ð
>
>
> Best,
>
> Markus
>
>
>
> ------------------------------------------------------------------------
> *From:* Dr. Markus Stabrin
> *Sent:* Thursday, May 11, 2023 1:35:00 PM
> *To:* htcondor-users@xxxxxxxxxxx
> *Subject:* Propagate DOCKER_CONFIG to docker universe
>
> Hello everybody,
>
>
> I am new to Condor and I am struggling to connect to a private docker 
> registry, which I have no control over, so a nginx proxy does not work.
>
> I also checked this: 
> https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml <https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml>
>
> But unfortunately it does not seem to work for me and I get an `access 
> forbidden` and I do not find much more information about the topic.
>
>
> I am running a setup where the actual owner is executing the job and the 
> credentials are stored in `${HOME}/.docker/config.json`.
>
> This fails:
> ```
> universe ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= docker
> docker_image ÂÂÂÂÂÂÂÂÂÂÂ= <private-image>
> RunAsOwner ÂÂÂÂÂÂÂÂÂÂÂÂÂ= True
> output ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdout.log
> error ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stderr.log
> log ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdlog.log
> queue
> ```
>
> But this works:
> ```
> executable ÂÂÂÂÂÂÂÂÂÂÂÂ = /usr/bin/docker
> arguments ÂÂ Â Â Â ÂÂ Â = "pull <private-image>"
> output ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdout.log
> error ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stderr.log
> log ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdlog.log
> RunAsOwner ÂÂÂÂÂÂÂÂÂÂÂÂÂ= True
> queue
> ```
> as well as if there is no home directory at all and the credentials are 
> stored in `/etc/docker/userconf/config.json`:
> ```
> executable ÂÂÂÂÂÂÂÂÂÂÂÂ = /usr/bin/docker
> arguments ÂÂ Â Â Â ÂÂ Â = "pull <private-image>"
> output ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdout.log
> error ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stderr.log
> log ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdlog.log
> RunAsOwner ÂÂÂÂÂÂÂÂÂÂÂÂÂ= True
> environment ÂÂÂÂÂÂÂÂÂÂÂ = "DOCKER_CONFIG=/etc/docker/userconf"
> queue
> ```
>
> So apparently, the docker universe does not read the `config.json` for 
> the user or from the DOCKER_CONFIG variable?
> I tried to put
> ```
> DOCKER_CONFIG = /etc/docker/userconf
> ```
> in `/etc/condor/config.d/01-execute.config` , but this does not seem to 
> be applied.
>
> I also tried to set the variable in a PreEnvironment, but I read that it 
> is not propagated to the execution environment.
>
> Does anyone have further ideas how to use the docker universe with a 
> private docker registry?
>
> Thank you very much,
> Markus
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature