Hello Thomas,
I am also looking into Apptainer and it actually appears to be the
better fit for my use case since I mainly need the OCI semantics.
However, in my first thought, I also planned to fetch the apptainer SIFs
from the registry...
But now that you mention it, it might put a heavy strain on the endpoint.
Would you suggest to rely on a shared filesystem to cache the apptainer
files?
I think that should be possible.
Best,
Markus
------------------------------------------------------------------------
*From:* Thomas Hartmann <thomas.hartmann@xxxxxxx>
*Sent:* Thursday, May 11, 2023 3:05:38 PM
*To:* HTCondor-Users Mail List; Markus Stabrin
*Subject:* Re: [HTCondor-users] Propagate DOCKER_CONFIG to docker universe
Hi Markus,
is there a strong need to use Docker as container runtime engine? I am
always a bit wary due to needing the powerful daemon and container
images not necessarily isolated from different users. Pulling and
deleting image tarballs each time in a job sounds also like putting
heavy strain on the endpoint (not that you hit a rate limit....
Would something like Apptainer in the the executing user's space also
work for you? (if you need just OCI semantics and not networking and so on).
If you hit scaling issues, you could deploy your container images into a
scalable caching tier system like CVMFS.
Cheers,
ÂÂ Thomas
On 11/05/2023 14.20, Dr. Markus Stabrin wrote:
Hello everybody,
actually while writing my previous mail I had an idea about how to solve
it and apparently it works, so I will share my finding:
I use the PreCmd to pull the image into the local cache using the
DOCKER_CONFIG variable.
The docker universe then uses the pre-fetched image to execute the job.
The PostCmd then deletes the image.
I think this could lead to some issues when the image gets removed while
being used by other jobs, so something more sophisticated might be more
suitable, but it works for now:
```
universe ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= docker
docker_image ÂÂÂÂÂÂÂÂÂÂÂ= <private-image>
RunAsOwner ÂÂÂÂÂÂÂÂÂÂÂÂÂ= True
output ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdout.log
error ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stderr.log
log ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdlog.log
+PreCmd = "/usr/bin/docker"
+PreArguments = "pull <private-image>"
+PreEnvironment = "DOCKER_CONFIG=/etc/docker/userconf"
+PostCmd = "/usr/bin/docker"
+PostArguments = "image rm -f <private-image>"
queue
```
If there are other best-practice solutions, I am still open for
suggestions ð
Best,
Markus
------------------------------------------------------------------------
*From:* Dr. Markus Stabrin
*Sent:* Thursday, May 11, 2023 1:35:00 PM
*To:* htcondor-users@xxxxxxxxxxx
*Subject:* Propagate DOCKER_CONFIG to docker universe
Hello everybody,
I am new to Condor and I am struggling to connect to a private docker
registry, which I have no control over, so a nginx proxy does not work.
I also checked this:
https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml <https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml> <https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml <https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml>>
But unfortunately it does not seem to work for me and I get an `access
forbidden` and I do not find much more information about the topic.
I am running a setup where the actual owner is executing the job and the
credentials are stored in `${HOME}/.docker/config.json`.
This fails:
```
universe ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= docker
docker_image ÂÂÂÂÂÂÂÂÂÂÂ= <private-image>
RunAsOwner ÂÂÂÂÂÂÂÂÂÂÂÂÂ= True
output ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdout.log
error ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stderr.log
log ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdlog.log
queue
```
But this works:
```
executable ÂÂÂÂÂÂÂÂÂÂÂÂ = /usr/bin/docker
arguments ÂÂ Â Â Â ÂÂ Â = "pull <private-image>"
output ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdout.log
error ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stderr.log
log ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdlog.log
RunAsOwner ÂÂÂÂÂÂÂÂÂÂÂÂÂ= True
queue
```
as well as if there is no home directory at all and the credentials are
stored in `/etc/docker/userconf/config.json`:
```
executable ÂÂÂÂÂÂÂÂÂÂÂÂ = /usr/bin/docker
arguments ÂÂ Â Â Â ÂÂ Â = "pull <private-image>"
output ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdout.log
error ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stderr.log
log ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ= stdlog.log
RunAsOwner ÂÂÂÂÂÂÂÂÂÂÂÂÂ= True
environment ÂÂÂÂÂÂÂÂÂÂÂ = "DOCKER_CONFIG=/etc/docker/userconf"
queue
```
So apparently, the docker universe does not read the `config.json` for
the user or from the DOCKER_CONFIG variable?
I tried to put
```
DOCKER_CONFIG = /etc/docker/userconf
```
in `/etc/condor/config.d/01-execute.config` , but this does not seem to
be applied.
I also tried to set the variable in a PreEnvironment, but I read that it
is not propagated to the execution environment.
Does anyone have further ideas how to use the docker universe with a
private docker registry?
Thank you very much,
Markus
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
<https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users>
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
<https://lists.cs.wisc.edu/archive/htcondor-users/>
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif){kind=logo}