Hi Markus,
if you have already a shared file system, than using it for Apptainer
(or "Singularity" before the fork) containers should be pretty straight
forward - it should work by pointing apptainer on the path of the local
container image [1] (SIF image or unpacked "sandbox" directory dir).
With the container universe, you can directly tell Condor to use the
image path for your job
https://htcondor.readthedocs.io/en/latest/users-manual/container-universe-jobs.html
If you or your community want to use containers on multiple sites, then
something like CVMFS (which is used by the HEP community as CDN) might
be an option. But that would need a bit of infrastructure and
preparations and might be overkill ;)
Cheers,
Thomas
[1]
<hartmath@naf-atlas21:~> apptainer shell
/nfs/dust/atlas/user/hartmath/container/almalinux9.d/
INFO: Environment variable SINGULARITY_TMPDIR is set, but
APPTAINER_TMPDIR is preferred
bash: warning: setlocale: LC_ALL: cannot change locale (en_DK.utf8): No
such file or directory
bash: git-branch-prompt: command not found
Apptainer> cat /etc/redhat-release
AlmaLinux release 9.1 (Lime Lynx)
bash: git-branch-prompt: command not found
On 11/05/2023 19.37, Markus Stabrin wrote:
> Hello Thomas,
>
>
> I am also looking into Apptainer and it actually appears to be the
> better fit for my use case since I mainly need the OCI semantics.
>
> However, in my first thought, I also planned to fetch the apptainer SIFs
> from the registry...
>
> But now that you mention it, it might put a heavy strain on the endpoint.
>
>
> Would you suggest to rely on a shared filesystem to cache the apptainer
> files?
>
> I think that should be possible.
>
>
> Best,
>
> Markus
>
> ------------------------------------------------------------------------
> *From:* Thomas Hartmann <thomas.hartmann@xxxxxxx>
> *Sent:* Thursday, May 11, 2023 3:05:38 PM
> *To:* HTCondor-Users Mail List; Markus Stabrin
> *Subject:* Re: [HTCondor-users] Propagate DOCKER_CONFIG to docker universe
> Hi Markus,
>
> is there a strong need to use Docker as container runtime engine? I am
> always a bit wary due to needing the powerful daemon and container
> images not necessarily isolated from different users. Pulling and
> deleting image tarballs each time in a job sounds also like putting
> heavy strain on the endpoint (not that you hit a rate limit....
>
> Would something like Apptainer in the the executing user's space also
> work for you? (if you need just OCI semantics and not networking and so on).
> If you hit scaling issues, you could deploy your container images into a
> scalable caching tier system like CVMFS.
>
> Cheers,
> Thomas
>
>
> On 11/05/2023 14.20, Dr. Markus Stabrin wrote:
>> Hello everybody,
>>
>>
>> actually while writing my previous mail I had an idea about how to solve
>> it and apparently it works, so I will share my finding:
>>
>>
>> I use the PreCmd to pull the image into the local cache using the
>> DOCKER_CONFIG variable.
>>
>> The docker universe then uses the pre-fetched image to execute the job.
>>
>> The PostCmd then deletes the image.
>>
>>
>> I think this could lead to some issues when the image gets removed while
>> being used by other jobs, so something more sophisticated might be more
>> suitable, but it works for now:
>>
>>
>> ```
>> universe = docker
>> docker_image = <private-image>
>> RunAsOwner = True
>> output = stdout.log
>> error = stderr.log
>> log = stdlog.log
>>
>> +PreCmd = "/usr/bin/docker"
>> +PreArguments = "pull <private-image>"
>> +PreEnvironment = "DOCKER_CONFIG=/etc/docker/userconf"
>> +PostCmd = "/usr/bin/docker"
>> +PostArguments = "image rm -f <private-image>"
>> queue
>>
>> ```
>>
>>
>> If there are other best-practice solutions, I am still open for
>> suggestions ð
>>
>>
>> Best,
>>
>> Markus
>>
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Dr. Markus Stabrin
>> *Sent:* Thursday, May 11, 2023 1:35:00 PM
>> *To:* htcondor-users@xxxxxxxxxxx
>> *Subject:* Propagate DOCKER_CONFIG to docker universe
>>
>> Hello everybody,
>>
>>
>> I am new to Condor and I am struggling to connect to a private docker
>> registry, which I have no control over, so a nginx proxy does not work.
>>
>> I also checked this:
>>
https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml <
https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml> <
https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml
<
https://www-auth.cs.wisc.edu/lists/htcondor-users/2017-October/msg00113.shtml>>
>>
>> But unfortunately it does not seem to work for me and I get an `access
>> forbidden` and I do not find much more information about the topic.
>>
>>
>> I am running a setup where the actual owner is executing the job and the
>> credentials are stored in `${HOME}/.docker/config.json`.
>>
>> This fails:
>> ```
>> universe = docker
>> docker_image = <private-image>
>> RunAsOwner = True
>> output = stdout.log
>> error = stderr.log
>> log = stdlog.log
>> queue
>> ```
>>
>> But this works:
>> ```
>> executable = /usr/bin/docker
>> arguments = "pull <private-image>"
>> output = stdout.log
>> error = stderr.log
>> log = stdlog.log
>> RunAsOwner = True
>> queue
>> ```
>> as well as if there is no home directory at all and the credentials are
>> stored in `/etc/docker/userconf/config.json`:
>> ```
>> executable = /usr/bin/docker
>> arguments = "pull <private-image>"
>> output = stdout.log
>> error = stderr.log
>> log = stdlog.log
>> RunAsOwner = True
>> environment = "DOCKER_CONFIG=/etc/docker/userconf"
>> queue
>> ```
>>
>> So apparently, the docker universe does not read the `config.json` for
>> the user or from the DOCKER_CONFIG variable?
>> I tried to put
>> ```
>> DOCKER_CONFIG = /etc/docker/userconf
>> ```
>> in `/etc/condor/config.d/01-execute.config` , but this does not seem to
>> be applied.
>>
>> I also tried to set the variable in a PreEnvironment, but I read that it
>> is not propagated to the execution environment.
>>
>> Does anyone have further ideas how to use the docker universe with a
>> private docker registry?
>>
>> Thank you very much,
>> Markus
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>>
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> <
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users>
>>
>> The archives can be found at:
>>
https://lists.cs.wisc.edu/archive/htcondor-users/
> <
https://lists.cs.wisc.edu/archive/htcondor-users/>
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif){kind=logo}