[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] set_DockerImage for local image

Hi Todd. 

Some suggestions for using docker inages. 
1. Pull from private registry. 
2. Allow users to provide user and password for the docker pull. This will allow diffrent team using diffrent images.
3. Allow users to force pull even of image exists. 
4. Allow user to configure swap space for docker. 


Get Outlook for Android

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Todd Tannenbaum <tannenba@xxxxxxxxxxx>
Sent: Friday, May 28, 2021, 21:29
To: HTCondor-Users Mail List; jcaballero.hep@xxxxxxxxx; Condor-Users Mail List
Subject: Re: [HTCondor-users] set_DockerImage for local image

On 5/28/2021 3:21 AM, jcaballero.hep@xxxxxxxxx wrote:

I understand that the image set in classad DockerImage is being pulled
from DockerHub, is that correct?
I would be interested in using the image from local filesystem rather
than a docker registry.
Is that something that can be done via that, or any other, classad?

Thanks a lot in advance.

Hi Jose,

IIRC, the image set in the DockerImage is currently only pulled over the network from a hub if the image is not already stored on local disk in Docker's own image cache (typically in /var/lib/Docker/...). Not sure if this fits your use case, but if you just want some special image to be available on all your worker nodes, as the sysadmin you could issue a "docker load ..." or "docker import ..." command (depending on if you created the image on your local filesystem with 'docker save' or 'docker export', respectively) on each worker node outside of HTCondor in order to copy the image from local disk into Docker's own local image cache.    Then jobs could use this image via Docker Universe as usual, no hub involved.  

One complication: by default HTCondor does garbage collection of images in the Docker local cache to keep the number if images there from infinitely growing.  If your Docker Universe jobs use a many different images, this could result in the special image you loaded into the cache being removed.   You may want to customize HTCondor config knob DOCKER_IMAGE_CACHE_SIZE to prevent this from happening.

Finally, more flexible docker image management is something we are planning to improve in HTCondor v9.1.x series.  Including support for private images, and images transferred by HTCondor file transfer as part of the job's usual sandbox of files.  So hopefully the above will get easier in the near future.

Hope this helps,


HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at:

Todd Tannenbaum <tannenba@xxxxxxxxxxx>  University of Wisconsin-Madison
Center for High Throughput Computing    Department of Computer Sciences
Calendar: https://tinyurl.com/yd55mtgd  1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                   Madison, WI 53706-1685