[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Submitting to a remote condor queue

On 2/8/2016 8:11 AM, Brian Candler wrote:

I would like to run some code inside a docker container, which submits
jobs to a condor schedd running on the underlying host outside the

| +-----------+            |
| | container | ---.       |
| +-----------+    v       |
|                schedd    |
|                          |

The final goal here is to run some fairly complex code which writes out
DAGs, and have that code bundled together with all its dependencies in a
docker container.

P.S. Docker containers can include arbitrary usernames/UIDs, so it would
probably best if all jobs submitted by these containers were mapped to a
single userID in htcondor land. I'm sure I remember reading some way to get

P.P.S. I realise that the DAG itself and the submit files used by that
DAG also need to be visible on the host where dagman is running.  I
expect I'll use a docker volume to expose some bit of host filesystem
for this purpose.  But first I want to be clear on the right way to
submit simple jobs remotely.

Hi Brian,

Before getting into HTCondor config details, I think you have a fundamental architecture decision to make. Given all your comments above and given goal of your container is to write out DAG files, I question why the container itself should be submitting the DAG jobs.

Consider an alternative similar to the following:

Do a volume mount so that your container and your host share some subdirectory on the host file system. In this subdirectory, create a "runme" directory where your container will atomically write out DAG files along with their corresponding submit files. Meanwhile on the host schedd, have a local universe job (submitted by whatever user you choose) that peridoically scans the "runme" directory for .dag files, submits them, and then renames the submitted .dag file to .dag.submitted.jobX.Y.

This way you do not need to do any reconfiguration of your host schedd, you don't need to have any trust relationships between your container and your host schedd, and you don't need to pay any extra overhead of having HTCondor move files in and out of the container via file transfer.

If you don't like the above and really want the container to submit jobs directly to the host schedd, you will need to configure some sort of shared trust between the container and the host schedd (either by telling the host schedd to use FS_REMOTE authentication to a volume mount, or giving the container an SSL or GSI credential that the host schedd maps to a user), and then you will want to do a condor_submit -remote to the host schedd. Or if on Linux you could check out the Remote HTCondor contrib module at

But esp if you are planning to require a docker volume mount in any event, I would urge you to consider the above alternate architecture.

hope the above helps