Re: [HTCondor-users] Submitting to a remote condor queue

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 2/8/2016 8:11 AM, Brian Candler wrote:
> Hello,
>
> I would like to run some code inside a docker container, which submits
> jobs to a condor schedd running on the underlying host outside the
> container.
>
> +--------------------------+
> | +-----------+            |
> | | container | ---.       |
> | +-----------+    v       |
> |                schedd    |
> |                          |
> +--------------------------+
>
> The final goal here is to run some fairly complex code which writes out
> DAGs, and have that code bundled together with all its dependencies in a
> docker container.
[snip]
> P.S. Docker containers can include arbitrary usernames/UIDs, so it would
> probably best if all jobs submitted by these containers were mapped to a
> single userID in htcondor land. I'm sure I remember reading some way to get
>
> P.P.S. I realise that the DAG itself and the submit files used by that
> DAG also need to be visible on the host where dagman is running.  I
> expect I'll use a docker volume to expose some bit of host filesystem
> for this purpose.  But first I want to be clear on the right way to
> submit simple jobs remotely.

Hi Brian,

Before getting into HTCondor config details, I think you have a 
fundamental architecture decision to make.  Given all your comments 
above and given goal of your container is to write out DAG files, I 
question why the container itself should be submitting the DAG jobs.

Consider an alternative similar to the following:

Do a volume mount so that your container and your host share some 
subdirectory on the host file system.  In this subdirectory, create a 
"runme" directory where your container will atomically write out DAG 
files along with their corresponding submit files. Meanwhile on the host 
schedd, have a local universe job (submitted by whatever user you 
choose) that peridoically scans the "runme" directory for .dag files, 
submits them, and then renames the submitted .dag file to 
.dag.submitted.jobX.Y.

This way you do not need to do any reconfiguration of your host schedd, 
you don't need to have any trust relationships between your container 
and your host schedd, and you don't need to pay any extra overhead of 
having HTCondor move files in and out of the container via file transfer.

If you don't like the above and really want the container to submit jobs 
directly to the host schedd, you will need to configure some sort of 
shared trust between the container and the host schedd (either by 
telling the host schedd to use FS_REMOTE authentication to a volume 
mount, or giving the container an SSL or GSI credential that the host 
schedd maps to a user), and then you will want to do a condor_submit 
-remote to the host schedd.  Or if on Linux you could check out the 
Remote HTCondor contrib module at
  https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=RemoteCondor

But esp if you are planning to require a docker volume mount in any 
event, I would urge you to consider the above alternate architecture.


hope the above helps
Todd