[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[HTCondor-users] Condor + Docker?
- Date: Thu, 10 Jul 2014 13:57:43 -0500
- From: Branden Timm <btimm@xxxxxxxxxxxxxxx>
- Subject: [HTCondor-users] Condor + Docker?
This may be a better question for htcondor-devel, but I wonder if
anybody has thought of or tried using Condor in conjunction with
Docker? I can imagine several advantages that could be leveraged via
Docker, including -
1) Deploying jobs to containers running pre-configured images containing
all the software for that job (instead of, say, having software
pre-installed across your cluster or setting up/tearing down
requirements on job start/completion). Docker images tend to be
lightweight, and they're cached once they're downloaded, so on repeated
runs almost nothing would have to be transferred to the execute node.
2) Taking advantage of the underlying cgroups features of the Linux
kernel to actually limit jobs to the requirements specified in their
submit file while the system is under pressure (cpu, memory, and disk IO
can all be throttled). Of course, the other advantage is that it's a
share-based system, so you can guarantee a minimum amount of CPU access
while still allowing it to burst beyond that on an otherwise idle system.
3) Complete namespace isolation (process, filesystem, and network) of
running jobs from the rest of the system.
I can imagine this being implemented in a sort of hacky way in the
vanilla universe, where the executable for a job is simply a "docker run
<image> <command>" and it gets scheduled like any other condor job,
using macros in the submit file to control resource limits within the
arguments section. I could also imagine the implementation of a
Docker/container universe where Condor automatically does these things
for you based on normal arguments to the submit file, which would
obviously be substantially more work but a more elegant solution.
Just curious if anybody else has been thinking of this. Cheers!
Wisconsin Energy Institute