[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Condor + Docker?

On 07/10/2014 01:57 PM, Branden Timm wrote:
This may be a better question for htcondor-devel, but I wonder if anybody has thought of or tried using Condor in conjunction with Docker? I can imagine several advantages that could be leveraged via Docker, including -

1) Deploying jobs to containers running pre-configured images containing all the software for that job (instead of, say, having software pre-installed across your cluster or setting up/tearing down requirements on job start/completion). Docker images tend to be lightweight, and they're cached once they're downloaded, so on repeated runs almost nothing would have to be transferred to the execute node. 2) Taking advantage of the underlying cgroups features of the Linux kernel to actually limit jobs to the requirements specified in their submit file while the system is under pressure (cpu, memory, and disk IO can all be throttled). Of course, the other advantage is that it's a share-based system, so you can guarantee a minimum amount of CPU access while still allowing it to burst beyond that on an otherwise idle system. 3) Complete namespace isolation (process, filesystem, and network) of running jobs from the rest of the system.

Docker is a technology that us HTCondor folks are keeping an eye on. Note that for bullet point #2 above, HTCondor today has support for using cgroups to sandbox the cpu and memory usage of a job, and to measure resource usage. HTCondor also optionally support process isolation on Linux systems with kernel support.