On Thu, Aug 12, 2010 at 6:52 PM, Mag Gam
<magawake@xxxxxxxxx> wrote:
We compile a lot of software; each build takes about 3-4 hours. I was
curious if anyone is using condor to build their software. For
example, make -j will be greedy and run as many programs as possible
on your local box. But is there a way to submit these tasks to the
condor pool? Are there any techniques for this?
Hi Mag,
I've done Condor-pool based compiles, but it's always been a custom build flow. Makefile-driven, but with distinct subsystems and enforced rules for inter-subsystem dependencies which guaranteed I could parallelize the complies of most of the subsystems.
I broke it down into what was, more or less, a DAG that had the following levels:
1. Pre-parallel compilation, done on one single machine where make could do it's local parallelization
2. Fully parallelized subsystem builds over multiple machines, no parallelism in make (or very minor, like maybe 2 or 3 threads)
3. A post-parallel compilation stage for some subsystems that had dependencies. There was Condor-level parallelism here but it was for data generation that went along with our builds, not for compiling code.
4. Any post-compilation linking
5. Testing
6. Publishing build
I didn't actually using DAGMan to drive it. It was all driven from custom DAG-like tools because we felt there was too much conductor-type work that needed to be done, especially when we started to doing recovery from failures and such, that we could only do with custom logic.
It was all done on a shared filesystem. Linux targeted builds. We never got it working for Windows.
- Ian
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif){kind=logo}