Re: [Condor-users] building software with condor

[Mag Gam <magawake@xxxxxxxxx>]

Thanks for the response. It seems like a lot of work. I was expecting
an easy conversion of make -j :-)



On Thu, Aug 12, 2010 at 9:03 PM, Ian Chesal <ichesal@xxxxxxxxxxxxxxxxxx> wrote:
> On Thu, Aug 12, 2010 at 6:52 PM, Mag Gam <magawake@xxxxxxxxx> wrote:
>>
>> We compile a lot of software; each build takes about 3-4 hours. I was
>> curious if anyone is using condor to build their software. For
>> example, make -j will be greedy and run as many programs as possible
>> on your local box. But is there a way to submit these tasks to the
>> condor pool? Are there any techniques for this?
>
> Hi Mag,
> I've done Condor-pool based compiles, but it's always been a custom build
> flow. Makefile-driven, but with distinct subsystems and enforced rules for
> inter-subsystem dependencies which guaranteed I could parallelize the
> complies of most of the subsystems.
> I broke it down into what was, more or less, a DAG that had the following
> levels:
> 1. Pre-parallel compilation, done on one single machine where make could do
> it's local parallelization
> 2. Fully parallelized subsystem builds over multiple machines, no
> parallelism in make (or very minor, like maybe 2 or 3 threads)
> 3. A post-parallel compilation stage for some subsystems that had
> dependencies. There was Condor-level parallelism here but it was for data
> generation that went along with our builds, not for compiling code.
> 4. Any post-compilation linking
> 5. Testing
> 6. Publishing build
> I didn't actually using DAGMan to drive it. It was all driven from custom
> DAG-like tools because we felt there was too much conductor-type work that
> needed to be done, especially when we started to doing recovery from
> failures and such, that we could only do with custom logic.
> It was all done on a shared filesystem. Linux targeted builds. We never got
> it working for Windows.
> - Ian
> ________________________________
> Cycle Computing, LLC
> The Leader in Open Compute Solutions for Clouds, Servers, and Desktops
> Enterprise Condor Support and Management Tools
>
> http://www.cyclecomputing.com
> http://www.cyclecloud.com
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
>