Re: [Condor-users] Max number of nodes on a condor cluster

[Ian Chesal <ichesal@xxxxxxxxxxxxxxxxxx>]

On Tuesday, 16 August, 2011 at 9:39 AM, Stian Mikelsen wrote:

Hi, all.

 

I am comparing Condor to LSF and Torque, and just wanted to know how how many nodes a Condor cluster can handle? Would for instance 100k nodes be a problem?

That's a somewhat apples-to-oranges comparison, but okay.

A 100,000 execute slots is definitely a very, very big pool. I'm doubtful you could make this work as a single pool. But it is certainly possible to link multiple pool together to make this kind of capacity available -- the decentralized approach to grid computing. In some ways, this is what FNAL and others are doing over OSG and TeraGrid. You can search through some past CondorWeek presentations for some talks on scalability from Igor Sfiligoi, Dennis Box and others for details on how they do this: http://www.cs.wisc.edu/condor/past_condor_weeks.html

There's an interesting paper by Wolinsky et al. on decentralized grid architectures here: http://www.davidwolinsky.com/static/files/DecentralizedGrids.pdf -- that references some work by The Condor Group at U.Wisc. Unfortunately no published date on that paper. It has some upper bound numbers in it, though I can't say how accurate they are.

Looking that current public pool data I have, the biggest single pool is ~11k slots.

I'd doubt even *GE or LSF could do a 100,000 execution slots in a *single* pool. That's the realm of distributed, loosely coupled pools, no matter what technology is driving the scheduling of jobs. I am, however, often wrong. :)

Regards,

- Ian

---

Ian Chesal

Cycle Computing, LLC

Leader in Open Compute Solutions for Clouds, Servers, and Desktops

Enterprise Condor Support and Management Tools

http://www.cyclecomputing.com

http://www.cyclecloud.com

http://twitter.com/cyclecomputing 

 

Best Regards

Stian Mikelsen

-------------------------------------------------------------------
The information contained in this message may be CONFIDENTIAL and is
intended for the addressee only. Any unauthorised use, dissemination of the
information or copying of this message is prohibited. If you are not the
addressee, please notify the sender immediately by return e-mail and delete
this message.
Thank you

_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/