[Condor-users] condor questions....

["bruce" <bedouglas@xxxxxxxxxxxxx>]

hi...

i'm new to the condor list, so be patient!!

i'm trying to get a better understanding of condor, and whether it can be
used in a project i'm considering. i'm looking at how to create a
distributed platform for creating a web crawling application.

the application will consist of ~1000-2000 perl scripts that are used to
parse a given web site. i'd like to have a small network of linux/windows
servers be used as client machines that actually run the perl apps, with the
management of the client servers/perl apps being handled by some
master/parent app/server. i'd like the ability to be able to have a basic
round-robin type of approach when determining how/when an app is run on a
given machine. (something along the lines of if a client machine is up, and
has the available resources (cpu/mem/etc..) then run the next app.)

i'd like to be able to integrate the management/operations of the
management/queuing app using a web based interface as well.

i'm curious as to whether condor can easily handle the management task
without too much complexity, or is condor overkill for what i'm envisioning.

if condor is overkill, do you know of any other app that might be
used/modified to accommodate what i've described?

thanks for any thoughts/comments/criticisms...

regards,

-bruce

ps. assuming condor can work, how easy/difficult is it to get condor
up/running!!!!