[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] condor questions....

bruce wrote:


i'm new to the condor list, so be patient!!

i'm trying to get a better understanding of condor, and whether it can be
used in a project i'm considering. i'm looking at how to create a
distributed platform for creating a web crawling application.

the application will consist of ~1000-2000 perl scripts that are used to
parse a given web site. i'd like to have a small network of linux/windows
servers be used as client machines that actually run the perl apps, with the
management of the client servers/perl apps being handled by some
master/parent app/server. i'd like the ability to be able to have a basic
round-robin type of approach when determining how/when an app is run on a
given machine. (something along the lines of if a client machine is up, and
has the available resources (cpu/mem/etc..) then run the next app.)

seems like condor should work for this. you will need to have some sort of script/program which creates jobs for condor to handle.

i'd like to be able to integrate the management/operations of the
management/queuing app using a web based interface as well.

Condor is just a bunch of command line tools. I've made a WWW based interface which is targeted towards my use of Condor but i assume you can do the same.
The WWW interface will collect some data from the user and the WWW pages will create the needed text files and launch the condor_submit command.

i'm curious as to whether condor can easily handle the management task
without too much complexity, or is condor overkill for what i'm envisioning.

Condor is not too hard. I've got it working in a mixxed environment of Windows and Linux. File system / permissions will be the largest headahce if you have a mixxed OS situation. you have to have entries for users in samba and windows or just use very relaxed sambe security and you will need to disable condors authentication checking....