Re: [Condor-users] condor questions....

[John Wheez <john@xxxxxxxxxx>]

bruce wrote:

> hi...
>
> i'm new to the condor list, so be patient!!
>
> i'm trying to get a better understanding of condor, and whether it can be
> used in a project i'm considering. i'm looking at how to create a
> distributed platform for creating a web crawling application.
>
> the application will consist of ~1000-2000 perl scripts that are used to
> parse a given web site. i'd like to have a small network of linux/windows
> servers be used as client machines that actually run the perl apps, with the
> management of the client servers/perl apps being handled by some
> master/parent app/server. i'd like the ability to be able to have a basic
> round-robin type of approach when determining how/when an app is run on a
> given machine. (something along the lines of if a client machine is up, and
> has the available resources (cpu/mem/etc..) then run the next app.)
>
>  
seems like condor should work for this. you will need to have some sort 
of script/program which creates jobs for condor to handle.

> i'd like to be able to integrate the management/operations of the
> management/queuing app using a web based interface as well.
>
>  
Condor is just a bunch of command line tools. I've made a WWW based 
interface which is targeted towards my use of Condor but i assume you 
can do the same.
The WWW interface will collect some data from the user and the WWW pages 
will create the needed text files and launch the condor_submit command.


> i'm curious as to whether condor can easily handle the management task
> without too much complexity, or is condor overkill for what i'm envisioning.
>  
Condor is not too hard. I've got it working in a mixxed environment of 
Windows and Linux. File system / permissions will be the largest 
headahce if you have a mixxed OS situation. you have to have entries for 
users in samba and windows or just use very relaxed sambe security and 
you will need to disable condors authentication checking....