[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] condor questions....
- Date: Tue, 05 Oct 2004 09:31:01 -0700
- From: John Wheez <john@xxxxxxxxxx>
- Subject: Re: [Condor-users] condor questions....
hi...seems like condor should work for this. you will need to have some sort
of script/program which creates jobs for condor to handle.
i'm new to the condor list, so be patient!!
i'm trying to get a better understanding of condor, and whether it can be
used in a project i'm considering. i'm looking at how to create a
distributed platform for creating a web crawling application.
the application will consist of ~1000-2000 perl scripts that are used to
parse a given web site. i'd like to have a small network of linux/windows
servers be used as client machines that actually run the perl apps, with the
management of the client servers/perl apps being handled by some
master/parent app/server. i'd like the ability to be able to have a basic
round-robin type of approach when determining how/when an app is run on a
given machine. (something along the lines of if a client machine is up, and
has the available resources (cpu/mem/etc..) then run the next app.)
Condor is just a bunch of command line tools. I've made a WWW based
interface which is targeted towards my use of Condor but i assume you
can do the same.
i'd like to be able to integrate the management/operations of the
management/queuing app using a web based interface as well.
The WWW interface will collect some data from the user and the WWW pages
will create the needed text files and launch the condor_submit command.
i'm curious as to whether condor can easily handle the management taskCondor is not too hard. I've got it working in a mixxed environment of
Windows and Linux. File system / permissions will be the largest
headahce if you have a mixxed OS situation. you have to have entries for
users in samba and windows or just use very relaxed sambe security and
you will need to disable condors authentication checking....
without too much complexity, or is condor overkill for what i'm envisioning.