[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] condor speed processing question..



Bruce,

>From my understanding Condor is not designed to run many fast/short
jobs, but more aimed at running longer jobs, where the loss of a few
minutes in between jobs finishing and jobs starting is not a real
concern.

As you as testing condor at the moment you may be using very short test
jobs that run and finish and then leave the computers idle for a minute
or two, between Condor renegotiating a new job to be run on the
computer.

In another respect if you have a single CPU and multiple similar tasks,
then it is likely that running one at a time is one of the most
efficient ways to execute each of the jobs because it prevents context
switching (including memory structures). If these tasks are very small
then you could wrap them up in a script and sequentially process the
scripts. Again running the tasks sequentially in the script may be the
best approach.

Finally there is a parameter somewhere in the config files that allows
you to change how often jobs are put onto the clients. Alternatively you
may be looking for a high performance computing solution, rather than a
high throughput computing solution which is what condor is.

Hope this helps, particularly as I am not an expert, 

Peter

Dr Peter Myerscough-Jackopson  -  Engineer
MULTIPLE ACCESS COMMUNICATIONS LIMITED
Delta House, The University of Southampton Science Park, Southampton,
SO16 7NS,
United Kingdom.
Tel: +44 (0)23 8076 7808 Fax: +44 (0)23 8076 0602
Web: http://www.macltd.com/  Email:
peter.myerscough-jackopson@xxxxxxxxxx

-----Original Message-----
From: condor-users-bounces@xxxxxxxxxxx
[mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of bruce
Sent: 08 January 2007 05:41
To: 'Condor-Users Mail List'
Subject: Re: [Condor-users] condor speed processing question..

update...

upon further tests, it appears that even though the information from
'condor_q' indicates the multiple instances are running in a parallel
manner, it looks as though only one (maybe two) real instances are being
processed simultaneously.

i did a continual 'condor_q' keeping track of the number of jobs that
were still left. i expected to see this reduce by more than the 1/2 per
step that i noticed. the test app that the condor process is running is
a pretty simple process which creates/writes to a new file.

so... my question. is there a way/process to get condor to fire off/run
10-20 copies at the 'same' time!

thanks

-bruce



-----Original Message-----
From: condor-users-bounces@xxxxxxxxxxx
[mailto:condor-users-bounces@xxxxxxxxxxx]On Behalf Of bruce
Sent: Sunday, January 07, 2007 8:59 PM
To: 'Condor-Users Mail List'
Subject: [Condor-users] condor speed processing question..


hi.

i have a test condor node running. i'm trying to get it to create/run as
many child processes as possible. i'm currently running the config using
the 'testing' attributes, and experimenting with changing the "num_cpus"
parameter.

it appears that if the 'num_cpus' param is modified to be '100' condor
doesn't run at all, whereas if i set it to '20' condor runs...

is there another approach to use, to try to get condor to run more
simultaneous processes.

or would i have to tweak the source to ignore certain attributes.

thanks

-bruce

_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with
a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at either
https://lists.cs.wisc.edu/archive/condor-users/
http://www.opencondor.org/spaces/viewmailarchive.action?key=CONDOR

_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with
a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at either
https://lists.cs.wisc.edu/archive/condor-users/
http://www.opencondor.org/spaces/viewmailarchive.action?key=CONDOR