[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [condor-users] controlling number of jobs running at once
- Date: Tue, 2 Dec 2003 12:19:43 +0000 (GMT)
- From: Adam Mercer <ram@xxxxxxxxxxxxxxxxxx>
- Subject: Re: [condor-users] controlling number of jobs running at once
On Tue, 2 Dec 2003, Paul Wilson wrote:
> Here are two simple suggestions.
> You could submit 540 at once, but change the configuration file "maximum
> number of jobs to spawn shadows for" line to:
> MAX_JOBS_RUNNING = 10
> Then your schedd would only run 10 jobs at a time, even with 540 jobs in
> the queue. But, it would affect all users.
That would effect too many people.
> Alternatively, but requiring some pre-submission admin work:
> condor dagman can do this sequential submission for you.
> You'ld have to create 540 submit files, one for each run, then specify
> them in your condor_dag submit file and have something like:
> Parent a b c d e f g h i j Child k l m n o p q r s t u v.
> Parent w x y z aa bb cc dd ee ff gg hh ii jj Child kk ll mm nn oo ...
> and so on until you have specified 54x 10 Parent/Child job steps.
> This way, the next ten jobs would only be submitted to the schedd when
> the previous ten have finished.
> Also, you could use dagman's pre/post script feature to automate file
> admin you want to do between each ten jobs.
> Creating and editing the submit files should not take long- I presume
> all these jobs are similar.
This looks better, thanks I'll take a look.
> how any Mb's data is transfered/
each jobs needs around 1-1.5 Gb!
> what's the job length/
once the node has all the data it only takes a couple of minutes to
complete the analysis. this involves taking an fft and then some
multiplication - nothing that processor intensive yet.
> which universe/
> what's your network bandwidth?
its Gb ethernet, with 200 nodes all trying to get their relevant chunk of
data things slow right down!
Astrophysics & Space Research
School of Physics and Astronomy Tel: +44 (0)121 414 3721
University of Birmingham Fax: +44 (0)121 414 3722
B15 2TT, UK
Condor Support Information:
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>