Re: [condor-users] controlling number of jobs running at once

[Adam Mercer <ram@xxxxxxxxxxxxxxxxxx>]

On Tue, 2 Dec 2003, Paul Wilson wrote:

> Here are two simple suggestions.
>
> You could submit 540 at once, but change the configuration file "maximum
> number of jobs to spawn shadows for" line to:
>
> MAX_JOBS_RUNNING = 10
>
> Then your schedd would only run 10 jobs at a time, even with 540 jobs in
> the queue. But, it would affect all users.

That would effect too many people.

> Alternatively, but requiring some pre-submission admin work:
> condor dagman can do this sequential submission for you.
> You'ld have to create 540 submit files, one for each run, then specify
> them in your condor_dag submit file and have something like:
>
> Parent a b c d e f g h i j Child k l m n o p q r s t u v.
> Parent w x y z aa bb cc dd ee ff gg hh ii jj Child kk ll mm nn oo ...
> and so on until you have specified 54x 10 Parent/Child job steps.
>
> This way, the next ten jobs would only be submitted to the schedd when
> the previous ten have finished.
> Also, you could use dagman's pre/post script feature to automate file
> admin you want to do between each ten jobs.
>
> Creating and editing the submit files should not take long- I presume
> all these jobs are similar.

This looks better, thanks I'll take a look.

> BTW:
> how any Mb's data is transfered/

each jobs needs around 1-1.5 Gb!

> what's the job length/

once the node has all the data it only takes a couple of minutes to
complete the analysis. this involves taking an fft and then some
multiplication - nothing that processor intensive yet.

> which universe/

vanilla

> what's your network bandwidth?

its Gb ethernet, with 200 nodes all trying to get their relevant chunk of
data things slow right down!

Cheers

Adam

-- 
 Adam Mercer
 Astrophysics & Space Research
 School of Physics and Astronomy                Tel: +44 (0)121 414 3721
 University of Birmingham                       Fax: +44 (0)121 414 3722
 B15 2TT, UK

Condor Support Information:
http://www.cs.wisc.edu/condor/condor-support/
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>