Re: [condor-users] controlling number of jobs running at once

[Paul Wilson <p.b.wilson@xxxxxxxxx>]

Here are two simple suggestions.

You could submit 540 at once, but change the configuration file "maximum
number of jobs to spawn shadows for" line to:

MAX_JOBS_RUNNING = 10

Then your schedd would only run 10 jobs at a time, even with 540 jobs in
the queue. But, it would affect all users.

Alternatively, but requiring some pre-submission admin work: 
condor dagman can do this sequential submission for you.
You'ld have to create 540 submit files, one for each run, then specify
them in your condor_dag submit file and have something like:

Parent a b c d e f g h i j Child k l m n o p q r s t u v.
Parent w x y z aa bb cc dd ee ff gg hh ii jj Child kk ll mm nn oo ...
and so on until you have specified 54x 10 Parent/Child job steps.

This way, the next ten jobs would only be submitted to the schedd when
the previous ten have finished.
Also, you could use dagman's pre/post script feature to automate file
admin you want to do between each ten jobs.

Creating and editing the submit files should not take long- I presume
all these jobs are similar.

BTW:
how any Mb's data is transfered/what's the job length/which
universe/what's your network bandwidth?

Paul Wilson,
UCL Condor Pool. (900 Win2k nodes)

On Tue, 2003-12-02 at 11:30, Adam Mercer wrote:
> Hi
> 
> I've got a series of jobs that I need to run, around 540, that require a
> very large amount of data and don't take long to process. If I submit them
> all at once the network bandwidth is maxed out. Is there anyway to specify
> in the submit file to only run say 10 jobs at a time. I know I can specify
> in the requirements to only run on specified machines, but can I say to
> only run on 10 nodes (any that are availble).
> 
> Cheers
> 
> Adam
> 
> -- 
>  Adam Mercer
>  Astrophysics & Space Research
>  School of Physics and Astronomy                Tel: +44 (0)121 414 3721
>  University of Birmingham                       Fax: +44 (0)121 414 3722
>  B15 2TT, UK
> 
> Condor Support Information:
> http://www.cs.wisc.edu/condor/condor-support/
> To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
> unsubscribe condor-users <your_email_address>
> 
> 


Condor Support Information:
http://www.cs.wisc.edu/condor/condor-support/
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>