[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [condor-users] controlling number of jobs running at once
- Date: 02 Dec 2003 12:08:48 +0000
- From: Paul Wilson <p.b.wilson@xxxxxxxxx>
- Subject: Re: [condor-users] controlling number of jobs running at once
Here are two simple suggestions.
You could submit 540 at once, but change the configuration file "maximum
number of jobs to spawn shadows for" line to:
MAX_JOBS_RUNNING = 10
Then your schedd would only run 10 jobs at a time, even with 540 jobs in
the queue. But, it would affect all users.
Alternatively, but requiring some pre-submission admin work:
condor dagman can do this sequential submission for you.
You'ld have to create 540 submit files, one for each run, then specify
them in your condor_dag submit file and have something like:
Parent a b c d e f g h i j Child k l m n o p q r s t u v.
Parent w x y z aa bb cc dd ee ff gg hh ii jj Child kk ll mm nn oo ...
and so on until you have specified 54x 10 Parent/Child job steps.
This way, the next ten jobs would only be submitted to the schedd when
the previous ten have finished.
Also, you could use dagman's pre/post script feature to automate file
admin you want to do between each ten jobs.
Creating and editing the submit files should not take long- I presume
all these jobs are similar.
how any Mb's data is transfered/what's the job length/which
universe/what's your network bandwidth?
UCL Condor Pool. (900 Win2k nodes)
On Tue, 2003-12-02 at 11:30, Adam Mercer wrote:
> I've got a series of jobs that I need to run, around 540, that require a
> very large amount of data and don't take long to process. If I submit them
> all at once the network bandwidth is maxed out. Is there anyway to specify
> in the submit file to only run say 10 jobs at a time. I know I can specify
> in the requirements to only run on specified machines, but can I say to
> only run on 10 nodes (any that are availble).
> Adam Mercer
> Astrophysics & Space Research
> School of Physics and Astronomy Tel: +44 (0)121 414 3721
> University of Birmingham Fax: +44 (0)121 414 3722
> B15 2TT, UK
> Condor Support Information:
> To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
> unsubscribe condor-users <your_email_address>
Condor Support Information:
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>