Re: [condor-users] Submitting Variable Number of Jobs

[Erik Paulson <epaulson@xxxxxxxxxxx>]

On Thu, Sep 04, 2003 at 07:24:55PM +0100, Alexander Klyubin wrote:
> Hello!
> 
> 
> What I'd like to do with Condor is following. I submit a numbre of jobs 
> into a single cluster. Each time a job finishes a script gets executed. 
> This script should be able to submit new jobs to this cluster if it 
> decides to do so. This kind of scenario is usually encountered in 
> parallel searches, e.g., GAs. For example, in GA I'd like to evaluate 
> populations and then submit new runs based on the results of preceding 
> ones. I don't want to submit these new jobs to new clusters because it 
> gets messy to manage them. It is much better when they all are under one 
> "hood".
> 
> Is this scenario possible with Condor?

Not presently - there's no way to add a job to an existing cluster of
jobs. 

>  As far as I can see, at the moment DAGMan supports only fixed number of 
> jobs. 

That's a bit different. DAGMan requires that each job be in a seperate
cluster, and isn't likely to drop that requirement in the near future.
DAGMan also doesn't allow you to dynamically add jobs to the DAG (or 
dynamically change the DAG at all) but that is likely to change. (Each job
will still have to be in a different cluster)

> At the same time I don't think variable number of jobs goes against the 
> principles behind Condor.

I'd agree with that, but the current implementation makes it difficult to
support. 

Internally, clusters and procs have already mostly gone away. They're 
certainly very useful, especially with commands like 'condor_rm', that can
take either a <cluster.proc> pair or just a cluster number. For example,
if cluster 97 has 120 jobs, you can remove all 120 jobs just by going 
`condor_rm 97`

With the -constraint option to condor_rm, you can do the same thing:

`condor_rm -constraint ClusterId==97`

And you can do fancier things - if you stick in custom attributes in your
submit file, you can reference them later with the -constraint:

universe=vanilla
executable=/bin/foo
+color="blue"
queue

you can do a 

condor_rm -constraint color==\"blue\"

With that, you can add submit as many jobs as you want with color=="blue",
and use that constraint to manipulate the queue easily.

-Erik
Condor Support Information:
http://www.cs.wisc.edu/condor/condor-support/
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>