Re: [Condor-users] Quill Question

[Steven Timm <timm@xxxxxxxx>]

On Tue, 17 Feb 2009, Hoover Sam wrote:

> I'm curious, what is the highest number of jobs that Quill and Condor
> can handle gracefully?
>
> We've had up to 300,000 jobs on 2500 nodes, but 500,000 jobs broke
> things.
>
> Sam

There are many variables as to how many jobs condor can handle.
Quill is not always the limiting factor.  The start rate is
every bit as important as the total number of jobs.
I have personally successfully run as many as 13000 batch slots
advertising to a single collector, running only sleeper jobs however,
and had as many as 6K simultaneous running jobs from a single schedd.
Submission rate is just as important as the raw number of jobs, however.

The difficulty of quill is that the default schema doesn't have
very many indices so if you try to put a constraint on something
that isn't indexed, you are basically just trying to read the DB like
a sequential file, actually it is a bit slower than reading a sequential 
file, and it doesn't start spewing out data until the whole postgres
query has been buffered and processed.  Also there have been
discontinuities in the schema from time to time, or bugs in the program
that unloads the old schema and loads the new, that have forced
us to drop our DB over condor upgrades and start from scratch.

Steve

> On Feb 17, 2009, at 10:48 AM, Henning Fehrmann wrote:
>
> > Hi Matt,
> > > How well has Quill scaled for you in this configuration with
> > > thousands
> > > of execute nodes?
> >
> > We tried Quill on PostgreSQL 8.3 database
> > with 1680 execute nodes, appr. 30 user and sometimes several 10000
> > jobs in
> > the queue. From time to time the access to the database took almost a
> > minute. Even a direct access to the database took us 10s to list a
> > table
> > with 600 000 lines.
> > It was condor v7.0.1. We have not tried it yet with v7.2.0.
> > We also assume that the database can be tuned.
> >
> > Cheers,
> > Henning
> >
> > _______________________________________________
> > Condor-users mailing list
> > To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
> > with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/condor-users/
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/

--
------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525
timm@xxxxxxxx  http://home.fnal.gov/~timm/
Fermilab Computing Division, Scientific Computing Facilities,
Grid Facilities Department, FermiGrid Services Group, Assistant Group Leader.