[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Condor-users] Dagman/submit is slow
- Date: Thu, 16 Jun 2005 05:35:22 -0500
- From: "Peter F. Couvares" <pfc@xxxxxxxxxxx>
- Subject: Re: [Condor-users] Dagman/submit is slow
It should be possible for us to speed this up, given certain important
If all Condor jobs in your DAG do not share a single userlog file, then
DAGMan must rely on the userlog event timestamps to correctly sequence
events when reading the logs to recover after a crash or reboot.
Because the Condor userlog timestamps have one-second resolution,
DAGMan submits at most one job per second to ensure that the timestamps
can be sorted properly.
However, if you use only one userlog DAGMan could rely solely on the
order of events in the log, and submit as fast as condor_submit can be
called for each job (which may be somewhat faster than once per
I will try to add this as an automatic optimization in the
single-userlog case, and let you know when it's available.
On Jun 13, 2005, at 4:19 AM, Cargnelli, Matthieu wrote:
I'm trying to use condor to distrbute jobs within a cluster. I'm using
dagman to describe the 1200 tasks I wish to execute in parallel. It
takes about 10 minutes to have a hundred of these tasks running. I have
set the number of jobs the dagman can submit per interval to its
: 1000, and the interval to 0. But it submits only one or two tasks
second. It's sufficient to feed the procs I use for now, but I plan on
using hundreds of processors soon, so I'd like to know if there's a way
to speed things up.
The datafile I'm dealing with is approximately 2MB, and the network is
one GB. From my experience in transmitting files over that network it
mustn't take more than 0.2 s to be passed between the master and the
The nodes are configured to accept any work, never to preempt their
and are dedicated, do they are never interrupted by keypress or mouse.
The universe I use is vanilla.
Thanks in advance for your help.
Peter Couvares University of Wisconsin-Madison
Condor Project Research Department of Computer Sciences
pfc@xxxxxxxxxxx 1210 W. Dayton St. Rm #4241
(608) 265-8936 Madison, WI 53706-1685