[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Condor-users] Spawning jobs depending on system load and start_delay, start_count
- Date: Tue, 03 Nov 2009 16:28:48 +0100
- From: Heiko Burghardt <hb-tech@xxxxxxxxx>
- Subject: [Condor-users] Spawning jobs depending on system load and start_delay, start_count
I just started studying the documentation of condor and did some tests
with it. Condor is very complex and flexible as far as I can see at my
stage of knowledge about it but I still have not found on how I can
configure Condor to solve a problem for us. So, I hope anyone on this
list may help me.
We have a partner we are sending messages to. For each message we send
we normally receive between 3 and 10 messages in return after some time.
Finally, we talk about thousands of messages a day.
The sending and receiving part is done by scripts that connect to a
database engine, protocol the message and to find references for further
processing. Sometimes we are sending /receiving so many messages that
the load of the system explodes rapidly, only because of too many
That's why we like to establish Condor as queuing mechanism because it
enables us to rectify the mass of jobs for a better throughput on the
system. But until now I was only able to find out on how we can start a
number of jobs in a certain time frame. To do so I use the local
universe for the jobs and the following parameters in the configuration
JOB_START_DELAY = 1
JOB_START_COUNT = 5
Actually, this works fine for queuing the jobs in general. But we like
to combine this feature with a load dependency rule in Condor, means:
stop to run jobs at a load higher than value X. That's where I fail as
the UWCS_START, UWCS_SUSPEND, ... rules do not seem to work in the local
universe. Using e.g. the scheduler universe the situation is visa verce:
load dependency works (with some restrictions) but there is no
restriction in number of jobs in a time frame.
Does somebody have an idea on how to set up Condor for our requirements?
Thanks in advance.
Riege Software International GmbH Fon: +49 (2159) 9148 0
Mollsfeld 10 Fax: +49 (2159) 9148 11
40670 Meerbusch Web: www.riege.com
Germany E-Mail: hb-tech@xxxxxxxxx
Handelsregister: Managing Directors:
Amtsgericht Neuss HRB-NR 4207 Christian Riege
USt-ID-Nr.: DE120585842 Gabriele Riege
YOU CARE FOR FREIGHT, WE CARE FOR YOU