[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] HTCondor concurrency limit



Hmmm… I should possibly add that I am running with partitionable slots.

 

 

M

 

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Marc Volovic
Sent: Monday, August 11, 2014 10:51 AM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] HTCondor concurrency limit

 

Let's review this, again:

 

marcv@htcondor[~]$ condor_config_val -dump | grep LIMIT

CONCURRENCY_LIMIT_DEFAULT = 30

MARCV_LIMIT = 1

 

This should, to the best if my understanding, allow me to run no more than one job at any given one time.

 

Note that the schedulers and the executors have

 

ConcurrencyLimits = Owner

STARTD_ATTRS = ConcurrencyLimits

SUBMIT_ATTRS = ConcurrencyLimits

 

Yet – when I submit 10 jobs, after a short while (a few minutes) – I have more than one running. Eventually, all 10 will run. What am I doing incorrectly?

 

To wit:

 

marcv@s1[~]$ condor_q -g | grep marcv

marcv@s1[~]$ condor_q -r | grep marcv

marcv@s1 [~]$ condor_userprio | grep marcv

marcv@ezchip.com              0.59      1.00      0        15.51    0+00:01

marcv@s1[~]$ condor_q

 

-- Submitter: s1.ezchip.com : <1.2.3.4:42660> : s1.ezchip.com

ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD

 

0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended

 

marcv@s1[~]$ condor_run -a 'request_disk=10' -a 'request_memory=10' sleep 1d &

[1] 41818

marcv@s1[~]$ condor_run -a 'request_disk=10' -a 'request_memory=10' sleep 1d &

[2] 42013

marcv@s1[~]$ condor_run -a 'request_disk=10' -a 'request_memory=10' sleep 1d &

[3] 42180

marcv@s1[~]$ condor_run -a 'request_disk=10' -a 'request_memory=10' sleep 1d &

[4] 42463

marcv@s1[~]$ condor_run -a 'request_disk=10' -a 'request_memory=10' sleep 1d &

[5] 42779

marcv@s1[~]$ condor_run -a 'request_disk=10' -a 'request_memory=10' sleep 1d &

[6] 43033

marcv@s1[~]$ condor_run -a 'request_disk=10' -a 'request_memory=10' sleep 1d &

[7] 43390

marcv@s1[~]$ condor_run -a 'request_disk=10' -a 'request_memory=10' sleep 1d &

[8] 43590

marcv@s1[~]$ condor_run -a 'request_disk=10' -a 'request_memory=10' sleep 1d &

[9] 43736

marcv@s1[~]$ condor_run -a 'request_disk=10' -a 'request_memory=10' sleep 1d &

[10] 43938

 

marcv@s1[~]$ condor_q

 

 

-- Submitter: s1.ezchip.com : <1.2.3.4:42660> : s1.ezchip.com

ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD

110.0   marcv           8/11 10:44   0+00:01:09 R  0   0.0  .condor_run.41818

111.0   marcv           8/11 10:44   0+00:00:49 R  0   0.0  .condor_run.42013

112.0   marcv           8/11 10:44   0+00:00:29 R  0   0.0  .condor_run.42180

113.0   marcv           8/11 10:44   0+00:00:08 R  0   0.0  .condor_run.42463

114.0   marcv           8/11 10:44   0+00:00:00 I  0   0.0  .condor_run.42779

115.0   marcv           8/11 10:44   0+00:00:00 I  0   0.0  .condor_run.43033

116.0   marcv           8/11 10:44   0+00:00:00 I  0   0.0  .condor_run.43390

117.0   marcv           8/11 10:44   0+00:00:00 I  0   0.0  .condor_run.43590

118.0   marcv           8/11 10:44   0+00:00:00 I  0   0.0  .condor_run.43736

119.0   marcv           8/11 10:44   0+00:00:00 I  0   0.0  .condor_run.43938

 

10 jobs; 0 completed, 0 removed, 6 idle, 4 running, 0 held, 0 suspended

 

 

 

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Marc Volovic
Sent: Monday, August 11, 2014 10:18 AM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] HTCondor concurrency limit

 

J

 

Thanks. I am acquainted with his page. That's the problem – I set up everything according to what I seem to read there, set my own personal concurrency limit to 1 and – lo and behold – can run a gazillion jobs.

 

M

 

 

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Miha Ahronovitz
Sent: Sunday, August 10, 2014 7:46 PM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] HTCondor concurrency limit

 

Ok... Maybe I do not understand the concurrency.

 

You are not the only one :-(  Most of the people on this thread do not understand it, but they are afraid to admit it

 

A blogger decided to publish his own discoveries Concurrency Limits: Protecting shared resources

 

See if it inspires you.

 

M

 

--- --- --- --- --- --- --- --- --- --- --- --- ---

Miha Ahronovitz

Principal Ahrono Associates

Web: http://www.ahrono.com/

Blog: http://my-inner-voice.blogspot.com/

c: 408 422 2757

emiha.ahronovitz@xxxxxxxxxx

tw: @myinnervoice

--- --- --- --- --- --- --- --- --- --- --- --- ---

 

 

On Sun, Aug 10, 2014 at 4:55 AM, Marc Volovic <marcv@xxxxxxxxxx> wrote:

Hello, sorry for the latish response - we were on a weekend :-)...

Concurrency limits are indeed listed in negotiator log and in jobs.

As a test, I set my own concurrency to 1 and started initiating condor_run 'sleep 10000' jobs.

Slowly, bit by bit, there came more and more of them running. Not what I wants :(....

Ok... Maybe I do not understand the concurrency.

Assume I am the only user of the whole cluster. I want no more than 2 of my jobs running at any given time, no matter how many free slots there are. Should I set 'marcv_limit = 2' and set SCHED_ATTRS = Owner, etc, or should I do something more violent?

Many thanks in advance,

marc




-----Original Message-----
From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Marc Volovic
Sent: Friday, August 08, 2014 7:29 AM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] HTCondor concurrency limit

I did a condor_reconfig -full -all and am pretty sure limits are in the ads. Will check logs.

The annoying bit is that a test cluster works, the production one does not.


Sent from my iPad

> On 7 Aug 2014, at 22:36, "Ben Cotton" <ben.cotton@xxxxxxxxxxxxxxxxxx> wrote:
>
> Marc,
>
> I was able to enforce limits with a configuration as you described. I
> assume you did a condor_reconfig on both your central manager and your
> scheduler? Do the concurrency limits appear in the job ad? Does
> "Concurrency Limit" appear in your negotiator log?
>
>
> Thanks,
> BC
>
> --
> Ben Cotton
> main: 888.292.5320
>
> Cycle Computing
> Leader in Utility HPC Software
>
> http://www.cyclecomputing.com
> twitter: @cyclecomputing
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/