Re: [HTCondor-users] Machine allocation - unused machines

[John M Knoeller <johnkn@xxxxxxxxxxx>]

Having your jobs request more memory will probably help yes.

-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Peter Ellevseth
Sent: Wednesday, January 13, 2021 11:16 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Machine allocation - unused machines

Hi

I see that the one single machine running all my jobs at the moment is using swap memory, slowing down execution. I am wondering if I can solve this by setting requestmemory in my submit file? Will this tell the negotiator to spread the jobs around?

P

-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of John M Knoeller
Sent: onsdag 13. januar 2021 16.42
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Machine allocation - unused machines

> when I submit a batch of jobs, condor will use one machine to full 
> capacity,

This is a normal consequence of the way negotiation works when the execute nodes are configured  to use PartitionableSlots. 

You can change this behavior, at the cost of slowing down the initial job startup by telling the schedd that it should only start one job per Partitionable slot per negotiation cycle.  You do this by configuring 

CLAIM_PARTITIONABLE_LEFTOVERS = false

In the schedd.  This knob defaults to true.

-tj


-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Peter Ellevseth
Sent: Wednesday, January 13, 2021 7:47 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Machine allocation - unused machines

Carsten

Thank you for the quick response.

I was unaware of this macro. As far as I could see, there was no mention of that macro in any of my config-files. I am running version 8.6.

I tried setting that macro to false on my negotiator, so we'll see what happens.

P

-----Original Message-----
From: Carsten Aulbert <carsten.aulbert@xxxxxxxxxx>
Sent: onsdag 13. januar 2021 12.36
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>; Peter Ellevseth <Peter.Ellevseth@xxxxxxxxxx>
Subject: Re: [HTCondor-users] Machine allocation - unused machines

Hi Peter,

On 13.01.21 12:29, Peter Ellevseth wrote:
> I have a question, perhaps mis-informed, but nonetheless. We run a 
> small cluster (approx. 70 cores) for doing CFD and other small jobs. I 
> see that when I submit a batch of jobs, condor will use one machine to 
> full capacity, leaving the other machines in the cluster alone. 
> Wouldn't it be beneficial to spread the jobs out, with regard to use of memory etc?
> Is there perhaps some configuration I can set to enable this?

I think this should be the domain of NEGOTIATOR_DEPTH_FIRST[1] also briefly discussed in the wiki[2].

However, I wonder why the non-default depth first seems to be in effect at your pool.

Cheers

Carsten

[1]
https://htcondor.readthedocs.io/en/latest/admin-manual/configuration-macros.html#NEGOTIATOR_DEPTH_FIRST

[2]
https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=HowToFillPoolDepthFirst


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/