Re: [HTCondor-users] Updating MEMORY without restarting HTCondor

[John M Knoeller <johnkn@xxxxxxxxxxx>]

The HTCondor STARTD will read the resource configuration and assign resources to slots when it starts up.  It does not re-read the resource configuration when it is re-configured.  Because it doesn’t have any way to “give back” resources, it will ignore changes to the resource configuration when it gets a condor_reconfig.

 

So when you change the MEMORY variable in the configuration, you MUST restart the STARTD in order to have that change take effect – condor_config_val will show you the changed value.  but the old value will be used (and condor_status will show the old value) until the STARTD is restarted.

 

If you want to reserve some memory for interactive processes that you have already given to the STARTD, you could use a dummy HTCondor job to “claim” the memory so that no real job will use it.  or you could use an interactive condor job.  (see condor_submit -interactive)

 

Or you could split your STARTD into 2 or more slots, and have one or more of the slots use a “desktop” policy, where it will kill the HTCondor job when it detects an interactive user on the machine.

 

a desktop policy looks something like this.  notice SLOTS_CONNECTED_TO_KEYBOARD

 

use POLICY:DESKTOP is

        WANT_VACATE=TRUE

        PREEMPT=($(KeyboardBusy) || ( (CpuBusyTime > 120) && $(ActivationTimer) > 90))

        START=((KeyboardIdle > $(StartIdleTime)) && ( $(CPUIdle) || (State != "Unclaimed" && State != "Owner")) )

        KILL=False

        MaxJobRetirementTime=0

        CLAIM_WORKLIFE=

        SLOTS_CONNECTED_TO_KEYBOARD=$(NUM_CPUS)

        IS_OWNER=(START =?= False)

 

-tj

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Vaurynovich, Siarhei
Sent: Monday, January 7, 2019 10:20 AM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Updating MEMORY without restarting HTCondor

 

 

Hello,

 

Here is the setup: a server is used for both to run HTCondor jobs and to run some interactive processes which require significant amount of RAM.

 

I found here ho to setup the amount of RAM available to HTCondor (i.e. the MEMORY variable):

 

https://www-auth.cs.wisc.edu/lists/htcondor-users/2015-June/msg00013.shtml

 

which I hope to combine with dynamic allocation of RAM:

 

https://lists.cs.wisc.edu/archive/htcondor-users/2016-March/msg00110.shtml

 

Question: once I change the configuration file to reflect the amount of RAM I want currently to be available to HTCondor on the server, how can I make HTCondor to get aware of the change without killing running jobs (I want to be able to do it several times per day)?

 

Running condor_reconfig does not seem to be enough: the new value of MEMORY is updated in the ‘condor_config_val -dump’ output, but condor_status still shows the old value of ‘Mem’ available in the ‘Unclaimed’ slot (I use dynamic slots, and so the unclaimed slot should reflect how much RAM HTCondor thinks is available to it to run jobs).

 

Thank you for your help,

Siarhei.

 

............................................................................

Trading instructions sent electronically to Bernstein shall not be deemed
accepted until a representative of Bernstein acknowledges receipt
electronically or by telephone.  Comments in this e-mail transmission and
any attachments are part of a larger body of investment analysis. For our
research reports, which contain information that may be used to support
investment decisions, and disclosures see our website at
www.bernsteinresearch.com.

For further important information about AllianceBernstein please click here
http://www.alliancebernstein.com/disclaimer/email/disclaimer.html