Re: [HTCondor-users] why does htcondor change sysctl params, and why is this done outside of /etc/sysctl.{d, conf} ?

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 1/14/2016 5:06 AM, SCHAER Frederic wrote:
> 
> Unfortunately, my monitoring is looking at all processes between 1 and 
> pid_max… and this setting is causing useless load on servers – and it is 
> killing some VMs which I use for dev.
> 
> If htcondor wants to modify such  system settings, could this please be 
> done in a reversible/documented/standard way ?
> 

As if HTCondor v8.3.8, the condor_master daemon now runs a script when it starts up. This script tunes several Linux kernel parameters to the values we suggest for better scalability. New configuration variables ENABLE_KERNEL_TUNING, KERNEL_TUNING_LOG, and LINUX_KERNEL_TUNING_SCRIPT enable the use of the script and specify file locations. All of these config knobs are documented in the Manual.  If you want to disable this functionality, put
   ENABLE_KERNEL_TUNING = False
in your config.

The addition of this mechanism also was documented in the Version History.  For folks who are not interested in following the detailed version history, at the start of each new stable series we distill it down to important changes admins may want to know when upgrading; this mechanism is also documented in this much shorted list at
 http://htcondor.org/manual/v8.4/10_2Upgrading_from.html

We purposefully made this mechanism a simple transparent script (as opposed to performing these steps in the binary code) so it would be easy for admins to look at or change any detail of what is being done.

As for why not use /etc/sysctl.d, my understanding is unfortunately it does not exist/work on all the Linux distros we need to support (fwiw, it does not exist on the RHEL machine I am using right now).  I encourage people to convince the LSB (Linux Standards Base) to make it required.  As for why not edit /etc/sysctl.conf upon installation, packaging standards for several distros discourage (or flat out disallow) one package to change the configuration file for a different service, and in any event changes HTCondor would make to this file would be typically rewritten by many sites via configuration automation tools like puppet / chef / cfengine. Perhaps we could modify the tuning script to at least append to the sysctl.conf file a set of comments that tells the admin about the existence of the ENABLE_KERNEL_TUNING script -- this would hopefully be OK with packaging rules (since it is just comments), and yet perhaps would've helped you figure out what was going on more quick!
 ly....  
do you like this idea? open to suggestions...

Also, by using an ENABLE_KERNEL_TUNING script we can go beyond just static sysctl settings to do things like tune differently on submit hosts vs the central manager, or things like inspecting the machine to see if cgroups can be safely utilized and if so, performing the necessary steps so HTCondor can leverage it.  

The motivation for adding this mechanism in the first place was the list of admin "things to do" to install and tune HTCondor kept growing and growing.  Especially on busy HTCondor submit machines which are managing many thousands of jobs or on central managers handling tens of thousands of slots, users, admin, and HTCondor developers have spent much time troubleshooting problems that ultimately boiled down discovering (or re-discovering) all the various system configuration/tuning that is required to scale to large sizes, or to enable non-default kernel services like cgroups, etc, etc.  Even when we documented what to do, usually much time was wasted by admins and developers figuring out that the problem could be solved if they followed the instructions on page 790 of the Manual.  It is a balancing act, but with the ENABLE_KERNEL_TUNING mechanism we trying to lean a bit more towards the "if you install it, it will work" out of the box.

hope this helps explain the thinking,
Todd