Re: [HTCondor-users] "include command : " in configuration files

[John M Knoeller <johnkn@xxxxxxxxxxx>]

> If I go to a worker node (startd), as admin, and call "condor_config_val -startd -dump", I immediately see the new value created from an included script

The startd will not load a new config unless something is *else* is forcing the startd to reconfig.   condor_config_val will not force a reconfig, and "condor_config_val -startd -dump"  will cause condor_config_val print values from the startd's (possibly stale) config.

HOWEVER!  each time you run a tool, like condor_q, condor_submit, or condor_config_val.  the tool will parse the config and run any scripts as part of the tool initialization.  (this is needed so that the tools can find the collector, etc).

So a config script WILL run as part of "condor_config_val -startd -dump",  but the output of condor_config_val will be the values it got from the startd, not the values it got by parsing the config itself.

-tj



-----Original Message-----
From: Oliver Freyermuth [mailto:freyermuth@xxxxxxxxxxxxxxxxxx] 
Sent: Monday, November 6, 2017 6:57 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>; John M Knoeller <johnkn@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] "include command : " in configuration files

Dear John, 

Am 06.11.2017 um 23:58 schrieb John M Knoeller:
> The problem here is that you should not let just anyone make the cached config files.  
> You should instead make sure that ONLY a HTCondor daemon creates that cache file, all other config readers should just use the cache file.
>
> The reason for this your config files provide a way to get HTCondor daemons (probably running as root!) to run arbitrary code.   This means
> that if any of your config files (including the cache files) is writable by non-root then you have created a root exploit.

Agreed. That's why I would prefer if the cache file was generated by HTCondor and not by my script itself,
which is why I suggested the "maxage" setting. 

> You should ONLY use cached config files with the following pattern
> 
> if $(IsMaster)
>   include ifexist command into $(LOCAL_DIR)/stuff.config : perl $(LOCAL_DIR)/stuff.pl
> else
>   include ifexist : $(LOCAL_DIR)/stuff.config 
> endif
> 
> You could use $(IsStartd) or $(CondorIsAdmin) instead.  These all guarantee that only a daemon running as root
> (or Admin on Windows) will create the cache file.
That's a nice best-practice, thanks! 

> Of course, once you do this it becomes clear why having a timeout on the cache file doesn't do much good.  The master
> doesn't re-parse the config unless you reconfig or restart it - so a timeout isn't going to do much good unless you have
> a cron job reconfig-ing the master regularly - and if you do you might as well have the cron job delete the stuff.config file
> just before it reconfigs the master.
That would be the way I would have gone, and would have assumed things to work from the docs - but then, apparently, the answer to my original question is wrong. 
My original question was, "which daemon" "how often" re-evaluates the configuration.
If I go to a worker node (startd), as admin, and call "condor_config_val -startd -dump", I immediately see the new value created from an included script
(if no cache file is used). 

How does this work? Why does this cause the startd to re-parse the configuration? 
Surely there's no daemon being restarted when I type "condor_config_val -startd -dump" - or does this actually restart the startd? 

Cheers and many thanks,
Oliver


> 
> -tj
> 
> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Oliver Freyermuth
> Sent: Friday, November 3, 2017 2:15 PM
> To: htcondor-users <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] "include command : " in configuration files
> 
> Dear Michael, 
> 
> many thanks for your reply!
> 
> Am 03.11.2017 um 20:05 schrieb Michael Pelletier:
>> Oliver,
>>
>> It does indeed run every time the config is read, including when condor_submit or condor_config_val are run by any user.
>>
>> This caused some issues with condor_gpu_discovery in an earlier version which emitted a CUDA error which tripped up the configuration. I don't recall my support ticket number on that, but it should show up in the recent release notes.
>>
>> The workaround I did is to set up any config-included script to cache the persistent results and display the cached value if it's fresh enough.
> This is a very good idea to allow to define the caching period ourselves! 
> I'll go with a similar solution, then, once the script gets more complicated (as of now, it just does a "stat()" and a "cat" of a file). 
> 
> Since it appears this is not only affecting me, but also others, 
> it might be a good idea though (for the long run) to have something like (with maxage 1200 specifying a 20 minute caching):
> 
> include ifexist command into $(LOCAL_DIR)/stuff.config maxage 1200 : perl $(LOCAL_DIR)/stuff.pl
> 
> in HTCondor (which will do just the same we are right now doing with external workarounds). 
> Since HTCondor has to check for the existence of the cache-file in any case, this should be very straightforward to implement in the code. 
> 
> Is this mailing list also the suitable place to collect such ideas / feature requests? 
> 
> Cheers and many thanks,
> Oliver
> 
>>
>> 	-Michael Pelletier.
>>
>>
>>
>>> -----Original Message-----
>>> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf
>>> Of Oliver Freyermuth
>>> Sent: Friday, November 03, 2017 1:05 PM
>>> To: htcondor-users <htcondor-users@xxxxxxxxxxx>
>>> Subject: [External] [HTCondor-users] "include command : " in configuration
>>> files
>>>
>>> Dear HTCondor experts,
>>>
>>> I am now using a statement like:
>>> include command : /some/shell/script
>>> (i.e. without cache) in a configuration file to set a variable which is
>>> later used to define SINGULARITY_IMAGE_EXPR, i.e. the images used for
>>> jobs.
>>> We want these to be updated according to an external file stored on CVMFS,
>>> which is edited on a different host, so the "polling" is actually
>>> necessary.
>>>
>>> I am wondering, though, how often "/some/shell/script" is actually called.
>>> The documentation only states:
>>> "If the into option is not used, the command line will be executed every
>>> time the configuration file is referenced. "
>>> From this, I would have guessed this happens as often as a daemon is
>>> restarted (in this case, the condor_startd).
>>>
>>> However, it seems that this happens more often - does it happen each time
>>> *any* configuration parameter is queried?
>>> Is there a regular, minimal refresh interval?
>>>
>>> Many thanks and all the best,
>>> 	Oliver
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
>