[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] "include command : " in configuration files



Dear John, 

Am 07.11.2017 um 18:33 schrieb John M Knoeller:
>> If I go to a worker node (startd), as admin, and call "condor_config_val -startd -dump", I immediately see the new value created from an included script
> 
> The startd will not load a new config unless something is *else* is forcing the startd to reconfig.   condor_config_val will not force a reconfig, and "condor_config_val -startd -dump"  will cause condor_config_val print values from the startd's (possibly stale) config.
> 
> HOWEVER!  each time you run a tool, like condor_q, condor_submit, or condor_config_val.  the tool will parse the config and run any scripts as part of the tool initialization.  (this is needed so that the tools can find the collector, etc).

Thanks! This is exactly what I observed, and I deduced from that that the startd was itself updating it's configuration. 
So that explains it well. Since our usecase right now is that the called script in any case only does a "cat" of a world-readable file on a read-only FS and
some string concatenation, and the script itself is only writable for root, I consider our current usecase as safe. 
But your explanations have helped significantly to understand what's going on and shape our upcoming usecases - and we'll certainly limit script execution to the startd only. 

Many thanks and all the best! 
Oliver

> 
> So a config script WILL run as part of "condor_config_val -startd -dump",  but the output of condor_config_val will be the values it got from the startd, not the values it got by parsing the config itself.
> 
> -tj
> 
> 
> 
> -----Original Message-----
> From: Oliver Freyermuth [mailto:freyermuth@xxxxxxxxxxxxxxxxxx] 
> Sent: Monday, November 6, 2017 6:57 PM
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>; John M Knoeller <johnkn@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] "include command : " in configuration files
> 
> Dear John, 
> 
> Am 06.11.2017 um 23:58 schrieb John M Knoeller:
>> The problem here is that you should not let just anyone make the cached config files.  
>> You should instead make sure that ONLY a HTCondor daemon creates that cache file, all other config readers should just use the cache file.
>>
>> The reason for this your config files provide a way to get HTCondor daemons (probably running as root!) to run arbitrary code.   This means
>> that if any of your config files (including the cache files) is writable by non-root then you have created a root exploit.
> 
> Agreed. That's why I would prefer if the cache file was generated by HTCondor and not by my script itself,
> which is why I suggested the "maxage" setting. 
> 
>> You should ONLY use cached config files with the following pattern
>>
>> if $(IsMaster)
>>   include ifexist command into $(LOCAL_DIR)/stuff.config : perl $(LOCAL_DIR)/stuff.pl
>> else
>>   include ifexist : $(LOCAL_DIR)/stuff.config 
>> endif
>>
>> You could use $(IsStartd) or $(CondorIsAdmin) instead.  These all guarantee that only a daemon running as root
>> (or Admin on Windows) will create the cache file.
> That's a nice best-practice, thanks! 
> 
>> Of course, once you do this it becomes clear why having a timeout on the cache file doesn't do much good.  The master
>> doesn't re-parse the config unless you reconfig or restart it - so a timeout isn't going to do much good unless you have
>> a cron job reconfig-ing the master regularly - and if you do you might as well have the cron job delete the stuff.config file
>> just before it reconfigs the master.
> That would be the way I would have gone, and would have assumed things to work from the docs - but then, apparently, the answer to my original question is wrong. 
> My original question was, "which daemon" "how often" re-evaluates the configuration.
> If I go to a worker node (startd), as admin, and call "condor_config_val -startd -dump", I immediately see the new value created from an included script
> (if no cache file is used). 
> 
> How does this work? Why does this cause the startd to re-parse the configuration? 
> Surely there's no daemon being restarted when I type "condor_config_val -startd -dump" - or does this actually restart the startd? 
> 
> Cheers and many thanks,
> Oliver
> 
> 
>>
>> -tj
>>
>> -----Original Message-----
>> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Oliver Freyermuth
>> Sent: Friday, November 3, 2017 2:15 PM
>> To: htcondor-users <htcondor-users@xxxxxxxxxxx>
>> Subject: Re: [HTCondor-users] "include command : " in configuration files
>>
>> Dear Michael, 
>>
>> many thanks for your reply!
>>
>> Am 03.11.2017 um 20:05 schrieb Michael Pelletier:
>>> Oliver,
>>>
>>> It does indeed run every time the config is read, including when condor_submit or condor_config_val are run by any user.
>>>
>>> This caused some issues with condor_gpu_discovery in an earlier version which emitted a CUDA error which tripped up the configuration. I don't recall my support ticket number on that, but it should show up in the recent release notes.
>>>
>>> The workaround I did is to set up any config-included script to cache the persistent results and display the cached value if it's fresh enough.
>> This is a very good idea to allow to define the caching period ourselves! 
>> I'll go with a similar solution, then, once the script gets more complicated (as of now, it just does a "stat()" and a "cat" of a file). 
>>
>> Since it appears this is not only affecting me, but also others, 
>> it might be a good idea though (for the long run) to have something like (with maxage 1200 specifying a 20 minute caching):
>>
>> include ifexist command into $(LOCAL_DIR)/stuff.config maxage 1200 : perl $(LOCAL_DIR)/stuff.pl
>>
>> in HTCondor (which will do just the same we are right now doing with external workarounds). 
>> Since HTCondor has to check for the existence of the cache-file in any case, this should be very straightforward to implement in the code. 
>>
>> Is this mailing list also the suitable place to collect such ideas / feature requests? 
>>
>> Cheers and many thanks,
>> Oliver
>>
>>>
>>> 	-Michael Pelletier.
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf
>>>> Of Oliver Freyermuth
>>>> Sent: Friday, November 03, 2017 1:05 PM
>>>> To: htcondor-users <htcondor-users@xxxxxxxxxxx>
>>>> Subject: [External] [HTCondor-users] "include command : " in configuration
>>>> files
>>>>
>>>> Dear HTCondor experts,
>>>>
>>>> I am now using a statement like:
>>>> include command : /some/shell/script
>>>> (i.e. without cache) in a configuration file to set a variable which is
>>>> later used to define SINGULARITY_IMAGE_EXPR, i.e. the images used for
>>>> jobs.
>>>> We want these to be updated according to an external file stored on CVMFS,
>>>> which is edited on a different host, so the "polling" is actually
>>>> necessary.
>>>>
>>>> I am wondering, though, how often "/some/shell/script" is actually called.
>>>> The documentation only states:
>>>> "If the into option is not used, the command line will be executed every
>>>> time the configuration file is referenced. "
>>>> From this, I would have guessed this happens as often as a daemon is
>>>> restarted (in this case, the condor_startd).
>>>>
>>>> However, it seems that this happens more often - does it happen each time
>>>> *any* configuration parameter is queried?
>>>> Is there a regular, minimal refresh interval?
>>>>
>>>> Many thanks and all the best,
>>>> 	Oliver
>>>
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>>
>>
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>
> 

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature