Re: [HTCondor-users] setting up condor-9.0.4 with some newbie questions

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 8/16/2021 11:52 AM, Mary Hester wrote:

> Hello,
>
> I'm setting up a test htcondor cluster coupled with an ARC-CE, and I've
> started grasping at straws for how to correctly implement the new 9.0
> security model using the IDTOKENs for authentication and then defining
> the correct authorisation rules. I'm still new to htcondor and I've been
> following the instructions closely between these two sections:
>
> https://htcondor.readthedocs.io/en/v9_0/admin-manual/security.html#quick-configuration-of-security
>
> https://htcondor.readthedocs.io/en/v9_0/admin-manual/security.html#example-of-authorization-security-configuration
>
> however, for the authorisation, I'm still seeing these errors in
> /var/log/condor/CollectorLog:
>
> 08/16/21 18:40:51 (Sending 3 ads in response to query)
> 08/16/21 18:40:51 Query info: matched=3; skipped=8; query_time=0.000392;
> send_time=0.000731; type=Any; requirements={(((MyType == "Submitter"))
> || ((MyType == "Machine")))}; locate=0; limit=0; from=COLLECTOR;
> peer=<XXXX>; projection={}; filter_private_ads=0
> 08/16/21 18:40:51 QueryWorker: forked new high priority worker with id
> 32536 ( max 4 active 1 pending 0 )
> 08/16/21 18:41:00 DC_AUTHENTICATE: authentication of <XXXX> was
> successful but resulted in a limited authorization which did not include
> this command (5 QUERY_STARTD_ADS), so aborting.
> 08/16/21 18:41:00 DC_AUTHENTICATE: Command not authorized, done!
>
> I'm using the same ALLOW_* statements defined at
> https://htcondor.readthedocs.io/en/v9_0/admin-manual/security.html#example-of-authorization-security-configuration
> but this doesn't seem to be the case, and if I define nothing or some
> very 'open' version of those rules, I still see the same error. I ran
> the condor_token_request_auto_approve for the netblock block the servers
> are in so that shouldn't be the blocking factor (I think)... Can you
> give me some hints as to how to fix the limited authorisation problem?
>
> Thanks,
>
> Mary
>     
    
    Hi Mary,
    
    So from the above, it sounds like "condor_status" is not working for
    you after trying to configure your pool to use IDTOKENS for
    security.  Is this correct?
    
    Some thoughts / pointers:
    
    1.  I find condor_ping to be a helpful tool when debugging
    authentication/authorization configuration.  In the same environment
    as observing condor_status failing (i.e. logged in as the same user
    on the same machine), try the following command:
           condor_ping -type collector -table all
    This command will connect to the daemon specified (in this case the
    collector) and output a table showing, for each authorization level,
    what authentication method was used/attempted, if authentication
    succeeded, what identity the authentication mapped to, and
    ultimately if that authorization level is allowed or denied. 
    Handy!  So for queries to the collector, you will be interested is
    looking at the row corresponding to READ authorization.
    
    2. Sometimes it is best to start fresh....  IMO the easiest/fastest
    way setup a secure pool  with HTCondor v9.0.x using IDTOKENS is to
    just use "get_htcondor" tool to install and configure a new pool. 
    Having all this configuration capability is powerful and flexible,
    but this comes at a cost of complexity... you can avoid this
    complexity by starting out from a good/known/tested configuration,
    which is exactly what you get by using "get_htcondor".  The
    get_htcondor tool will install HTCondor via native packaging (RPMs
    or DEBs), and also put a default configuration in place whereby all
    daemons are authenticated via IDTOKENS, READ operations are allowed
    by anyone (thus condor_status works), and end-users are
    authenticated via either FS (filesystem) or their own IDTOKEN
    (perhaps obtained by the user themselves via condor_token_fetch). 
    This is the recommended security setup for the majority of sites. 
    Using get_htcondor to setup a new pool is very fast and easy; you
    just issue one command on your central manager, one command on each
    access point (submit machine), and one command on each worker node. 
    See
       
https://htcondor.readthedocs.io/en/v9_0/getting-htcondor/admin-quick-start.html
    for the commands to cut-n-paste.  You can also see a video demo of
    me installing a new secure HTCondor pool w/ IDTOKENS in ~3  minutes 
    -  the video is at https://youtu.be/BsE0xvts-iA and the demo starts
    at 3min 40secs.  Once your new secure pool is happily running, you
    can bring over any custom configuration from your old pool in a
    controlled manner by dropping files into /etc/condor/config.d.  I
    suggest you omit any config params dealing with
    security/authorization settings in your old setup, since
    get_htcondor set that all up for you in your new setup.  
    
    3. If you don't want to start fresh with get_htcondor, please
    reconsider :).  If you still don't want to, I suggest you at least
    take a look at how get_htcondor will configure the security setup
    and perhaps just replicate it in your setup.  Specifically, it just
    includes the following configuration template in the config:  "use
    security:get_htcondor_idtokens".   To inspect the what this template
    expands to be, enter the following command:
         condor_config_val use security:get_htcondor_idtokens
    In the output from this command, you will see there are lots of
    comments to explain why things are configured the way they are.  To
    deal with the collector query problem you encountered, note that
    this config allows ANONYMOUS authentication solely for READ
    operations (all other operations require either IDTOKENS or FS
    authentication).  
    Also note that when you setup a pool with get_htcondor, it places a
    copy of the signing key on each server in your pool in
    /etc/condor/passwords.d.  This is required to allow proper operation
    for command coming from tools like condor_off, condor_drain, etc
    that connect directly to daemons.
    
    4.  All the stuff about approval of token requests (auto or
    otherwise) is helpful when dealing with more complex situations like
    glideins (pilot poolls) or flocking, but is not useful IMO when
    dealing with setting up a pool on a cluster of servers where
    HTCondor runs as root.  Again, this is because you will want a copy
    of the signing key on each server in your pool in
    /etc/condor/passwords.d.....
    
    I hope the above pointers help!  Of course, feel free to follow-up
    if you still are stuck.
    
    regards,
    Todd