Mailing List Archives Public Access	UW Madison Computer Sciences Department Computer Systems Lab
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Kerberos Permission Denied error

Date: Fri, 6 Sep 2019 13:09:58 +0000
From: Zach Miller <zmiller@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Kerberos Permission Denied error
Hi Asvija,

If you want to force all authenticated transactions to use krb, you can just set this one setting:

SEC_DEFAULT_AUTHENTICATION_METHODS = KERBEROS


But forcing any of the authentication settings to "NEVER" means that krb authentication cannot/will-not happen.

(Also, there might be a typo in the config that you posted, but you should remove those lines anyway.  I would start with just the above single setting.)


Cheers,
-zach


ïOn 9/6/19, 2:04 AM, "HTCondor-users on behalf of Asvija B" <htcondor-users-bounces@xxxxxxxxxxx on behalf of asvijab@xxxxxxx> wrote:

    Dear all,
    I am trying to use Kerberos authentication for submitting jobs to HT-Condor.  However on the client side the submission fails complaining 'AUTHENTICATE:1002:Failure performing handshake'.   The schedd log tells that the permission was denied with this error:  
     DaemonCore: PERMISSION DENIED for 1112 (QMGMT_WRITE_CMD) via TCP from host <10.180.141.148:15918> (access level WRITE)
    It is a simple setup to test the Kerberos integration with condor.  The KDC is running on the same machine (10.180.141.148).   The same machine has been configured to run as both condor submit node and worker nodes.
    
    
    I have given the most open options for security in the condor_config file.   Following are the excerpts from condor_config file,  client debug messages and the schedd log entries:
    condor_config file excerpt:
    SEC_DEFAULT_NEGOTIATION = OPTIONAL
    SEC_DEFAULT_AUTHENTICATION = NEVER
    SEC_CLIENT_AUTHENCTICATION = NEVER
    SEC_DEFAULT_AUTHENTICATION_METHODS = KERBEROS
    KERBEROS_MAP_FILE = $(RELEASE_DIR)/etc/condor.kmap
    SCHEDD.ALLOW_WRITE = *@*/*, 10.180.141.148
    SEC_WRITE_AUTHENTICATION = NEVER
    
    
    
    condor.kmap contents:
    
    [root@gridfs log]# cat /usr/local/nsg/condor/etc/condor.kmap
    NSGTEST.CDAC.IN = nsgtest.cdac.in
    
    
    
    
    Kerberos klist output on client side:
    
    [asvija@gridfs condor]$  klist
    Ticket cache: KEYRING:persistent:1005:1005
    Default principal: 
    asvija@xxxxxxxxxxxxxxx <mailto:asvija@xxxxxxxxxxxxxxx>
    
    Valid starting       Expires              Service principal
    09/06/2019 12:18:30  09/07/2019 12:18:30  
    krbtgt/NSGTEST.CDAC.IN@xxxxxxxxxxxxxxx <mailto:krbtgt/NSGTEST.CDAC.IN@xxxxxxxxxxxxxxx>
    
    Debug output from condor_submit :
    
    [asvija@gridfs condor]$ _condor_TOOL_DEBUG=D_SECURITY condor_submit -debug condor-universe.job 2>&1 | tee out                                                                                                                               
    
    
    09/06/19 12:21:05 KEYCACHE: created: 0x239a150
    09/06/19 12:21:05 Can't open directory "/opt/condor//config" as PRIV_UNKNOWN, errno: 2 (No such file or directory)
    09/06/19 12:21:05 Cannot open /opt/condor//config: No such file or directory
    Submitting job(s)09/06/19 12:21:05 CRED: NO MODULES REQUESTED
    09/06/19 12:21:05 SECMAN: command 1112 QMGMT_WRITE_CMD to schedd at <10.180.141.148:9618> from TCP port 22376 (blocking).
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission ALLOW
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission READ
    09/06/19 12:21:05 ipverify: READ optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission WRITE
    09/06/19 12:21:05 ipverify: WRITE optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission NEGOTIATOR
    09/06/19 12:21:05 ipverify: NEGOTIATOR optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission ADMINISTRATOR
    09/06/19 12:21:05 ipverify: ADMINISTRATOR optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission OWNER
    09/06/19 12:21:05 ipverify: OWNER optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission CONFIG
    09/06/19 12:21:05 ipverify: CONFIG optimized to deny everyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission DAEMON
    09/06/19 12:21:05 ipverify: DAEMON optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission SOAP
    09/06/19 12:21:05 ipverify: SOAP optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission DEFAULT
    09/06/19 12:21:05 ipverify: DEFAULT optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission CLIENT
    09/06/19 12:21:05 ipverify: CLIENT optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission ADVERTISE_STARTD
    09/06/19 12:21:05 ipverify: ADVERTISE_STARTD optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission ADVERTISE_SCHEDD
    09/06/19 12:21:05 ipverify: ADVERTISE_SCHEDD optimized to allow anyone
    09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
    09/06/19 12:21:05 IPVERIFY: Permission ADVERTISE_MASTER
    09/06/19 12:21:05 ipverify: ADVERTISE_MASTER optimized to allow anyone
    09/06/19 12:21:05 AUTHENTICATE: setting timeout for <10.180.141.148:9618?addrs=10.180.141.148-9618&noUDP&sock=83499_42eb_4> to 20.
    09/06/19 12:21:05 HANDSHAKE: in handshake(my_methods = 'KERBEROS')
    09/06/19 12:21:05 HANDSHAKE: handshake() - i am the client
    09/06/19 12:21:05 HANDSHAKE: sending (methods == 64) to server
    09/06/19 12:21:05 HANDSHAKE: server replied (method = 64)
    09/06/19 12:21:05 KERBEROS: krb5_unparse_name: 
    condor@xxxxxxxxxxxxxxx <mailto:condor@xxxxxxxxxxxxxxx>
    09/06/19 12:21:05 KERBEROS: param server princ: condor
    09/06/19 12:21:05 KERBEROS: no user yet determined, will grab up to slash
    09/06/19 12:21:05 KERBEROS: picked user: condor
    09/06/19 12:21:05 Client is 
    condor@xxxxxxxxxxxxxxx <mailto:condor@xxxxxxxxxxxxxxx>
    09/06/19 12:21:05 KERBEROS: Server principal is 
    condor@xxxxxxxxxxxxxxx <mailto:condor@xxxxxxxxxxxxxxx>
    09/06/19 12:21:05 Acquiring credential for user
    09/06/19 12:21:05 Successfully located credential cache
    09/06/19 12:21:05 condor_write(): Socket closed when trying to write 13 bytes to schedd at <10.180.141.148:9618>, fd is 4
    09/06/19 12:21:05 Buf::write(): condor_write() failed
    09/06/19 12:21:05 AUTHENTICATE: method 64 (KERBEROS) failed.
    09/06/19 12:21:05 HANDSHAKE: in handshake(my_methods = '')
    09/06/19 12:21:05 HANDSHAKE: handshake() - i am the client
    09/06/19 12:21:05 HANDSHAKE: sending (methods == 0) to server
    09/06/19 12:21:05 condor_write(): Socket closed when trying to write 13 bytes to schedd at <10.180.141.148:9618>, fd is 4
    09/06/19 12:21:05 Buf::write(): condor_write() failed
    09/06/19 12:21:05 AUTHENTICATE: handshake failed!
    09/06/19 12:21:05 Authentication was a FAILURE.
    
    ERROR: Failed to connect to local queue manager
    AUTHENTICATE:1002:Failure performing handshake
    AUTHENTICATE:1004:Failed to authenticate using KERBEROS
    
    
    
    Schedd log:
    09/06/19 12:26:22 (pid:83694) ******************************************************
    09/06/19 12:26:22 (pid:83694) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
    09/06/19 12:26:22 (pid:83694) ** /usr/local/nsg/condor/sbin/condor_schedd
    09/06/19 12:26:22 (pid:83694) ** SubsystemInfo: name=SCHEDD type=SCHEDD(5) class=DAEMON(1)
    09/06/19 12:26:22 (pid:83694) ** Configuration: subsystem:SCHEDD local:<NONE> class:DAEMON
    09/06/19 12:26:22 (pid:83694) ** $CondorVersion: 8.8.4 Jul 09 2019 BuildID: 474941 $
    09/06/19 12:26:22 (pid:83694) ** $CondorPlatform: x86_64_RedHat7 $
    09/06/19 12:26:22 (pid:83694) ** PID = 83694
    09/06/19 12:26:22 (pid:83694) ** Log last touched 9/6 12:26:13
    09/06/19 12:26:22 (pid:83694) ******************************************************
    09/06/19 12:26:22 (pid:83694) Using config source: /usr/local/nsg/condor/etc/condor_config
    09/06/19 12:26:22 (pid:83694) Using local config sources:
    09/06/19 12:26:22 (pid:83694)    /opt/condor//condor_config.local
    09/06/19 12:26:22 (pid:83694) config Macros = 99, Sorted = 99, StringBytes = 3606, TablesBytes = 3612
    09/06/19 12:26:22 (pid:83694) CLASSAD_CACHING is ENABLED
    09/06/19 12:26:22 (pid:83694) Daemon Log is logging: D_ALWAYS D_ERROR
    09/06/19 12:26:22 (pid:83694) SharedPortEndpoint: waiting for connections to named socket 83647_c6f0_4
    09/06/19 12:26:22 (pid:83694) DaemonCore: command socket at <10.180.141.148:9618?addrs=10.180.141.148-9618&noUDP&sock=83647_c6f0_4>
    09/06/19 12:26:22 (pid:83694) DaemonCore: private command socket at <10.180.141.148:9618?addrs=10.180.141.148-9618&noUDP&sock=83647_c6f0_4>
    09/06/19 12:26:22 (pid:83694) History file rotation is enabled.
    09/06/19 12:26:22 (pid:83694)   Maximum history file size is: 20971520 bytes
    09/06/19 12:26:22 (pid:83694)   Number of rotated history files is: 2
    09/06/19 12:26:22 (pid:83694) Reloading job factories
    09/06/19 12:26:22 (pid:83694) Loaded 0 job factories, 0 were paused, 0 failed to load
    09/06/19 12:26:28 (pid:83694) TransferQueueManager stats: active up=0/100 down=0/100; waiting up=0 down=0; wait time up=0s down=0s
    09/06/19 12:26:28 (pid:83694) TransferQueueManager upload 1m I/O load: 0 bytes/s  0.000 disk load  0.000 net load
    09/06/19 12:26:28 (pid:83694) TransferQueueManager download 1m I/O load: 0 bytes/s  0.000 disk load  0.000 net load
    09/06/19 12:27:01 (pid:83694) DaemonCore: PERMISSION DENIED for 1112 (QMGMT_WRITE_CMD) via TCP from host <10.180.141.148:26321> (access level WRITE)
    
    
    Thanks and regards,
    Asvija
    
     
    ------------------------------------------------------------------------------------------------------------
    
    [ C-DAC is on Social-Media too. Kindly follow us at: 
    Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] 
    
    This e-mail is for the sole use of the intended recipient(s) and may 
    contain confidential and privileged information. If you are not the 
    intended recipient, please contact the sender by reply e-mail and destroy 
    all copies and the original message. Any unauthorized review, use, 
    disclosure, dissemination, forwarding, printing or copying of this email 
    is strictly prohibited and appropriate legal action will be taken. 
    ------------------------------------------------------------------------------------------------------------
Follow-Ups:
- Re: [HTCondor-users] Kerberos Permission Denied error
  - From: asvijab
References:
- [HTCondor-users] Kerberos Permission Denied error
  - From: Asvija B
Prev by Date: Re: [HTCondor-users] HTCondor flock and Negotiator interval
Next by Date: Re: [HTCondor-users] HTCondor 8.8.5 Released
Previous by thread: [HTCondor-users] Kerberos Permission Denied error
Next by thread: Re: [HTCondor-users] Kerberos Permission Denied error
Index(es):
- Date
- Thread
Mailing List Archives

Public Access

Re: [HTCondor-users] Kerberos Permission Denied error
