[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] submitting jobs with API



I got this to work by adding this to my config files:

SEC_DEFAULT_AUTHENTICATION_METHODS = ClaimToBe

On Thu, Dec 21, 2017 at 4:23 PM, Larry Martell <larry.martell@xxxxxxxxx> wrote:
> $ condor_submit  -remote bach.foo.local -pool 192.168.10.2 job.sub
> Submitting job(s)
> ERROR: Failed to connect to queue manager bach.foo.local
> AUTHENTICATE:1003:Failed to authenticate with any method
> AUTHENTICATE:1004:Failed to authenticate using GSI
> GSI:5003:Failed to authenticate.  Globus is reporting error
> (851968:50).  There is probably a problem with your credentials.  (Did
> you run grid-proxy-init?)
> AUTHENTICATE:1004:Failed to authenticate using KERBEROS
> AUTHENTICATE:1004:Failed to authenticate using FS
>
> [lmartell@chopin ~]$ _condor_TOOL_DEBUG=D_SECURITY,D_FULLDEBUG
> condor_submit -debug -remote bach.foo.local -pool 192.168.10.2 job.sub
> 12/21/17 16:20:32 KEYCACHE: created: 0x1d32a40
> 12/21/17 16:20:32 Result of reading /etc/issue:  \S
>
> 12/21/17 16:20:32 Result of reading /etc/redhat-release:  Red Hat
> Enterprise Linux Server release 7.4 (Maipo)
>
> 12/21/17 16:20:32 Growing processor array to 64
> 12/21/17 16:20:32 Growing processor array to 128
> 12/21/17 16:20:32 Growing processor array to 256
> 12/21/17 16:20:32 Using IDs: 176 processors, 88 CPUs, 88 HTs
> 12/21/17 16:20:32 Reading condor configuration from '/etc/condor/condor_config'
> 12/21/17 16:20:32 Enumerating interfaces: lo 127.0.0.1 up
> 12/21/17 16:20:32 Enumerating interfaces: bond0 192.168.10.15 up
> 12/21/17 16:20:32 Enumerating interfaces: virbr0 192.168.122.1 up
> 12/21/17 16:20:32 Enumerating interfaces: vmnet1 192.168.232.1 up
> 12/21/17 16:20:32 Enumerating interfaces: docker0 172.17.0.1 up
> 12/21/17 16:20:32 Enumerating interfaces: vmnet8 172.16.208.1 up
> 12/21/17 16:20:32 Enumerating interfaces: lo ::1 up
> 12/21/17 16:20:32 Will use TCP to update collector <192.168.10.2:9618>
> 12/21/17 16:20:32 Trying to query collector <192.168.10.2:9618>
> 12/21/17 16:20:32 SECMAN: command 6 QUERY_SCHEDD_ADS to collector at
> <192.168.10.2:9618> from TCP port 46713 (blocking).
> 12/21/17 16:20:32 SECMAN:: default CLIENT methods: FS,KERBEROS,GSI,CLAIMTOBE
> 12/21/17 16:20:32 SECMAN: no cached key for {<192.168.10.2:9618>,<6>}.
> 12/21/17 16:20:32 SECMAN: Security Policy:
> SessionLease = 3600
> NewSession = "YES"
> CryptoMethods = "3DES,BLOWFISH"
> OutgoingNegotiation = "PREFERRED"
> Authentication = "OPTIONAL"
> Encryption = "OPTIONAL"
> ServerPid = 106213
> Integrity = "OPTIONAL"
> Subsystem = "SUBMIT"
> Enact = "NO"
> AuthMethods = "FS,KERBEROS,GSI,CLAIMTOBE"
> SessionDuration = "60"
> 12/21/17 16:20:32 SECMAN: negotiating security for command 6.
> 12/21/17 16:20:32 SECMAN: sending DC_AUTHENTICATE command
> 12/21/17 16:20:32 SECMAN: sending following classad:
> RemoteVersion = "$CondorVersion: 8.6.8 Nov 13 2017 BuildID: 424045 $"
> SessionLease = 3600
> NewSession = "YES"
> CryptoMethods = "3DES,BLOWFISH"
> OutgoingNegotiation = "PREFERRED"
> Authentication = "OPTIONAL"
> Encryption = "OPTIONAL"
> ServerPid = 106213
> Integrity = "OPTIONAL"
> Subsystem = "SUBMIT"
> Enact = "NO"
> Command = 6
> AuthMethods = "FS,KERBEROS,GSI,CLAIMTOBE"
> SessionDuration = "60"
> 12/21/17 16:20:32 SECMAN: server responded with:
> Encryption = "NO"
> Integrity = "NO"
> AuthMethodsList = "FS,KERBEROS,GSI,CLAIMTOBE"
> AuthMethods = "FS"
> CryptoMethods = "3DES,BLOWFISH"
> Authentication = "NO"
> SessionDuration = "60"
> SessionLease = 3600
> RemoteVersion = "$CondorVersion: 8.6.8 Nov 13 2017 BuildID: 424045 $"
> Enact = "YES"
> 12/21/17 16:20:32 SECMAN: received post-auth classad:
> User = "unauthenticated@unmapped"
> Sid = "bach:32117:1513891232:39694"
> ValidCommands = "60007,457,60020,68,5,6,7,9,12,43,20,46,78,50,56,62,65,48,71,74"
> ReturnCode = "AUTHORIZED"
> 12/21/17 16:20:32 SECMAN: policy to be cached:
> User = "unauthenticated@unmapped"
> ValidCommands = "60007,457,60020,68,5,6,7,9,12,43,20,46,78,50,56,62,65,48,71,74"
> Sid = "bach:32117:1513891232:39694"
> MyRemoteUserName = "unauthenticated@unmapped"
> UseSession = "YES"
> AuthMethodsList = "FS,KERBEROS,GSI,CLAIMTOBE"
> RemoteVersion = "$CondorVersion: 8.6.8 Nov 13 2017 BuildID: 424045 $"
> SessionLease = 3600
> CryptoMethods = "3DES,BLOWFISH"
> OutgoingNegotiation = "PREFERRED"
> Authentication = "NO"
> Encryption = "NO"
> Integrity = "NO"
> Subsystem = "SUBMIT"
> Enact = "YES"
> Command = 6
> AuthMethods = "FS"
> SessionDuration = "60"
> 12/21/17 16:20:32 SECMAN: added session bach:32117:1513891232:39694 to
> cache for 60 seconds (3600s lease).
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<60007>} mapped
> to session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<457>} mapped
> to session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<60020>} mapped
> to session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<68>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<5>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<6>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<7>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<9>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<12>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<43>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<20>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<46>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<78>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<50>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<56>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<62>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<65>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<48>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<71>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<74>} mapped to
> session bach:32117:1513891232:39694.
> 12/21/17 16:20:32 SECMAN: startCommand succeeded.
> 12/21/17 16:20:32 Authorizing server 'unauthenticated@unmapped/192.168.10.2'.
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission ALLOW
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission READ
> 12/21/17 16:20:32 ipverify: READ optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission WRITE
> 12/21/17 16:20:32 ipverify: WRITE optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission NEGOTIATOR
> 12/21/17 16:20:32 ipverify: NEGOTIATOR optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission ADMINISTRATOR
> 12/21/17 16:20:32 ipverify: ADMINISTRATOR optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission OWNER
> 12/21/17 16:20:32 ipverify: OWNER optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission CONFIG
> 12/21/17 16:20:32 ipverify: CONFIG optimized to deny everyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission DAEMON
> 12/21/17 16:20:32 ipverify: DAEMON optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission SOAP
> 12/21/17 16:20:32 ipverify: SOAP optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission DEFAULT
> 12/21/17 16:20:32 ipverify: DEFAULT optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission CLIENT
> 12/21/17 16:20:32 ipverify: CLIENT optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission ADVERTISE_STARTD
> 12/21/17 16:20:32 ipverify: ADVERTISE_STARTD optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission ADVERTISE_SCHEDD
> 12/21/17 16:20:32 ipverify: ADVERTISE_SCHEDD optimized to allow anyone
> 12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
> 12/21/17 16:20:32 IPVERIFY: Permission ADVERTISE_MASTER
> 12/21/17 16:20:32 ipverify: ADVERTISE_MASTER optimized to allow anyone
> 12/21/17 16:20:32 Initialized the following authorization table:
> 12/21/17 16:20:32 Authorizations yet to be resolved:
> Submitting job(s)12/21/17 16:20:32 SharedPortClient: sent connection
> request to schedd at <192.168.10.2:9618> for shared port id
> 32054_f4d2_4
> 12/21/17 16:20:32 SECMAN: command 1112 QMGMT_WRITE_CMD to schedd at
> <192.168.10.2:9618> from TCP port 39442 (blocking).
> 12/21/17 16:20:32 SECMAN:: default CLIENT methods: FS,KERBEROS,GSI,CLAIMTOBE
> 12/21/17 16:20:32 SECMAN: no cached key for
> {<192.168.10.2:9618?addrs=192.168.10.2-9618+[--1]-9618&noUDP&sock=32054_f4d2_4>,<1112>}.
> 12/21/17 16:20:32 SECMAN: Security Policy:
> SessionLease = 3600
> NewSession = "YES"
> CryptoMethods = "3DES,BLOWFISH"
> OutgoingNegotiation = "PREFERRED"
> Authentication = "OPTIONAL"
> Encryption = "OPTIONAL"
> ServerPid = 106213
> Integrity = "OPTIONAL"
> Subsystem = "SUBMIT"
> Enact = "NO"
> AuthMethods = "FS,KERBEROS,GSI,CLAIMTOBE"
> SessionDuration = "60"
> 12/21/17 16:20:32 SECMAN: negotiating security for command 1112.
> 12/21/17 16:20:32 SECMAN: sending DC_AUTHENTICATE command
> 12/21/17 16:20:32 SECMAN: sending following classad:
> RemoteVersion = "$CondorVersion: 8.6.8 Nov 13 2017 BuildID: 424045 $"
> SessionLease = 3600
> NewSession = "YES"
> CryptoMethods = "3DES,BLOWFISH"
> OutgoingNegotiation = "PREFERRED"
> Authentication = "OPTIONAL"
> Encryption = "OPTIONAL"
> ServerPid = 106213
> Integrity = "OPTIONAL"
> Subsystem = "SUBMIT"
> Enact = "NO"
> Command = 1112
> AuthMethods = "FS,KERBEROS,GSI,CLAIMTOBE"
> SessionDuration = "60"
> 12/21/17 16:20:32 SECMAN: server responded with:
> ServerTime = 1513891232
> Encryption = "NO"
> Integrity = "NO"
> AuthMethodsList = "FS,KERBEROS,GSI"
> AuthMethods = "FS"
> CryptoMethods = "3DES,BLOWFISH"
> Authentication = "YES"
> SessionDuration = "60"
> SessionLease = 3600
> RemoteVersion = "$CondorVersion: 8.6.8 Nov 13 2017 BuildID: 424045 $"
> Enact = "YES"
> 12/21/17 16:20:32 SECMAN: new session, doing initial authentication.
> 12/21/17 16:20:32 SECMAN: authenticating RIGHT NOW.
> 12/21/17 16:20:32 SECMAN: AuthMethodsList: FS,KERBEROS,GSI
> 12/21/17 16:20:32 SECMAN: Auth methods: FS,KERBEROS,GSI
> 12/21/17 16:20:32 AUTHENTICATE: setting timeout for
> <192.168.10.2:9618?addrs=192.168.10.2-9618+[--1]-9618&noUDP&sock=32054_f4d2_4>
> to 20.
> 12/21/17 16:20:32 AUTHENTICATE: in authenticate( addr ==
> '<192.168.10.2:9618?addrs=192.168.10.2-9618+[--1]-9618&noUDP&sock=32054_f4d2_4>',
> methods == 'FS,KERBEROS,GSI')
> 12/21/17 16:20:32 AUTHENTICATE: can still try these methods: FS,KERBEROS,GSI
> 12/21/17 16:20:32 HANDSHAKE: in handshake(my_methods = 'FS,KERBEROS,GSI')
> 12/21/17 16:20:32 HANDSHAKE: handshake() - i am the client
> 12/21/17 16:20:32 HANDSHAKE: sending (methods == 100) to server
> 12/21/17 16:20:32 HANDSHAKE: server replied (method = 4)
> 12/21/17 16:20:32 AUTHENTICATE: will try to use 4 (FS)
> 12/21/17 16:20:32 AUTHENTICATE: do_authenticate is 1.
> 12/21/17 16:20:32 AUTHENTICATE_FS: used dir /tmp/FS_XXXQCWvEK, status: 0
> 12/21/17 16:20:32 AUTHENTICATE: method 4 (FS) failed.
> 12/21/17 16:20:32 AUTHENTICATE: can still try these methods: KERBEROS,GSI
> 12/21/17 16:20:32 HANDSHAKE: in handshake(my_methods = 'KERBEROS,GSI')
> 12/21/17 16:20:32 HANDSHAKE: handshake() - i am the client
> 12/21/17 16:20:32 HANDSHAKE: sending (methods == 96) to server
> 12/21/17 16:20:32 HANDSHAKE: server replied (method = 64)
> 12/21/17 16:20:32 AUTHENTICATE: will try to use 64 (KERBEROS)
> 12/21/17 16:20:32 AUTHENTICATE: do_authenticate is 1.
> 12/21/17 16:20:32 KERBEROS: krb5_unparse_name: host/bach@
> 12/21/17 16:20:32 KERBEROS: no user yet determined, will grab up to slash
> 12/21/17 16:20:32 KERBEROS: picked user: host
> 12/21/17 16:20:32 KERBEROS: remapping 'host' to 'condor'
> 12/21/17 16:20:32 unable to open map file (null), errno 22
> 12/21/17 16:20:32 KERBEROS: mapping realm  to domain .
> 12/21/17 16:20:32 Client is condor@
> 12/21/17 16:20:32 KERBEROS: Server principal is host/bach@
> 12/21/17 16:20:32 Acquiring credential for user
> 12/21/17 16:20:32 KERBEROS: No credentials cache found
> 12/21/17 16:20:32 AUTHENTICATE: method 64 (KERBEROS) failed.
> 12/21/17 16:20:32 AUTHENTICATE: can still try these methods: GSI
> 12/21/17 16:20:32 HANDSHAKE: in handshake(my_methods = 'GSI')
> 12/21/17 16:20:32 HANDSHAKE: handshake() - i am the client
> 12/21/17 16:20:32 HANDSHAKE: sending (methods == 32) to server
> 12/21/17 16:20:32 HANDSHAKE: server replied (method = 32)
> 12/21/17 16:20:32 AUTHENTICATE: will try to use 32 (GSI)
> 12/21/17 16:20:32 AUTHENTICATE: do_authenticate is 1.
> 12/21/17 16:20:32 authenticate_self_gss: acquiring self credentials
> failed. Please check your Condor configuration file if this is a
> server process. Or the user environment variable if this is a user
> process.
>
> GSS Major Status: General failure
> GSS Minor Status Error Chain:
> globus_gsi_gssapi: Error with GSI credential
> globus_gsi_gssapi: Error with gss credential handle
> globus_credential: Valid credentials could not be found in any of the
> possible locations specified by the credential search order.
> Valid credentials could not be found in any of the possible locations
> specified by the credential search order.
> Attempt 1
> globus_credential: Error reading host credential
> globus_sysconfig: Could not find a valid certificate file: The host
> cert could not be found in:
> 1) env. var. X509_USER_CERT
> 2) /etc/grid-security/hostcert.pem
> 3) $GLOBUS_LOCATION/etc/hostcert.pem
> 4) $HOME/.globus/hostcert.pem
>
> The host key could not be found in:
> 1) env. var. X509_USER_KEY
> 2) /etc/grid-security/hostkey.pem
> 3) $GLOBUS_LOCATION/etc/hostkey.pem
> 4) $HOME/.globus/hostkey.pem
>
>
> Attempt 2
> globus_credential: Error reading proxy credential
> globus_sysconfig: Could not find a valid proxy certificate file location
> globus_sysconfig: Error with key filename
> globus_sysconfig: File does not exist: /tmp/x509up_u1104 is not a valid file
> Attempt 3
> globus_credential: Error reading user credential
> globus_sysconfig: Error with certificate filename: The user cert could
> not be found in:
> 1) env. var. X509_USER_CERT
> 2) $HOME/.globus/usercert.pem
> 3) $HOME/.globus/usercred.p12
>
>
>
> 12/21/17 16:20:32 authenticate: user creds not established
> 12/21/17 16:20:32 AUTHENTICATE: method 32 (GSI) failed.
> 12/21/17 16:20:32 AUTHENTICATE: can still try these methods:
> 12/21/17 16:20:32 HANDSHAKE: in handshake(my_methods = '')
> 12/21/17 16:20:32 HANDSHAKE: handshake() - i am the client
> 12/21/17 16:20:32 HANDSHAKE: sending (methods == 0) to server
> 12/21/17 16:20:32 HANDSHAKE: server replied (method = 0)
> 12/21/17 16:20:32 AUTHENTICATE: no available authentication methods succeeded!
> 12/21/17 16:20:32 SECMAN: required authentication with schedd at
> <192.168.10.2:9618> failed, so aborting command QMGMT_WRITE_CMD.
>
> ERROR: Failed to connect to queue manager bach.elucid.local
> AUTHENTICATE:1003:Failed to authenticate with any method
> AUTHENTICATE:1004:Failed to authenticate using GSI
> GSI:5003:Failed to authenticate.  Globus is reporting error
> (851968:50).  There is probably a problem with your credentials.  (Did
> you run grid-proxy-init?)
> AUTHENTICATE:1004:Failed to authenticate using KERBEROS
> AUTHENTICATE:1004:Failed to authenticate using FS
> 12/21/17 16:20:32 KEYCACHEENTRY: deleted: 0x1e88250
> 12/21/17 16:20:32 KEYCACHE: deleted: 0x1d32a40
>
>
>
> On Tue, Dec 19, 2017 at 10:14 PM, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:
>> Hi Larry,
>>
>> This is definitely an issue with the security subsystem, not the python API.  I suspect that you can reproduce it via the command line tools with something like:
>>
>> condor_submit -remote <schedd name> -pool <collector name> submit_file
>>
>> Sometimes it's a bit simpler to increase the logging via the CLI (the error messages don't always come back in a usable manner for the python API).
>>
>> If you can reproduce it with condor_submit, try:
>>
>> _condor_TOOL_DEBUG=D_SECURITY,D_FULLDEBUG condor_submit -debug -remote <schedd name> -pool <collector name> submit_file
>>
>> That should provide a full readout of the security handshake.
>>
>> The puzzling thing is that this line:
>>
>> use SECURITY : HOST_BASED
>>
>> in your server config (oh - did you do a condor_reconfig after the change?) should theoretically disable the attempts to do GSI-based security negotiation.  However, the logfiles clearly show it is being attempted.
>>
>> So -- this suggests something slightly wrong with the schedd configuration, but it's not clear what is wrong yet.
>>
>> Brian
>>
>>> On Dec 19, 2017, at 11:25 AM, Larry Martell <larry.martell@xxxxxxxxx> wrote:
>>>
>>> This is what was logged in SchedLog in the submit attempt. Note I have
>>> these security related settings in my config file. Do I need other
>>> settings to allow this to work?
>>>
>>> use SECURITY : HOST_BASED
>>> ALLOW_WRITE = 192.168.*
>>> ALLOW_READ = 192.168.*
>>>
>>>
>>> 12/19/17 11:13:13 (pid:32123) authenticate_self_gss: acquiring self
>>> credentials failed. Please check your Condor configuration file if
>>> this is a server process. Or the user environment variable if this is
>>> a user process.
>>>
>>> GSS Major Status: General failure
>>> GSS Minor Status Error Chain:
>>> globus_gsi_gssapi: Error with GSI credential
>>> globus_gsi_gssapi: Error with gss credential handle
>>> globus_credential: Valid credentials could not be found in any of the
>>> possible locations specified by the credential search order.
>>> Valid credentials could not be found in any of the possible locations
>>> specified by the credential search order.
>>> Attempt 1
>>> globus_credential: Error reading host credential
>>> globus_sysconfig: Could not find a valid certificate file: The host
>>> cert could not be found in:
>>> 1) env. var. X509_USER_CERT
>>> 2) /etc/grid-security/hostcert.pem
>>> 3) $GLOBUS_LOCATION/etc/hostcert.pem
>>> 4) $HOME/.globus/hostcert.pem
>>>
>>> The host key could not be found in:
>>> 1) env. var. X509_USER_KEY
>>> 2) /etc/grid-security/hostkey.pem
>>> 3) $GLOBUS_LOCATION/etc/hostkey.pem
>>> 4) $HOME/.globus/hostkey.pem
>>>
>>>
>>> Attempt 2
>>> globus_credential: Error reading proxy credential
>>> globus_sysconfig: Could not find a valid proxy certificate file location
>>> globus_sysconfig: Error with key filename
>>> globus_sysconfig: File does not exist: /tmp/x509up_u0 is not a valid file
>>> Attempt 3
>>> globus_credential: Error reading user credential
>>> globus_sysconfig: Error with certificate filename: The user cert could
>>> not be found in:
>>> 1) env. var. X509_USER_CERT
>>> 2) $HOME/.globus/usercert.pem
>>> 3) $HOME/.globus/usercred.p12
>>>
>>>
>>>
>>> 12/19/17 11:13:13 (pid:32123) DC_AUTHENTICATE: authentication of
>>> <192.168.10.15:45684> did not result in a valid mapped user name,
>>> which is required for this command (1112 QMGMT_WRITE_CMD), so
>>> aborting.
>>> 12/19/17 11:13:13 (pid:32123) DC_AUTHENTICATE: reason for
>>> authentication failure: AUTHENTICATE:1003:Failed to authenticate with
>>> any method|AUTHENTICATE:1004:Failed to authenticate using
>>> GSI|GSI:5003:Failed to authenticate.  Globus is reporting error
>>> (851968:152).  There is probably a problem with your credentials.
>>> (Did you run grid-proxy-init?)|AUTHENTICATE:1004:Failed to
>>> authenticate using KERBEROS|AUTHENTICATE:1004:Failed to authenticate
>>> using FS|FS:1004:Unable to lstat(/tmp/FS_XXX4oulm8)
>>>
>>>
>>> On Tue, Dec 19, 2017 at 10:33 AM, Jason Patton <jpatton@xxxxxxxxxxx> wrote:
>>>> I don't have a solution, but hopefully I can help get the ball rolling.
>>>> Without modifying my schedd config, I tried doing a remote submit following
>>>> the same steps, which failed with the same error. The error is a little
>>>> misleading/light on details, it's likely an authentication problem from not
>>>> being on the same system as the schedd. Doing essentially the same thing
>>>> using the client tools gives more info:
>>>>
>>>>>>> schedd.submit(ad)
>>>> Traceback (most recent call last):
>>>>  File "<stdin>", line 1, in <module>
>>>> RuntimeError: Failed to connect to schedd.
>>>>
>>>> $ condor_submit test.submit -remote condor-el7.test
>>>> Submitting job(s)
>>>> ERROR: Failed to connect to queue manager condor-el7.test
>>>> AUTHENTICATE:1003:Failed to authenticate with any method
>>>> AUTHENTICATE:1004:Failed to authenticate using GSI
>>>> GSI:5003:Failed to authenticate.  Globus is reporting error (851968:50).
>>>> There is probably a problem with your credentials.  (Did you run
>>>> grid-proxy-init?)
>>>> AUTHENTICATE:1004:Failed to authenticate using KERBEROS
>>>> AUTHENTICATE:1004:Failed to authenticate using FS
>>>>
>>>> You should see more details in SchedLog on your submit host.
>>>>
>>>> Hopefully someone more knowledgable about setting up the schedd to accept
>>>> remote job submissions can chime in. (ENABLE_SOAP and ENABLE_WEB_SERVER are
>>>> probably not needed.)
>>>>
>>>> Jason
>>>>
>>>> On Tue, Dec 19, 2017 at 9:02 AM, Larry Martell <larry.martell@xxxxxxxxx>
>>>> wrote:
>>>>>
>>>>> On Tue, Dec 19, 2017 at 9:29 AM, Larry Martell <larry.martell@xxxxxxxxx>
>>>>> wrote:
>>>>>> I am doing this:
>>>>>>
>>>>>> import htcondor
>>>>>> import classad
>>>>>> condor_host = '192.168.10.2'
>>>>>> coll = htcondor.Collector(condor_host)
>>>>>> schedd_ad = coll.locate(htcondor.DaemonTypes.Schedd)
>>>>>> schedd = htcondor.Schedd(schedd_ad)
>>>>>> ad = classad.ClassAd()
>>>>>>
>>>>>> # set up ad
>>>>>>
>>>>>> id = schedd.submit(ad)
>>>>>>
>>>>>> RuntimeError: 'Failed to connect to schedd.'
>>>>>>
>>>>>> On 192.168.10.2:
>>>>>>
>>>>>> 4 S condor     32054       1  0  80   0 - 18610 poll_s Dec12 ?
>>>>>> 00:00:15 /usr/sbin/condor_master -f
>>>>>> 4 S root       32112   32054  0  80   0 -  6652 poll_s Dec12 ?
>>>>>> 00:07:51 condor_procd -A /var/run/condor/procd_pipe -L
>>>>>> /var/log/condor/ProcLog -R 1000000 -S 60 -C 986
>>>>>> 4 S condor     32113   32054  0  80   0 - 13531 poll_s Dec12 ?
>>>>>> 00:00:44 condor_shared_port -f
>>>>>> 4 S condor     32117   32054  0  80   0 - 20511 poll_s Dec12 ?
>>>>>> 00:07:46 condor_collector -f
>>>>>> 4 S condor     32122   32054  0  80   0 - 15856 poll_s Dec12 ?
>>>>>> 00:31:40 condor_negotiator -f
>>>>>> 4 S condor     32123   32054  0  80   0 - 18808 poll_s Dec12 ?
>>>>>> 00:00:31 condor_schedd -f
>>>>>>
>>>>>> From the machine running the python code:
>>>>>>
>>>>>> $ nmap -p 9618 192.168.10.2
>>>>>>
>>>>>> Starting Nmap 6.40 ( http://nmap.org ) at 2017-12-19 09:28 EST
>>>>>> Nmap scan report for 192.168.10.2
>>>>>> Host is up (0.00018s latency).
>>>>>> PORT     STATE SERVICE
>>>>>> 9618/tcp open  condor
>>>>>>
>>>>>> Am I doing something wrong or missing something?
>>>>>
>>>>> Also let me add I have these settings in the config file:
>>>>>
>>>>> ENABLE_SOAP = True
>>>>> ENABLE_WEB_SERVER = True
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>
>>
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/