Mailing List Archives Public Access	UW Madison Computer Sciences Department Computer Systems Lab
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] submitting jobs with API

Date: Thu, 21 Dec 2017 16:23:31 -0500
From: Larry Martell <larry.martell@xxxxxxxxx>
Subject: Re: [HTCondor-users] submitting jobs with API
$ condor_submit  -remote bach.foo.local -pool 192.168.10.2 job.sub
Submitting job(s)
ERROR: Failed to connect to queue manager bach.foo.local
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using GSI
GSI:5003:Failed to authenticate.  Globus is reporting error
(851968:50).  There is probably a problem with your credentials.  (Did
you run grid-proxy-init?)
AUTHENTICATE:1004:Failed to authenticate using KERBEROS
AUTHENTICATE:1004:Failed to authenticate using FS

[lmartell@chopin ~]$ _condor_TOOL_DEBUG=D_SECURITY,D_FULLDEBUG
condor_submit -debug -remote bach.foo.local -pool 192.168.10.2 job.sub
12/21/17 16:20:32 KEYCACHE: created: 0x1d32a40
12/21/17 16:20:32 Result of reading /etc/issue:  \S

12/21/17 16:20:32 Result of reading /etc/redhat-release:  Red Hat
Enterprise Linux Server release 7.4 (Maipo)

12/21/17 16:20:32 Growing processor array to 64
12/21/17 16:20:32 Growing processor array to 128
12/21/17 16:20:32 Growing processor array to 256
12/21/17 16:20:32 Using IDs: 176 processors, 88 CPUs, 88 HTs
12/21/17 16:20:32 Reading condor configuration from '/etc/condor/condor_config'
12/21/17 16:20:32 Enumerating interfaces: lo 127.0.0.1 up
12/21/17 16:20:32 Enumerating interfaces: bond0 192.168.10.15 up
12/21/17 16:20:32 Enumerating interfaces: virbr0 192.168.122.1 up
12/21/17 16:20:32 Enumerating interfaces: vmnet1 192.168.232.1 up
12/21/17 16:20:32 Enumerating interfaces: docker0 172.17.0.1 up
12/21/17 16:20:32 Enumerating interfaces: vmnet8 172.16.208.1 up
12/21/17 16:20:32 Enumerating interfaces: lo ::1 up
12/21/17 16:20:32 Will use TCP to update collector <192.168.10.2:9618>
12/21/17 16:20:32 Trying to query collector <192.168.10.2:9618>
12/21/17 16:20:32 SECMAN: command 6 QUERY_SCHEDD_ADS to collector at
<192.168.10.2:9618> from TCP port 46713 (blocking).
12/21/17 16:20:32 SECMAN:: default CLIENT methods: FS,KERBEROS,GSI,CLAIMTOBE
12/21/17 16:20:32 SECMAN: no cached key for {<192.168.10.2:9618>,<6>}.
12/21/17 16:20:32 SECMAN: Security Policy:
SessionLease = 3600
NewSession = "YES"
CryptoMethods = "3DES,BLOWFISH"
OutgoingNegotiation = "PREFERRED"
Authentication = "OPTIONAL"
Encryption = "OPTIONAL"
ServerPid = 106213
Integrity = "OPTIONAL"
Subsystem = "SUBMIT"
Enact = "NO"
AuthMethods = "FS,KERBEROS,GSI,CLAIMTOBE"
SessionDuration = "60"
12/21/17 16:20:32 SECMAN: negotiating security for command 6.
12/21/17 16:20:32 SECMAN: sending DC_AUTHENTICATE command
12/21/17 16:20:32 SECMAN: sending following classad:
RemoteVersion = "$CondorVersion: 8.6.8 Nov 13 2017 BuildID: 424045 $"
SessionLease = 3600
NewSession = "YES"
CryptoMethods = "3DES,BLOWFISH"
OutgoingNegotiation = "PREFERRED"
Authentication = "OPTIONAL"
Encryption = "OPTIONAL"
ServerPid = 106213
Integrity = "OPTIONAL"
Subsystem = "SUBMIT"
Enact = "NO"
Command = 6
AuthMethods = "FS,KERBEROS,GSI,CLAIMTOBE"
SessionDuration = "60"
12/21/17 16:20:32 SECMAN: server responded with:
Encryption = "NO"
Integrity = "NO"
AuthMethodsList = "FS,KERBEROS,GSI,CLAIMTOBE"
AuthMethods = "FS"
CryptoMethods = "3DES,BLOWFISH"
Authentication = "NO"
SessionDuration = "60"
SessionLease = 3600
RemoteVersion = "$CondorVersion: 8.6.8 Nov 13 2017 BuildID: 424045 $"
Enact = "YES"
12/21/17 16:20:32 SECMAN: received post-auth classad:
User = "unauthenticated@unmapped"
Sid = "bach:32117:1513891232:39694"
ValidCommands = "60007,457,60020,68,5,6,7,9,12,43,20,46,78,50,56,62,65,48,71,74"
ReturnCode = "AUTHORIZED"
12/21/17 16:20:32 SECMAN: policy to be cached:
User = "unauthenticated@unmapped"
ValidCommands = "60007,457,60020,68,5,6,7,9,12,43,20,46,78,50,56,62,65,48,71,74"
Sid = "bach:32117:1513891232:39694"
MyRemoteUserName = "unauthenticated@unmapped"
UseSession = "YES"
AuthMethodsList = "FS,KERBEROS,GSI,CLAIMTOBE"
RemoteVersion = "$CondorVersion: 8.6.8 Nov 13 2017 BuildID: 424045 $"
SessionLease = 3600
CryptoMethods = "3DES,BLOWFISH"
OutgoingNegotiation = "PREFERRED"
Authentication = "NO"
Encryption = "NO"
Integrity = "NO"
Subsystem = "SUBMIT"
Enact = "YES"
Command = 6
AuthMethods = "FS"
SessionDuration = "60"
12/21/17 16:20:32 SECMAN: added session bach:32117:1513891232:39694 to
cache for 60 seconds (3600s lease).
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<60007>} mapped
to session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<457>} mapped
to session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<60020>} mapped
to session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<68>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<5>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<6>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<7>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<9>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<12>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<43>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<20>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<46>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<78>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<50>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<56>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<62>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<65>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<48>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<71>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: command {<192.168.10.2:9618>,<74>} mapped to
session bach:32117:1513891232:39694.
12/21/17 16:20:32 SECMAN: startCommand succeeded.
12/21/17 16:20:32 Authorizing server 'unauthenticated@unmapped/192.168.10.2'.
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission ALLOW
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission READ
12/21/17 16:20:32 ipverify: READ optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission WRITE
12/21/17 16:20:32 ipverify: WRITE optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission NEGOTIATOR
12/21/17 16:20:32 ipverify: NEGOTIATOR optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission ADMINISTRATOR
12/21/17 16:20:32 ipverify: ADMINISTRATOR optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission OWNER
12/21/17 16:20:32 ipverify: OWNER optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission CONFIG
12/21/17 16:20:32 ipverify: CONFIG optimized to deny everyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission DAEMON
12/21/17 16:20:32 ipverify: DAEMON optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission SOAP
12/21/17 16:20:32 ipverify: SOAP optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission DEFAULT
12/21/17 16:20:32 ipverify: DEFAULT optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission CLIENT
12/21/17 16:20:32 ipverify: CLIENT optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission ADVERTISE_STARTD
12/21/17 16:20:32 ipverify: ADVERTISE_STARTD optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission ADVERTISE_SCHEDD
12/21/17 16:20:32 ipverify: ADVERTISE_SCHEDD optimized to allow anyone
12/21/17 16:20:32 IPVERIFY: Subsystem SUBMIT
12/21/17 16:20:32 IPVERIFY: Permission ADVERTISE_MASTER
12/21/17 16:20:32 ipverify: ADVERTISE_MASTER optimized to allow anyone
12/21/17 16:20:32 Initialized the following authorization table:
12/21/17 16:20:32 Authorizations yet to be resolved:
Submitting job(s)12/21/17 16:20:32 SharedPortClient: sent connection
request to schedd at <192.168.10.2:9618> for shared port id
32054_f4d2_4
12/21/17 16:20:32 SECMAN: command 1112 QMGMT_WRITE_CMD to schedd at
<192.168.10.2:9618> from TCP port 39442 (blocking).
12/21/17 16:20:32 SECMAN:: default CLIENT methods: FS,KERBEROS,GSI,CLAIMTOBE
12/21/17 16:20:32 SECMAN: no cached key for
{<192.168.10.2:9618?addrs=192.168.10.2-9618+[--1]-9618&noUDP&sock=32054_f4d2_4>,<1112>}.
12/21/17 16:20:32 SECMAN: Security Policy:
SessionLease = 3600
NewSession = "YES"
CryptoMethods = "3DES,BLOWFISH"
OutgoingNegotiation = "PREFERRED"
Authentication = "OPTIONAL"
Encryption = "OPTIONAL"
ServerPid = 106213
Integrity = "OPTIONAL"
Subsystem = "SUBMIT"
Enact = "NO"
AuthMethods = "FS,KERBEROS,GSI,CLAIMTOBE"
SessionDuration = "60"
12/21/17 16:20:32 SECMAN: negotiating security for command 1112.
12/21/17 16:20:32 SECMAN: sending DC_AUTHENTICATE command
12/21/17 16:20:32 SECMAN: sending following classad:
RemoteVersion = "$CondorVersion: 8.6.8 Nov 13 2017 BuildID: 424045 $"
SessionLease = 3600
NewSession = "YES"
CryptoMethods = "3DES,BLOWFISH"
OutgoingNegotiation = "PREFERRED"
Authentication = "OPTIONAL"
Encryption = "OPTIONAL"
ServerPid = 106213
Integrity = "OPTIONAL"
Subsystem = "SUBMIT"
Enact = "NO"
Command = 1112
AuthMethods = "FS,KERBEROS,GSI,CLAIMTOBE"
SessionDuration = "60"
12/21/17 16:20:32 SECMAN: server responded with:
ServerTime = 1513891232
Encryption = "NO"
Integrity = "NO"
AuthMethodsList = "FS,KERBEROS,GSI"
AuthMethods = "FS"
CryptoMethods = "3DES,BLOWFISH"
Authentication = "YES"
SessionDuration = "60"
SessionLease = 3600
RemoteVersion = "$CondorVersion: 8.6.8 Nov 13 2017 BuildID: 424045 $"
Enact = "YES"
12/21/17 16:20:32 SECMAN: new session, doing initial authentication.
12/21/17 16:20:32 SECMAN: authenticating RIGHT NOW.
12/21/17 16:20:32 SECMAN: AuthMethodsList: FS,KERBEROS,GSI
12/21/17 16:20:32 SECMAN: Auth methods: FS,KERBEROS,GSI
12/21/17 16:20:32 AUTHENTICATE: setting timeout for
<192.168.10.2:9618?addrs=192.168.10.2-9618+[--1]-9618&noUDP&sock=32054_f4d2_4>
to 20.
12/21/17 16:20:32 AUTHENTICATE: in authenticate( addr ==
'<192.168.10.2:9618?addrs=192.168.10.2-9618+[--1]-9618&noUDP&sock=32054_f4d2_4>',
methods == 'FS,KERBEROS,GSI')
12/21/17 16:20:32 AUTHENTICATE: can still try these methods: FS,KERBEROS,GSI
12/21/17 16:20:32 HANDSHAKE: in handshake(my_methods = 'FS,KERBEROS,GSI')
12/21/17 16:20:32 HANDSHAKE: handshake() - i am the client
12/21/17 16:20:32 HANDSHAKE: sending (methods == 100) to server
12/21/17 16:20:32 HANDSHAKE: server replied (method = 4)
12/21/17 16:20:32 AUTHENTICATE: will try to use 4 (FS)
12/21/17 16:20:32 AUTHENTICATE: do_authenticate is 1.
12/21/17 16:20:32 AUTHENTICATE_FS: used dir /tmp/FS_XXXQCWvEK, status: 0
12/21/17 16:20:32 AUTHENTICATE: method 4 (FS) failed.
12/21/17 16:20:32 AUTHENTICATE: can still try these methods: KERBEROS,GSI
12/21/17 16:20:32 HANDSHAKE: in handshake(my_methods = 'KERBEROS,GSI')
12/21/17 16:20:32 HANDSHAKE: handshake() - i am the client
12/21/17 16:20:32 HANDSHAKE: sending (methods == 96) to server
12/21/17 16:20:32 HANDSHAKE: server replied (method = 64)
12/21/17 16:20:32 AUTHENTICATE: will try to use 64 (KERBEROS)
12/21/17 16:20:32 AUTHENTICATE: do_authenticate is 1.
12/21/17 16:20:32 KERBEROS: krb5_unparse_name: host/bach@
12/21/17 16:20:32 KERBEROS: no user yet determined, will grab up to slash
12/21/17 16:20:32 KERBEROS: picked user: host
12/21/17 16:20:32 KERBEROS: remapping 'host' to 'condor'
12/21/17 16:20:32 unable to open map file (null), errno 22
12/21/17 16:20:32 KERBEROS: mapping realm  to domain .
12/21/17 16:20:32 Client is condor@
12/21/17 16:20:32 KERBEROS: Server principal is host/bach@
12/21/17 16:20:32 Acquiring credential for user
12/21/17 16:20:32 KERBEROS: No credentials cache found
12/21/17 16:20:32 AUTHENTICATE: method 64 (KERBEROS) failed.
12/21/17 16:20:32 AUTHENTICATE: can still try these methods: GSI
12/21/17 16:20:32 HANDSHAKE: in handshake(my_methods = 'GSI')
12/21/17 16:20:32 HANDSHAKE: handshake() - i am the client
12/21/17 16:20:32 HANDSHAKE: sending (methods == 32) to server
12/21/17 16:20:32 HANDSHAKE: server replied (method = 32)
12/21/17 16:20:32 AUTHENTICATE: will try to use 32 (GSI)
12/21/17 16:20:32 AUTHENTICATE: do_authenticate is 1.
12/21/17 16:20:32 authenticate_self_gss: acquiring self credentials
failed. Please check your Condor configuration file if this is a
server process. Or the user environment variable if this is a user
process.

GSS Major Status: General failure
GSS Minor Status Error Chain:
globus_gsi_gssapi: Error with GSI credential
globus_gsi_gssapi: Error with gss credential handle
globus_credential: Valid credentials could not be found in any of the
possible locations specified by the credential search order.
Valid credentials could not be found in any of the possible locations
specified by the credential search order.
Attempt 1
globus_credential: Error reading host credential
globus_sysconfig: Could not find a valid certificate file: The host
cert could not be found in:
1) env. var. X509_USER_CERT
2) /etc/grid-security/hostcert.pem
3) $GLOBUS_LOCATION/etc/hostcert.pem
4) $HOME/.globus/hostcert.pem

The host key could not be found in:
1) env. var. X509_USER_KEY
2) /etc/grid-security/hostkey.pem
3) $GLOBUS_LOCATION/etc/hostkey.pem
4) $HOME/.globus/hostkey.pem


Attempt 2
globus_credential: Error reading proxy credential
globus_sysconfig: Could not find a valid proxy certificate file location
globus_sysconfig: Error with key filename
globus_sysconfig: File does not exist: /tmp/x509up_u1104 is not a valid file
Attempt 3
globus_credential: Error reading user credential
globus_sysconfig: Error with certificate filename: The user cert could
not be found in:
1) env. var. X509_USER_CERT
2) $HOME/.globus/usercert.pem
3) $HOME/.globus/usercred.p12



12/21/17 16:20:32 authenticate: user creds not established
12/21/17 16:20:32 AUTHENTICATE: method 32 (GSI) failed.
12/21/17 16:20:32 AUTHENTICATE: can still try these methods:
12/21/17 16:20:32 HANDSHAKE: in handshake(my_methods = '')
12/21/17 16:20:32 HANDSHAKE: handshake() - i am the client
12/21/17 16:20:32 HANDSHAKE: sending (methods == 0) to server
12/21/17 16:20:32 HANDSHAKE: server replied (method = 0)
12/21/17 16:20:32 AUTHENTICATE: no available authentication methods succeeded!
12/21/17 16:20:32 SECMAN: required authentication with schedd at
<192.168.10.2:9618> failed, so aborting command QMGMT_WRITE_CMD.

ERROR: Failed to connect to queue manager bach.elucid.local
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using GSI
GSI:5003:Failed to authenticate.  Globus is reporting error
(851968:50).  There is probably a problem with your credentials.  (Did
you run grid-proxy-init?)
AUTHENTICATE:1004:Failed to authenticate using KERBEROS
AUTHENTICATE:1004:Failed to authenticate using FS
12/21/17 16:20:32 KEYCACHEENTRY: deleted: 0x1e88250
12/21/17 16:20:32 KEYCACHE: deleted: 0x1d32a40



On Tue, Dec 19, 2017 at 10:14 PM, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:
> Hi Larry,
>
> This is definitely an issue with the security subsystem, not the python API.  I suspect that you can reproduce it via the command line tools with something like:
>
> condor_submit -remote <schedd name> -pool <collector name> submit_file
>
> Sometimes it's a bit simpler to increase the logging via the CLI (the error messages don't always come back in a usable manner for the python API).
>
> If you can reproduce it with condor_submit, try:
>
> _condor_TOOL_DEBUG=D_SECURITY,D_FULLDEBUG condor_submit -debug -remote <schedd name> -pool <collector name> submit_file
>
> That should provide a full readout of the security handshake.
>
> The puzzling thing is that this line:
>
> use SECURITY : HOST_BASED
>
> in your server config (oh - did you do a condor_reconfig after the change?) should theoretically disable the attempts to do GSI-based security negotiation.  However, the logfiles clearly show it is being attempted.
>
> So -- this suggests something slightly wrong with the schedd configuration, but it's not clear what is wrong yet.
>
> Brian
>
>> On Dec 19, 2017, at 11:25 AM, Larry Martell <larry.martell@xxxxxxxxx> wrote:
>>
>> This is what was logged in SchedLog in the submit attempt. Note I have
>> these security related settings in my config file. Do I need other
>> settings to allow this to work?
>>
>> use SECURITY : HOST_BASED
>> ALLOW_WRITE = 192.168.*
>> ALLOW_READ = 192.168.*
>>
>>
>> 12/19/17 11:13:13 (pid:32123) authenticate_self_gss: acquiring self
>> credentials failed. Please check your Condor configuration file if
>> this is a server process. Or the user environment variable if this is
>> a user process.
>>
>> GSS Major Status: General failure
>> GSS Minor Status Error Chain:
>> globus_gsi_gssapi: Error with GSI credential
>> globus_gsi_gssapi: Error with gss credential handle
>> globus_credential: Valid credentials could not be found in any of the
>> possible locations specified by the credential search order.
>> Valid credentials could not be found in any of the possible locations
>> specified by the credential search order.
>> Attempt 1
>> globus_credential: Error reading host credential
>> globus_sysconfig: Could not find a valid certificate file: The host
>> cert could not be found in:
>> 1) env. var. X509_USER_CERT
>> 2) /etc/grid-security/hostcert.pem
>> 3) $GLOBUS_LOCATION/etc/hostcert.pem
>> 4) $HOME/.globus/hostcert.pem
>>
>> The host key could not be found in:
>> 1) env. var. X509_USER_KEY
>> 2) /etc/grid-security/hostkey.pem
>> 3) $GLOBUS_LOCATION/etc/hostkey.pem
>> 4) $HOME/.globus/hostkey.pem
>>
>>
>> Attempt 2
>> globus_credential: Error reading proxy credential
>> globus_sysconfig: Could not find a valid proxy certificate file location
>> globus_sysconfig: Error with key filename
>> globus_sysconfig: File does not exist: /tmp/x509up_u0 is not a valid file
>> Attempt 3
>> globus_credential: Error reading user credential
>> globus_sysconfig: Error with certificate filename: The user cert could
>> not be found in:
>> 1) env. var. X509_USER_CERT
>> 2) $HOME/.globus/usercert.pem
>> 3) $HOME/.globus/usercred.p12
>>
>>
>>
>> 12/19/17 11:13:13 (pid:32123) DC_AUTHENTICATE: authentication of
>> <192.168.10.15:45684> did not result in a valid mapped user name,
>> which is required for this command (1112 QMGMT_WRITE_CMD), so
>> aborting.
>> 12/19/17 11:13:13 (pid:32123) DC_AUTHENTICATE: reason for
>> authentication failure: AUTHENTICATE:1003:Failed to authenticate with
>> any method|AUTHENTICATE:1004:Failed to authenticate using
>> GSI|GSI:5003:Failed to authenticate.  Globus is reporting error
>> (851968:152).  There is probably a problem with your credentials.
>> (Did you run grid-proxy-init?)|AUTHENTICATE:1004:Failed to
>> authenticate using KERBEROS|AUTHENTICATE:1004:Failed to authenticate
>> using FS|FS:1004:Unable to lstat(/tmp/FS_XXX4oulm8)
>>
>>
>> On Tue, Dec 19, 2017 at 10:33 AM, Jason Patton <jpatton@xxxxxxxxxxx> wrote:
>>> I don't have a solution, but hopefully I can help get the ball rolling.
>>> Without modifying my schedd config, I tried doing a remote submit following
>>> the same steps, which failed with the same error. The error is a little
>>> misleading/light on details, it's likely an authentication problem from not
>>> being on the same system as the schedd. Doing essentially the same thing
>>> using the client tools gives more info:
>>>
>>>>>> schedd.submit(ad)
>>> Traceback (most recent call last):
>>>  File "<stdin>", line 1, in <module>
>>> RuntimeError: Failed to connect to schedd.
>>>
>>> $ condor_submit test.submit -remote condor-el7.test
>>> Submitting job(s)
>>> ERROR: Failed to connect to queue manager condor-el7.test
>>> AUTHENTICATE:1003:Failed to authenticate with any method
>>> AUTHENTICATE:1004:Failed to authenticate using GSI
>>> GSI:5003:Failed to authenticate.  Globus is reporting error (851968:50).
>>> There is probably a problem with your credentials.  (Did you run
>>> grid-proxy-init?)
>>> AUTHENTICATE:1004:Failed to authenticate using KERBEROS
>>> AUTHENTICATE:1004:Failed to authenticate using FS
>>>
>>> You should see more details in SchedLog on your submit host.
>>>
>>> Hopefully someone more knowledgable about setting up the schedd to accept
>>> remote job submissions can chime in. (ENABLE_SOAP and ENABLE_WEB_SERVER are
>>> probably not needed.)
>>>
>>> Jason
>>>
>>> On Tue, Dec 19, 2017 at 9:02 AM, Larry Martell <larry.martell@xxxxxxxxx>
>>> wrote:
>>>>
>>>> On Tue, Dec 19, 2017 at 9:29 AM, Larry Martell <larry.martell@xxxxxxxxx>
>>>> wrote:
>>>>> I am doing this:
>>>>>
>>>>> import htcondor
>>>>> import classad
>>>>> condor_host = '192.168.10.2'
>>>>> coll = htcondor.Collector(condor_host)
>>>>> schedd_ad = coll.locate(htcondor.DaemonTypes.Schedd)
>>>>> schedd = htcondor.Schedd(schedd_ad)
>>>>> ad = classad.ClassAd()
>>>>>
>>>>> # set up ad
>>>>>
>>>>> id = schedd.submit(ad)
>>>>>
>>>>> RuntimeError: 'Failed to connect to schedd.'
>>>>>
>>>>> On 192.168.10.2:
>>>>>
>>>>> 4 S condor     32054       1  0  80   0 - 18610 poll_s Dec12 ?
>>>>> 00:00:15 /usr/sbin/condor_master -f
>>>>> 4 S root       32112   32054  0  80   0 -  6652 poll_s Dec12 ?
>>>>> 00:07:51 condor_procd -A /var/run/condor/procd_pipe -L
>>>>> /var/log/condor/ProcLog -R 1000000 -S 60 -C 986
>>>>> 4 S condor     32113   32054  0  80   0 - 13531 poll_s Dec12 ?
>>>>> 00:00:44 condor_shared_port -f
>>>>> 4 S condor     32117   32054  0  80   0 - 20511 poll_s Dec12 ?
>>>>> 00:07:46 condor_collector -f
>>>>> 4 S condor     32122   32054  0  80   0 - 15856 poll_s Dec12 ?
>>>>> 00:31:40 condor_negotiator -f
>>>>> 4 S condor     32123   32054  0  80   0 - 18808 poll_s Dec12 ?
>>>>> 00:00:31 condor_schedd -f
>>>>>
>>>>> From the machine running the python code:
>>>>>
>>>>> $ nmap -p 9618 192.168.10.2
>>>>>
>>>>> Starting Nmap 6.40 ( http://nmap.org ) at 2017-12-19 09:28 EST
>>>>> Nmap scan report for 192.168.10.2
>>>>> Host is up (0.00018s latency).
>>>>> PORT     STATE SERVICE
>>>>> 9618/tcp open  condor
>>>>>
>>>>> Am I doing something wrong or missing something?
>>>>
>>>> Also let me add I have these settings in the config file:
>>>>
>>>> ENABLE_SOAP = True
>>>> ENABLE_WEB_SERVER = True
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
>
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
Follow-Ups:
- Re: [HTCondor-users] submitting jobs with API
  - From: Larry Martell
References:
- [HTCondor-users] submitting jobs with API
  - From: Larry Martell
- Re: [HTCondor-users] submitting jobs with API
  - From: Larry Martell
- Re: [HTCondor-users] submitting jobs with API
  - From: Jason Patton
- Re: [HTCondor-users] submitting jobs with API
  - From: Larry Martell
- Re: [HTCondor-users] submitting jobs with API
  - From: Brian Bockelman
Prev by Date: Re: [HTCondor-users] Arguments to condor_submit
Next by Date: Re: [HTCondor-users] core.STARTER generated
Previous by thread: Re: [HTCondor-users] submitting jobs with API
Next by thread: Re: [HTCondor-users] submitting jobs with API
Index(es):
- Date
- Thread
Mailing List Archives

Public Access

Re: [HTCondor-users] submitting jobs with API
