Re: [Condor-users] Globus - GT 4.0.1: GLOBUS_SCRATCH_DIR problem?

["duane waktu" <duawaktu@xxxxxxxxxxx>]

Jaime,

FYI:

======
Source
======
GT 4.0
<src_host>: source hostname
<src_ip>: source IP-address

======
Dest
======
GT 4.0.1
<dest_host>: dest hostname
<dest_ip>: dest IP-address


Below is the output generated after I added the debug mode for GridManager 
(very long!):

==========================================================

9/21 09:38:54 PASSWD_CACHE_REFRESH is undefined, using default value of 300

9/21 09:38:54 ******************************************************
9/21 09:38:54 ** condor_gridmanager (CONDOR_GRIDMANAGER) STARTING UP
9/21 09:38:54 ** /usr/local/condor/sbin/condor_gridmanager
9/21 09:38:54 ** $CondorVersion: 6.7.8 Jun  9 2005 $
9/21 09:38:54 ** $CondorPlatform: I386-LINUX_RH9 $
9/21 09:38:54 ** PID = 23640
9/21 09:38:54 ******************************************************
9/21 09:38:54 Using config file: /usr/local/condor/etc/condor_config
9/21 09:38:54 Using local config files: 
/usr/local/condor/local.<src_host>/condor_config.local
9/21 09:38:54 DaemonCore: Command Socket at <<src_host>:57362>
9/21 09:38:54 SEC_DEFAULT_SESSION_DURATION is undefined, using default value 
of 3600
9/21 09:38:54 Welcome to the all-singing, all dancing, "amazing" 
GridManager!
9/21 09:38:54 [23640] GRIDMANAGER_TIMEOUT_MULTIPLIER is undefined, using 
default value of 0
9/21 09:38:54 [23640] GRIDMANAGER_CHECKPROXY_INTERVAL is undefined, using 
default value of 600
9/21 09:38:54 [23640] GRIDMANAGER_MINIMUM_PROXY_TIME is undefined, using 
default value of 180
9/21 09:38:54 [23640] GRIDMANAGER_CONTACT_SCHEDD_DELAY is undefined, using 
default value of 5
9/21 09:38:54 [23640] GRIDMANAGER_CHECKPROXY_INTERVAL is undefined, using 
default value of 600
9/21 09:38:54 [23640] GRIDMANAGER_MINIMUM_PROXY_TIME is undefined, using 
default value of 180
9/21 09:38:54 [23640] GRIDMANAGER_GAHPCLIENT_DEBUG is undefined, using 
default value of True
9/21 09:38:54 [23640] GRIDMANAGER_GAHPCLIENT_DEBUG_SIZE is undefined, using 
default value of 0
9/21 09:38:54 [23640] GAHP_USE_XML_CLASSADS is undefined, using default 
value of False
9/21 09:38:54 [23640] GRIDMANAGER_MAX_PENDING_REQUESTS is undefined, using 
default value of 50
9/21 09:38:54 [23640] MIRROR_JOB_POLL_INTERVAL is undefined, using default 
value of 300
9/21 09:38:54 [23640] GRIDMANAGER_GAHP_CALL_TIMEOUT is undefined, using 
default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_CONNECT_FAILURE_RETRY_COUNT is undefined, 
using default value of 3
9/21 09:38:54 [23640] MIRROR_JOB_LEASE_INTERVAL is undefined, using default 
value of 1800
9/21 09:38:54 [23640] INFN_JOB_POLL_INTERVAL is undefined, using default 
value of 300
9/21 09:38:54 [23640] GRIDMANAGER_GAHP_CALL_TIMEOUT is undefined, using 
default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_CONNECT_FAILURE_RETRY_COUNT is undefined, 
using default value of 3
9/21 09:38:54 [23640] CONDOR_JOB_POLL_INTERVAL is undefined, using default 
value of 300
9/21 09:38:54 [23640] GRIDMANAGER_GAHP_CALL_TIMEOUT is undefined, using 
default value of 28800
9/21 09:38:54 [23640] GRIDMANAGER_GAHP_CALL_TIMEOUT_CONDOR is undefined, 
using default value of 28800
9/21 09:38:54 [23640] GRIDMANAGER_CONNECT_FAILURE_RETRY_COUNT is undefined, 
using default value of 3
9/21 09:38:54 [23640] GRIDMANAGER_JOB_PROBE_INTERVAL is undefined, using 
default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_RESOURCE_PROBE_INTERVAL is undefined, 
using default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_GAHP_CALL_TIMEOUT is undefined, using 
default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_CONNECT_FAILURE_RETRY_COUNT is undefined, 
using default value of 3
9/21 09:38:54 [23640] GRIDMANAGER_JOB_PROBE_INTERVAL is undefined, using 
default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_RESOURCE_PROBE_INTERVAL is undefined, 
using default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_GAHP_CALL_TIMEOUT is undefined, using 
default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_CONNECT_FAILURE_RETRY_COUNT is undefined, 
using default value of 3
9/21 09:38:54 [23640] GRIDMANAGER_JOB_PROBE_INTERVAL is undefined, using 
default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_RESOURCE_PROBE_INTERVAL is undefined, 
using default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_GAHP_CALL_TIMEOUT is undefined, using 
default value of 300
9/21 09:38:54 [23640] GRIDMANAGER_CONNECT_FAILURE_RETRY_COUNT is undefined, 
using default value of 3
9/21 09:38:54 [23640] ENABLE_GRID_MONITOR is undefined, using default value 
of False
9/21 09:38:54 [23640] Checking proxies
9/21 09:38:54 [23640] Getting monitoring info for pid 23640
9/21 09:38:55 [23640] DaemonCore: in SendAliveToParent()
9/21 09:38:55 [23640] DaemonCore: attempting to connect to 
'<<src_ip>:57348>'
9/21 09:38:55 [23640] GRIDMANAGER_TIMEOUT_MULTIPLIER is undefined, using 
default value of 0
9/21 09:38:55 [23640] condor_read(): nfds=7
9/21 09:38:55 [23640] condor_read(): nfound=1
9/21 09:38:55 [23640] condor_read(): nfds=7
9/21 09:38:55 [23640] condor_read(): nfound=1
9/21 09:38:55 [23640] condor_read(): nfds=7
9/21 09:38:55 [23640] condor_read(): nfound=1
9/21 09:38:55 [23640] condor_read(): nfds=7
9/21 09:38:55 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] Received ADD_JOBS signal
9/21 09:38:57 [23640] in doContactSchedd()
9/21 09:38:57 [23640] GRIDMANAGER_TIMEOUT_MULTIPLIER is undefined, using 
default value of 0
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] AUTHENTICATE_FS: used file /tmp/qmgr_9pcCQY, status: 1
9/21 09:38:57 [23640] querying for new jobs
9/21 09:38:57 [23640] Using constraint 
((Owner=?="my_username"&&JobUniverse==9)) && ((Matched =!= FALSE && 
JobStatus != 5) || Managed =?= TRUE)
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] Using job type GT4 for job 137.0
9/21 09:38:57 [23640] GRIDMANAGER_MAX_PENDING_REQUESTS is undefined, using 
default value of 50
9/21 09:38:57 [23640] Found job 137.0 --- inserting
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] Fetched 1 new job ads from schedd
9/21 09:38:57 [23640] querying for removed/held jobs
9/21 09:38:57 [23640] Using constraint 
((Owner=?="my_username"&&JobUniverse==9)) && (JobStatus == 3 || JobStatus == 
4 || (JobStatus == 5 && Managed =?= TRUE))
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] Fetched 0 job ads from schedd
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] condor_read(): nfds=6
9/21 09:38:57 [23640] condor_read(): nfound=1
9/21 09:38:57 [23640] leaving doContactSchedd()
9/21 09:38:57 [23640] (137.0) doEvaluateState called: gmState GM_INIT, 
globusState 32
9/21 09:38:57 [23640] GAHP server pid = 23643
9/21 09:38:58 [23640] GAHP server version: $GahpVersion: 1.4.0 Jun 02 2005 
GT4 GAHP (GT-4.0.0) $
9/21 09:38:58 [23640] GAHP[23643] <- 'COMMANDS'
9/21 09:38:58 [23640] GAHP[23643] -> 'S' 'ASYNC_MODE_OFF' 'ASYNC_MODE_ON' 
'CACHE_PROXY_FROM_FILE' 'COMMANDS' 'GASS_SERVER_INIT' 
'GT4_DELEGATE_CREDENTIAL' 'GT4_GENERATE_SUBMIT_ID' 'GT4_GRAM_CALLBACK_ALLOW' 
'GT4_GRAM_JOB_CALLBACK_REGISTER' 'GT4_GRAM_JOB_DESTROY' 'GT4_GRAM_JOB_START' 
'GT4_GRAM_JOB_STATUS' 'GT4_GRAM_JOB_SUBMIT' 'GT4_GRAM_PING' 
'GT4_REFRESH_CREDENTIAL' 'GT4_SET_TERMINATION_TIME' 'INITIALIZE_FROM_FILE' 
'QUIT' 'REFRESH_PROXY_FROM_FILE' 'RESPONSE_PREFIX' 'RESULTS' 'UNCACHE_PROXY' 
'USE_CACHED_PROXY' 'VERSION'
9/21 09:38:58 [23640] GAHP[23643] <- 'RESPONSE_PREFIX GAHP:'
9/21 09:38:58 [23640] GAHP[23643] -> 'S'
9/21 09:38:58 [23640] GAHP[23643] <- 'ASYNC_MODE_ON'
9/21 09:38:58 [23640] GAHP[23643] -> 'S'
9/21 09:38:58 [23640] GAHP[23643] <- 'INITIALIZE_FROM_FILE 
/tmp/condor_g_scratch.0x8779000.23606/master_proxy.2'
9/21 09:38:59 [23640] GAHP[23643] -> 'S'
9/21 09:38:59 [23640] GAHP[23643] <- 'CACHE_PROXY_FROM_FILE 2 
/tmp/condor_g_scratch.0x8779000.23606/master_proxy.2'
9/21 09:38:59 [23640] GAHP[23643] -> 'S'
9/21 09:38:59 [23640] GAHP[23643] <- 'CACHE_PROXY_FROM_FILE 1 
/tmp/x509up_u1001'
9/21 09:38:59 [23640] GAHP[23643] -> 'S'
9/21 09:38:59 [23640] GAHP[23643] <- 'GT4_GRAM_CALLBACK_ALLOW 2'
9/21 09:39:00 [23640] GAHP[23643] -> 'S' '1'
9/21 09:39:00 [23640] (137.0) gm state change: GM_INIT -> GM_START
9/21 09:39:00 [23640] (137.0) gm state change: GM_START -> GM_CLEAR_REQUEST
9/21 09:39:00 [23640] (137.0) gm state change: GM_CLEAR_REQUEST -> 
GM_UNSUBMITTED
9/21 09:39:00 [23640] (137.0) gm state change: GM_UNSUBMITTED -> 
GM_DELEGATE_PROXY
9/21 09:39:00 [23640] *** getDelegationURI(/tmp/x509up_u1001)
9/21 09:39:00 [23640]     creating new ProxyDelegation
9/21 09:39:00 [23640] GAHP[23643] <- 'GT4_GRAM_PING 3 
https://<dest_host>:8443'
9/21 09:39:00 [23640] GAHP[23643] -> 'S'
9/21 09:39:00 [23640] *** checkDelegation()
9/21 09:39:00 [23640]     new delegation
9/21 09:39:00 [23640] GAHP[23643] <- 'USE_CACHED_PROXY 1'
9/21 09:39:00 [23640] GAHP[23643] -> 'R'
9/21 09:39:00 [23640] GAHP[23643] -> 'S'
9/21 09:39:00 [23640] GAHP[23643] <- 'GT4_DELEGATE_CREDENTIAL 4 
https://<dest_host>:8443/wsrf/services/DelegationFactoryService'
9/21 09:39:00 [23640] GAHP[23643] -> 'S'
9/21 09:39:00 [23640] GAHP[23643] <- 'RESULTS'
9/21 09:39:00 [23640] GAHP[23643] -> 'S' '1'
9/21 09:39:00 [23640] GAHP[23643] -> '3' '0' 'NULL'
9/21 09:39:00 [23640] resource https://<dest_host>:8443 is now up
9/21 09:39:00 [23640] (137.0) doEvaluateState called: gmState 
GM_DELEGATE_PROXY, globusState 32
9/21 09:39:00 [23640] *** getDelegationURI(/tmp/x509up_u1001)
9/21 09:39:00 [23640]     found ProxyDelegation
9/21 09:39:03 [23640] GAHP[23643] <- 'RESULTS'
9/21 09:39:03 [23640] GAHP[23643] -> 'R'
9/21 09:39:03 [23640] GAHP[23643] -> 'S' '1'
9/21 09:39:03 [23640] GAHP[23643] -> '4' '0' 
'https://<dest_ip>:8443/wsrf/services/DelegationService?30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c' 
'NULL'
9/21 09:39:03 [23640] *** checkDelegation()
9/21 09:39:03 [23640]     new delegation
9/21 09:39:03 [23640]       
https://<dest_ip>:8443/wsrf/services/DelegationService?30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c
9/21 09:39:03 [23640]     signalling jobs for 
https://<dest_ip>:8443/wsrf/services/DelegationService?30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c
9/21 09:39:03 [23640] (137.0) doEvaluateState called: gmState 
GM_DELEGATE_PROXY, globusState 32
9/21 09:39:03 [23640] *** getDelegationURI(/tmp/x509up_u1001)
9/21 09:39:03 [23640]     found ProxyDelegation
9/21 09:39:03 [23640] (137.0) gm state change: GM_DELEGATE_PROXY -> 
GM_GENERATE_ID
9/21 09:39:03 [23640] GAHP[23643] <- 'GT4_GENERATE_SUBMIT_ID 5 '
9/21 09:39:03 [23640] GAHP[23643] -> 'S'
9/21 09:39:03 [23640] GAHP[23643] <- 'RESULTS'
9/21 09:39:03 [23640] GAHP[23643] -> 'R'
9/21 09:39:03 [23640] GAHP[23643] -> 'S' '1'
9/21 09:39:03 [23640] GAHP[23643] -> '5' 
'uuid:12cbfda0-2aa5-11da-bb38-c32612bdfbb7'
9/21 09:39:03 [23640] (137.0) doEvaluateState called: gmState 
GM_GENERATE_ID, globusState 32
9/21 09:39:03 [23640] (137.0) gm state change: GM_GENERATE_ID -> 
GM_SUBMIT_ID_SAVE
9/21 09:39:03 [23640] in doContactSchedd()
9/21 09:39:03 [23640] GRIDMANAGER_TIMEOUT_MULTIPLIER is undefined, using 
default value of 0
9/21 09:39:03 [23640] SEC_DEBUG_PRINT_KEYS is undefined, using default value 
of False
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] AUTHENTICATE_FS: used file /tmp/qmgr_rDXlF3, status: 1
9/21 09:39:03 [23640] querying for removed/held jobs
9/21 09:39:03 [23640] Using constraint 
((Owner=?="my_username"&&JobUniverse==9)) && (JobStatus == 3 || JobStatus == 
4 || (JobStatus == 5 && Managed =?= TRUE))
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] Fetched 0 job ads from schedd
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] Updating classad values for 137.0:
9/21 09:39:03 [23640]    GlobusDelegationUri = 
"https://<dest_ip>:8443/wsrf/services/DelegationService?30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c"
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640]    GlobusSubmitId = 
"uuid:12cbfda0-2aa5-11da-bb38-c32612bdfbb7"
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] condor_read(): nfds=6
9/21 09:39:03 [23640] condor_read(): nfound=1
9/21 09:39:03 [23640] leaving doContactSchedd()
9/21 09:39:03 [23640] (137.0) doEvaluateState called: gmState 
GM_SUBMIT_ID_SAVE, globusState 32
9/21 09:39:03 [23640] (137.0) gm state change: GM_SUBMIT_ID_SAVE -> 
GM_SUBMIT



<lots of blank lines here, before continuing with the line below..>



9/21 09:39:03 [23640] (137.0) doEvaluateState called: gmState 
GM_SUBMIT_ID_SAVE, globusState 32
9/21 09:39:03 [23640] (137.0) gm state change: GM_SUBMIT_ID_SAVE -> 
GM_SUBMIT
9/21 09:39:03 [23640] GAHP[23643] <- 'GT4_GRAM_JOB_SUBMIT 6 
uuid:12cbfda0-2aa5-11da-bb38-c32612bdfbb7 https://<dest_host>:8443 Fork 1 
<job><executable>/${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/hostname</executable><directory>/${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/</directory><stdout>/${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7//ad.output</stdout><stderr>/${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7//ad.error</stderr><fileStageIn><maxAttempts>5</maxAttempts><transferCredentialEndpoint\ 
xsi:type="ns1:EndpointReferenceType"\ 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"\ 
xmlns:ns1="http://schemas.xmlsoap.org/ws/2004/03/addressing";><ns1:Address\ 
xsi:type="ns1:AttributedURI">https://<dest_ip>:8443/wsrf/services/DelegationService</ns1:Address><ns1:ReferenceProperties\ 
xsi:type="ns1:ReferencePropertiesType"><ns1:DelegationKey\ 
xmlns:ns1="http://www.globus.org/08/2004/delegationService";>30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c</ns1:DelegationKey></ns1:ReferenceProperties><ns1:ReferenceParameters\ 
xsi:type="ns1:ReferenceParametersType"/></transferCredentialEndpoint><transfer><sourceUrl>gsiftp://<src_host>/tmp/condor_g_scratch.0x8779000.23606/empty_dir_u501/</sourceUrl><destinationUrl>file:///${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/</destinationUrl></transfer><transfer><sourceUrl>gsiftp://<src_host>/bin/hostname</sourceUrl><destinationUrl>file:///${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/hostname</destinationUrl></transfer></fileStageIn><fileStageOut><maxAttempts>5</maxAttempts><transferCredentialEndpoint\ 
xsi:type="ns1:EndpointReferenceType"\ 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"\ 
xmlns:ns1="http://schemas.xmlsoap.org/ws/2004/03/addressing";><ns1:Address\ 
xsi:type="ns1:AttributedURI">https://<dest_ip>:8443/wsrf/services/DelegationService</ns1:Address><ns1:ReferenceProperties\ 
xsi:type="ns1:ReferencePropertiesType"><ns1:DelegationKey\ 
xmlns:ns1="http://www.globus.org/08/2004/delegationService";>30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c</ns1:DelegationKey></ns1:ReferenceProperties><ns1:ReferenceParameters\ 
xsi:type="ns1:ReferenceParametersType"/></transferCredentialEndpoint><transfer><sourceUrl>file:///${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/ad.output</sourceUrl><destinationUrl>gsiftp://<src_host>/tmp/ad.output</destinationUrl></transfer><transfer><sourceUrl>file:///${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38 
c32612bdfbb7/ad.error</sourceUrl><destinationUrl>gsiftp://<src_host>/tmp/ad.error</destinationUrl></transfer></fileStageOut><fileCleanUp><transferCredentialEndpoint\ 
xsi:type="ns1:EndpointReferenceType"\ 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"\ 
xmlns:ns1="http://schemas.xmlsoap.org/ws/2004/03/addressing";><ns1:Address\ 
xsi:type="ns1:AttributedURI">https://<dest_ip>:8443/wsrf/services/DelegationService</ns1:Address><ns1:ReferenceProperties\ 
xsi:type="ns1:ReferencePropertiesType"><ns1:DelegationKey\ 
xmlns:ns1="http://www.globus.org/08/2004/delegationService";>30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c</ns1:DelegationKey></ns1:ReferenceProperties><ns1:ReferenceParameters\ 
xsi:type="ns1:ReferenceParametersType"/></transferCredentialEndpoint><deletion><file>file:///${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/</file></deletion></fileCleanUp><jobCredentialEndpoint\ 
xsi:type="ns1:EndpointReferenceType"\ 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"\ 
xmlns:ns1="http://schemas.xmlsoap.org/ws/2004/03/addressing";><ns1:Address\ 
xsi:type="ns1:AttributedURI">https://148.100.51.29:8443/wsrf/services/DelegationService</ns1:Address><ns1:ReferenceProperties\ 
xsi:type="ns1:ReferencePropertiesType"><ns1:DelegationKey\ 
xmlns:ns1="http://www.globus.org/08/2004/delegationService";>30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c</ns1:DelegationKey></ns1:ReferenceProperties><ns1:ReferenceParameters\ 
xsi:type="ns1:ReferenceParametersType"/></jobCredentialEndpoint><stagingCredentialEndpoint\ 
xsi:type="ns1:EndpointReferenceType"\ 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"\ 
xmlns:ns1="http://schemas.xmlsoap.org/ws/2004/03/addressing";><ns1:Address\ 
xsi:type="ns1:AttributedURI">https://<dest_ip>:8443/wsrf/services/DelegationService</ns1:Address><ns1:ReferenceProperties\ 
xsi:type="ns1:ReferencePropertiesType"><ns1:DelegationKey\ 
xmlns:ns1="http://www.globus.org/08/2004/delegationService";>30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c</ns1:DelegationKey></ns1:ReferenceProperties><ns1:ReferenceParameters\ 
xsi:type="ns1:ReferenceParametersType"/></stagingCredentialEndpoint><holdState>StageIn</holdState></job> 
NULL'
9/21 09:39:03 [23640] GAHP[23643] -> 'S'
9/21 09:39:09 [23640] DaemonCore::IsPidAlive(): kill returned EPERM, 
assuming pid 23606 is alive.
9/21 09:39:58 [23640] GAHP[23643] <- 'RESULTS'
9/21 09:39:58 [23640] GAHP[23643] -> 'S' '0'
9/21 09:40:58 [23640] GAHP[23643] <- 'RESULTS'
9/21 09:40:58 [23640] GAHP[23643] -> 'S' '0'
9/21 09:41:04 [23640] GAHP[23643] (stderr) -> ERROR Submitting Job: ; nested 
exception is:
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   
java.net.SocketTimeoutException: Read timed out
9/21 09:41:04 [23640] GAHP[23643] (stderr) -> AxisFault
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->  faultCode: 
{http://schemas.xmlsoap.org/soap/envelope/}Server.userException
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->  faultSubcode:
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->  faultString: 
java.net.SocketTimeoutException: Read timed out
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->  faultActor:
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->  faultNode:
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->  faultDetail:
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   
{http://xml.apache.org/axis/}stackTrace:java.net.SocketTimeoutException: 
Read timed out
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
java.net.SocketInputStream.socketRead0(Native Method)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
java.net.SocketInputStream.read(SocketInputStream.java:129)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.gsi.gssapi.SSLUtil.read(SSLUtil.java:37)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.gsi.gssapi.net.impl.GSIGssInputStream.readToken(GSIGssInputStream.java:64)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.gsi.gssapi.net.impl.GSIGssInputStream.readMsg(GSIGssInputStream.java:39)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.gsi.gssapi.net.GssInputStream.hasData(GssInputStream.java:81)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.gsi.gssapi.net.GssInputStream.read(GssInputStream.java:55)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:545)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:140)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.Call.invokeEngine(Call.java:2726)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.Call.invoke(Call.java:2709)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.Call.invoke(Call.java:2385)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.Call.invoke(Call.java:2308)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.Call.invoke(Call.java:1765)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.exec.generated.bindings.ManagedJobFactoryPortTypeSOAPBindingStub.createManagedJob(ManagedJobFactoryPortTypeSOAPBindingStub.java:1157)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.exec.client.GramJob.createJobEndpoint(GramJob.java:973)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.exec.client.GramJob.submit(GramJob.java:449)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
condor.gahp.gt4.Gt4GramJobSubmitHandler$MyTask.submitJob(Gt4GramJobSubmitHandler.java:245)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
condor.gahp.gt4.Gt4GramJobSubmitHandler$MyTask.run(Gt4GramJobSubmitHandler.java:191)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
java.lang.Thread.run(Thread.java:595)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   
{http://xml.apache.org/axis/}hostname:ucf-6.linuxclass.marist.edu
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->
9/21 09:41:04 [23640] GAHP[23643] (stderr) -> 
java.net.SocketTimeoutException: Read timed out
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:144)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.Call.invokeEngine(Call.java:2726)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.Call.invoke(Call.java:2709)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.Call.invoke(Call.java:2385)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.Call.invoke(Call.java:2308)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.client.Call.invoke(Call.java:1765)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.exec.generated.bindings.ManagedJobFactoryPortTypeSOAPBindingStub.createManagedJob(ManagedJobFactoryPortTypeSOAPBindingStub.java:1157)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.exec.client.GramJob.createJobEndpoint(GramJob.java:973)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.exec.client.GramJob.submit(GramJob.java:449)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
condor.gahp.gt4.Gt4GramJobSubmitHandler$MyTask.submitJob(Gt4GramJobSubmitHandler.java:245)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
condor.gahp.gt4.Gt4GramJobSubmitHandler$MyTask.run(Gt4GramJobSubmitHandler.java:191)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
java.lang.Thread.run(Thread.java:595)
9/21 09:41:04 [23640] GAHP[23643] (stderr) -> Caused by: 
java.net.SocketTimeoutException: Read timed out
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
java.net.SocketInputStream.socketRead0(Native Method)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
java.net.SocketInputStream.read(SocketInputStream.java:129)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.gsi.gssapi.SSLUtil.read(SSLUtil.java:37)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.gsi.gssapi.net.impl.GSIGssInputStream.readToken(GSIGssInputStream.java:64)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.gsi.gssapi.net.impl.GSIGssInputStream.readMsg(GSIGssInputStream.java:39)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.gsi.gssapi.net.GssInputStream.hasData(GssInputStream.java:81)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.globus.gsi.gssapi.net.GssInputStream.read(GssInputStream.java:55)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.transport.http.HTTPSender.readHeadersFromSocket(HTTPSender.java:545)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   at 
org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:140)
9/21 09:41:04 [23640] GAHP[23643] (stderr) ->   ... 15 more
9/21 09:41:04 [23640] GAHP[23643] <- 'RESULTS'
9/21 09:41:04 [23640] GAHP[23643] -> 'R'
9/21 09:41:04 [23640] GAHP[23643] -> 'S' '1'
9/21 09:41:04 [23640] GAHP[23643] -> '6' '1' 'NULL' 
'java.net.SocketTimeoutException: Read timed out'
9/21 09:41:04 [23640] (137.0) doEvaluateState called: gmState GM_SUBMIT, 
globusState 32
9/21 09:41:04 [23640] (137.0) gmState GM_SUBMIT, globusState 32: 
gt4_gram_client_job_create() failed


<lots of blank lines here, before continuing with the line below...>


9/21 09:41:04 [23640] (137.0) doEvaluateState called: gmState GM_SUBMIT, 
globusState 32
9/21 09:41:04 [23640] (137.0) gmState GM_SUBMIT, globusState 32: 
gt4_gram_client_job_create() failed
9/21 09:41:04 [23640] (137.0)    
RSL='<job><executable>/${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/hostname</executable><directory>/${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/</directory><stdout>/${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7//ad.output</stdout><stderr>/${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7//ad.error</stderr><fileStageIn><maxAttempts>5</maxAttempts><transferCredentialEndpoint 
xsi:type="ns1:EndpointReferenceType" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xmlns:ns1="http://schemas.xmlsoap.org/ws/2004/03/addressing";><ns1:Address 
xsi:type="ns1:AttributedURI">https://<dest_ip>:8443/wsrf/services/DelegationService</ns1:Address><ns1:ReferenceProperties 
xsi:type="ns1:ReferencePropertiesType"><ns1:DelegationKey 
xmlns:ns1="http://www.globus.org/08/2004/delegationService";>30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c</ns1:DelegationKey></ns1:ReferenceProperties><ns1:ReferenceParameters 
xsi:type="ns1:ReferenceParametersType"/></transferCredentialEndpoint><transfer><sourceUrl>gsiftp://<src_host>/tmp/condor_g_scratch.0x8779000.23606/empty_dir_u501/</sourceUrl><destinationUrl>file:///${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/</destinationUrl></transfer><transfer><sourceUrl>gsiftp://<src_host>/bin/hostname</sourceUrl><destinationUrl>file:///${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/hostname</destinationUrl></transfer></fileStageIn><fileStageOut><maxAttempts>5</maxAttempts><transferCredentialEndpoint 
xsi:type="ns1:EndpointReferenceType" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xmlns:ns1="http://schemas.xmlsoap.org/ws/2004/03/addressing";><ns1:Address 
xsi:type="ns1:AttributedURI">https://148.100.51.29:8443/wsrf/services/DelegationService</ns1:Address><ns1:ReferenceProperties 
xsi:type="ns1:ReferencePropertiesType"><ns1:DelegationKey 
xmlns:ns1="http://www.globus.org/08/2004/delegationService";>30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c</ns1:DelegationKey></ns1:ReferenceProperties><ns1:ReferenceParameters 
xsi:type="ns1:ReferenceParametersType"/></transferCredentialEndpoint><transfer><sourceUrl>file:///${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/ad.output</sourceUrl><destinationUrl>gsiftp://<src_host>/tmp/ad.output</destinationUrl></transfer><transfer><sourceUrl>file:///${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/ad.error</sourceUrl><destinationUrl>gsiftp://<src_host>/tmp/ad.error</destinationUrl></transfer></fileStageOut><fileCleanUp><transferCredentialEndpoint 
xsi:type="ns1:EndpointReferenceType" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xmlns:ns1="http://schemas.xmlsoap.org/ws/2004/03/addressing";><ns1:Address 
xsi:type="ns1:AttributedURI">https://<dest_ip>:8443/wsrf/services/DelegationService</ns1:Address><ns1:ReferenceProperties 
xsi:type="ns1:ReferencePropertiesType"><ns1:DelegationKey 
xmlns:ns1="http://www.globus.org/08/2004/delegationService";>30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c</ns1:DelegationKey></ns1:ReferenceProperties><ns1:ReferenceParameters 
xsi:type="ns1:ReferenceParametersType"/></transferCredentialEndpoint><deletion><file>file:///${GLOBUS_SCRATCH_DIR}/job_12cbfda0-2aa5-11da-bb38-c32612bdfbb7/</file></deletion></fileCleanUp><jobCredentialEndpoint 
xsi:type="ns1:EndpointReferenceType" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xmlns:ns1="http://schemas.xmlsoap.org/ws/2004/03/addressing";><ns1:Address 
xsi:type="ns1:AttributedURI">https://<dest_ip>:8443/wsrf/services/DelegationService</ns1:Address><ns1:ReferenceProperties 
xsi:type="ns1:ReferencePropertiesType"><ns1:DelegationKey 
xmlns:ns1="http://www.globus.org/08/2004/delegationService";>30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c</ns1:DelegationKey></ns1:ReferenceProperties><ns1:ReferenceParameters 
xsi:type="ns1:ReferenceParametersType"/></jobCredentialEndpoint><stagingCredentialEndpoint 
xsi:type="ns1:EndpointReferenceType" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; 
xmlns:ns1="http://schemas.xmlsoap.org/ws/2004/03/addressing";><ns1:Address 
xsi:type="ns1:AttributedURI">https://148.100.51.29:8443/wsrf/services/DelegationService</ns1:Address><ns1:ReferenceProperties 
xsi:type="ns1:ReferencePropertiesType"><ns1:DelegationKey 
xmlns:ns1="http://www.globus.org/08/2004/delegationService";>30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c</ns1:DelegationKey></ns1:ReferenceProperties><ns1:ReferenceParameters 
xsi:type="ns1:ReferenceParametersType"/></stagingCredentialEndpoint><holdState>StageIn</holdState></job>'
9/21 09:41:04 [23640] ENABLE_USERLOG_LOCKING is undefined, using default 
value of True
9/21 09:41:04 [23640] (137.0) Writing submit-failed record to user logfile
9/21 09:41:04 [23640] (137.0) gm state change: GM_SUBMIT -> GM_UNSUBMITTED
9/21 09:41:04 [23640] (137.0) gm state change: GM_UNSUBMITTED -> 
GM_DELEGATE_PROXY
9/21 09:41:04 [23640] (137.0) gm state change: GM_DELEGATE_PROXY -> 
GM_GENERATE_ID
9/21 09:41:04 [23640] (137.0) gm state change: GM_GENERATE_ID -> 
GM_SUBMIT_ID_SAVE
9/21 09:41:04 [23640] (137.0) gm state change: GM_SUBMIT_ID_SAVE -> 
GM_SUBMIT
9/21 09:41:04 [23640] (137.0) gm state change: GM_SUBMIT -> GM_HOLD
9/21 09:41:04 [23640] ENABLE_USERLOG_LOCKING is undefined, using default 
value of True
9/21 09:41:04 [23640] (137.0) Writing hold record to user logfile
9/21 09:41:04 [23640] (137.0) gm state change: GM_HOLD -> GM_DELETE
9/21 09:41:04 [23640] in doContactSchedd()
9/21 09:41:04 [23640] GRIDMANAGER_TIMEOUT_MULTIPLIER is undefined, using 
default value of 0
9/21 09:41:04 [23640] SEC_DEBUG_PRINT_KEYS is undefined, using default value 
of False
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] AUTHENTICATE_FS: used file /tmp/qmgr_v9Bf5e, status: 1
9/21 09:41:04 [23640] querying for removed/held jobs
9/21 09:41:04 [23640] Using constraint 
((Owner=?="my_username"&&JobUniverse==9)) && (JobStatus == 3 || JobStatus == 
4 || (JobStatus == 5 && Managed =?= TRUE))
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] Fetched 0 job ads from schedd
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] Updating classad values for 137.0:
9/21 09:41:04 [23640]    JobStatus = 5
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640]    EnteredCurrentStatus = 1127310064
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640]    HoldReason = "Globus error: 
java.net.SocketTimeoutException: Read timed out"
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640]    HoldReasonCode = 0
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640]    HoldReasonSubCode = 0
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640]    ReleaseReason = UNDEFINED
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640]    NumSystemHolds = 1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640]    Managed = FALSE
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] condor_read(): nfds=6
9/21 09:41:04 [23640] condor_read(): nfound=1
9/21 09:41:04 [23640] *** deleting delegation 
https://<dest_ip>:8443/wsrf/services/DelegationService?30e7f7d0-2aa5-11da-a5f6-9d28b96a4d7c
9/21 09:41:04 [23640] *** ~GT4Resource
9/21 09:41:04 [23640] GAHP[23643] <- 'UNCACHE_PROXY 1'
9/21 09:41:04 [23640] GAHP[23643] -> 'S'
9/21 09:41:04 [23640] GAHP[23643] <- 'USE_CACHED_PROXY 2'
9/21 09:41:04 [23640] GAHP[23643] -> 'S'
9/21 09:41:04 [23640] No jobs left, shutting down
9/21 09:41:04 [23640] leaving doContactSchedd()
9/21 09:41:04 [23640] Got SIGTERM. Performing graceful shutdown.
9/21 09:41:04 [23640] Started timer to call main_shutdown_fast in 1800 
seconds
9/21 09:41:04 [23640] **** condor_gridmanager (condor_GRIDMANAGER) EXITING 
WITH STATUS 0
=========================================================

_________________________________________________________________
On the road to retirement? Check out MSN Life Events for advice on how to 
get there! http://lifeevents.msn.com/category.aspx?cid=Retirement