Re: [Condor-users] [Globus-discuss] error submitting jobs to condor pool

["Nano Surbakti" <nano.surbakti@xxxxxxxxx>]

Hi Martin,

While I'm reading globusrun-ws manual, here is the container log:

--------------------------------------
2007-06-12 10:15:45,455 INFO  exec.StateMachine
[RunQueueThread_4,logJobAccepted:3193] Job
333f5540-1893-11dc-bb3f-aec5afd22587 accepted for local user 'nano'
2007-06-12 10:15:50,878 ERROR exec.StateMachine
[RunQueueThread_9,fileCleanUp:2730] A secondary fault occured while
trying to gracefully fail.
AxisFault
faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
faultSubcode:
faultString: java.rmi.RemoteException: Unable to create RFT resource;
nested exception is:
	org.globus.transfer.reliable.service.exception.RftException: Error
processing delegated credentialError getting delegation resource
[Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error
getting delegation resource [Caused by:
org.globus.wsrf.NoSuchResourceException]]
faultActor:
faultNode:
faultDetail:
	{http://xml.apache.org/axis/}stackTrace:java.rmi.RemoteException:
Unable to create RFT resource; nested exception is:
	org.globus.transfer.reliable.service.exception.RftException: Error
processing delegated credentialError getting delegation resource
[Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error
getting delegation resource [Caused by:
org.globus.wsrf.NoSuchResourceException]]
	at org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:245)
	at sun.reflect.GeneratedMethodAccessor287.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:384)
	at org.globus.axis.providers.RPCProvider.invokeMethodSub(RPCProvider.java:107)
	at org.globus.axis.providers.PrivilegedInvokeMethodAction.run(PrivilegedInvokeMethodAction.java:42)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.globus.gsi.jaas.GlobusSubject.runAs(GlobusSubject.java:55)
	at org.globus.gsi.jaas.JaasSubject.doAs(JaasSubject.java:90)
	at org.globus.axis.providers.RPCProvider.invokeMethod(RPCProvider.java:97)
	at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:281)
	at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:319)
	at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
	at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
	at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
	at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:450)
	at org.apache.axis.server.AxisServer.invoke(AxisServer.java:285)
	at org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:664)
	at org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:382)
	at org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:147)
	at org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:291)
Caused by: org.globus.transfer.reliable.service.exception.RftException:
Error processing delegated credentialError getting delegation resource
[Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error
getting delegation resource [Caused by:
org.globus.wsrf.NoSuchResourceException]]
	at org.globus.transfer.reliable.service.ReliableFileTransferResource.processDelegatedCredential(ReliableFileTransferResource.java:391)
	at org.globus.transfer.reliable.service.ReliableFileTransferResource.processDelegatedCredential(ReliableFileTransferResource.java:354)
	at org.globus.transfer.reliable.service.ReliableFileTransferHome.create(ReliableFileTransferHome.java:134)
	at org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:235)
	... 22 more

	{http://xml.apache.org/axis/}hostname:hobitton

java.rmi.RemoteException: Unable to create RFT resource; nested exception is:
	org.globus.transfer.reliable.service.exception.RftException: Error
processing delegated credentialError getting delegation resource
[Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error
getting delegation resource [Caused by:
org.globus.wsrf.NoSuchResourceException]]
	at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:221)
	at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:128)
	at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
	at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
	at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown
Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
	at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
	at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:645)
	at org.apache.axis.Message.getSOAPEnvelope(Message.java:424)
	at org.apache.axis.message.addressing.handler.AddressingHandler.processClientResponse(AddressingHandler.java:305)
	at org.apache.axis.message.addressing.handler.AddressingHandler.invoke(AddressingHandler.java:110)
	at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
	at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
	at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
	at org.apache.axis.client.AxisClient.invoke(AxisClient.java:190)
	at org.apache.axis.client.Call.invokeEngine(Call.java:2727)
	at org.apache.axis.client.Call.invoke(Call.java:2710)
	at org.apache.axis.client.Call.invoke(Call.java:2386)
	at org.apache.axis.client.Call.invoke(Call.java:2309)
	at org.apache.axis.client.Call.invoke(Call.java:1766)
	at org.globus.rft.generated.bindings.ReliableFileTransferFactoryPortTypeSOAPBindingStub.createReliableFileTransfer(ReliableFileTransferFactoryPortTypeSOAPBindingStub.java:874)
	at org.globus.exec.service.exec.utils.StagingHelper.submitStagingRequest(StagingHelper.java:168)
	at org.globus.exec.service.exec.StateMachine.fileCleanUp(StateMachine.java:2716)
	at org.globus.exec.service.exec.StateMachine.processFailureFileCleanUpState(StateMachine.java:2091)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.globus.exec.service.exec.StateMachine.processState(StateMachine.java:302)
	at org.globus.exec.service.exec.RunThread.run(RunThread.java:85)
2007-06-12 10:15:51,055 INFO  exec.StateMachine
[RunQueueThread_9,logJobFailed:3212] Job
333f5540-1893-11dc-bb3f-aec5afd22587 failed
--------------------------------------
This time I only submit one job, to minimize the log/error message.

To make the log complete :) ... here's what Condor log said about the same job:
--------------------------------------
017 (096.000.000) 06/12 10:15:50 Job submitted to Globus
   RM-Contact:
https://167.205.65.113:8443/wsrf/services/ManagedJobFactoryService
   JM-Contact:
https://167.205.65.113:8443/wsrf/services/ManagedExecutableJobService?333f5540-1893-11dc-bb3f-aec5afd22587
   Can-Restart-JM: 0
...
027 (096.000.000) 06/12 10:15:50 Job submitted to grid resource
   GridResource: gt4
https://167.205.65.113:8443/wsrf/services/ManagedJobFactoryService
Condor
   GridJobId: gt4
https://167.205.65.113:8443/wsrf/services/ManagedExecutableJobService?333f5540-1893-11dc-bb3f-aec5afd22587
...
012 (096.000.000) 06/12 10:15:51 Job was held.
	Globus error: Staging error for RSL element fileStageIn.
	Code 0 Subcode 0
--------------------------------------

Big THANKS !!

--
Nano Surbakti


On 6/12/07, feller@xxxxxxxxxxx <feller@xxxxxxxxxxx> wrote:
> Ok, what does the server-side GT4 container logfile say?
> If it's available, please post it to the list.
> If not: Do you have the Condor's GridmanagerLog?
> Also: please try to submit a staging job with globusrun-ws
> (instead of condor-g). What's the output of the client and
> what does the server-log say (if this fails too)?
> Martin