Re: [Condor-users] [Globus-discuss] error submitting jobs to condor pool

[feller@xxxxxxxxxxx]

Nano,
see
http://www.globus.org/toolkit/docs/4.0/admin/docbook/quickstart.html#q-gram2
for how to submit a staging job.
>From a first look it seems that delegation didn't work.
Please try the globusrun-ws job with staging and send the
the output of the client and the relevant parts of the
container logfile then.
Martin

> Hi Martin,
>
> While I'm reading globusrun-ws manual, here is the container log:
>
> --------------------------------------
> 2007-06-12 10:15:45,455 INFO  exec.StateMachine
> [RunQueueThread_4,logJobAccepted:3193] Job
> 333f5540-1893-11dc-bb3f-aec5afd22587 accepted for local user 'nano'
> 2007-06-12 10:15:50,878 ERROR exec.StateMachine
> [RunQueueThread_9,fileCleanUp:2730] A secondary fault occured while
> trying to gracefully fail.
> AxisFault
>  faultCode:
> {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
>  faultSubcode:
>  faultString: java.rmi.RemoteException: Unable to create RFT resource;
> nested exception is:
> 	org.globus.transfer.reliable.service.exception.RftException: Error
> processing delegated credentialError getting delegation resource
> [Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error
> getting delegation resource [Caused by:
> org.globus.wsrf.NoSuchResourceException]]
>  faultActor:
>  faultNode:
>  faultDetail:
> 	{http://xml.apache.org/axis/}stackTrace:java.rmi.RemoteException:
> Unable to create RFT resource; nested exception is:
> 	org.globus.transfer.reliable.service.exception.RftException: Error
> processing delegated credentialError getting delegation resource
> [Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error
> getting delegation resource [Caused by:
> org.globus.wsrf.NoSuchResourceException]]
> 	at
> org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:245)
> 	at sun.reflect.GeneratedMethodAccessor287.invoke(Unknown Source)
> 	at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at
> org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:384)
> 	at
> org.globus.axis.providers.RPCProvider.invokeMethodSub(RPCProvider.java:107)
> 	at
> org.globus.axis.providers.PrivilegedInvokeMethodAction.run(PrivilegedInvokeMethodAction.java:42)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.globus.gsi.jaas.GlobusSubject.runAs(GlobusSubject.java:55)
> 	at org.globus.gsi.jaas.JaasSubject.doAs(JaasSubject.java:90)
> 	at
> org.globus.axis.providers.RPCProvider.invokeMethod(RPCProvider.java:97)
> 	at
> org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:281)
> 	at
> org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:319)
> 	at
> org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
> 	at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
> 	at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
> 	at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:450)
> 	at org.apache.axis.server.AxisServer.invoke(AxisServer.java:285)
> 	at org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:664)
> 	at
> org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:382)
> 	at
> org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:147)
> 	at org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:291)
> Caused by: org.globus.transfer.reliable.service.exception.RftException:
> Error processing delegated credentialError getting delegation resource
> [Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error
> getting delegation resource [Caused by:
> org.globus.wsrf.NoSuchResourceException]]
> 	at
> org.globus.transfer.reliable.service.ReliableFileTransferResource.processDelegatedCredential(ReliableFileTransferResource.java:391)
> 	at
> org.globus.transfer.reliable.service.ReliableFileTransferResource.processDelegatedCredential(ReliableFileTransferResource.java:354)
> 	at
> org.globus.transfer.reliable.service.ReliableFileTransferHome.create(ReliableFileTransferHome.java:134)
> 	at
> org.globus.transfer.reliable.service.factory.ReliableFileTransferFactoryService.createReliableFileTransfer(ReliableFileTransferFactoryService.java:235)
> 	... 22 more
>
> 	{http://xml.apache.org/axis/}hostname:hobitton
>
> java.rmi.RemoteException: Unable to create RFT resource; nested exception
> is:
> 	org.globus.transfer.reliable.service.exception.RftException: Error
> processing delegated credentialError getting delegation resource
> [Caused by: org.globus.wsrf.NoSuchResourceException] [Caused by: Error
> getting delegation resource [Caused by:
> org.globus.wsrf.NoSuchResourceException]]
> 	at
> org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:221)
> 	at
> org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:128)
> 	at
> org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
> 	at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
> 	at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown
> Source)
> 	at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown
> Source)
> 	at
> org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown
> Source)
> 	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
> 	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
> 	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
> 	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
> 	at javax.xml.parsers.SAXParser.parse(SAXParser.java:395)
> 	at
> org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
> 	at org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:645)
> 	at org.apache.axis.Message.getSOAPEnvelope(Message.java:424)
> 	at
> org.apache.axis.message.addressing.handler.AddressingHandler.processClientResponse(AddressingHandler.java:305)
> 	at
> org.apache.axis.message.addressing.handler.AddressingHandler.invoke(AddressingHandler.java:110)
> 	at
> org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
> 	at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
> 	at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
> 	at org.apache.axis.client.AxisClient.invoke(AxisClient.java:190)
> 	at org.apache.axis.client.Call.invokeEngine(Call.java:2727)
> 	at org.apache.axis.client.Call.invoke(Call.java:2710)
> 	at org.apache.axis.client.Call.invoke(Call.java:2386)
> 	at org.apache.axis.client.Call.invoke(Call.java:2309)
> 	at org.apache.axis.client.Call.invoke(Call.java:1766)
> 	at
> org.globus.rft.generated.bindings.ReliableFileTransferFactoryPortTypeSOAPBindingStub.createReliableFileTransfer(ReliableFileTransferFactoryPortTypeSOAPBindingStub.java:874)
> 	at
> org.globus.exec.service.exec.utils.StagingHelper.submitStagingRequest(StagingHelper.java:168)
> 	at
> org.globus.exec.service.exec.StateMachine.fileCleanUp(StateMachine.java:2716)
> 	at
> org.globus.exec.service.exec.StateMachine.processFailureFileCleanUpState(StateMachine.java:2091)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at
> org.globus.exec.service.exec.StateMachine.processState(StateMachine.java:302)
> 	at org.globus.exec.service.exec.RunThread.run(RunThread.java:85)
> 2007-06-12 10:15:51,055 INFO  exec.StateMachine
> [RunQueueThread_9,logJobFailed:3212] Job
> 333f5540-1893-11dc-bb3f-aec5afd22587 failed
> --------------------------------------
> This time I only submit one job, to minimize the log/error message.
>
> To make the log complete :) ... here's what Condor log said about the same
> job:
> --------------------------------------
> 017 (096.000.000) 06/12 10:15:50 Job submitted to Globus
>     RM-Contact:
> https://167.205.65.113:8443/wsrf/services/ManagedJobFactoryService
>     JM-Contact:
> https://167.205.65.113:8443/wsrf/services/ManagedExecutableJobService?333f5540-1893-11dc-bb3f-aec5afd22587
>     Can-Restart-JM: 0
> ...
> 027 (096.000.000) 06/12 10:15:50 Job submitted to grid resource
>     GridResource: gt4
> https://167.205.65.113:8443/wsrf/services/ManagedJobFactoryService
> Condor
>     GridJobId: gt4
> https://167.205.65.113:8443/wsrf/services/ManagedExecutableJobService?333f5540-1893-11dc-bb3f-aec5afd22587
> ...
> 012 (096.000.000) 06/12 10:15:51 Job was held.
> 	Globus error: Staging error for RSL element fileStageIn.
> 	Code 0 Subcode 0
> --------------------------------------
>
> Big THANKS !!
>
> --
> Nano Surbakti
>
>
> On 6/12/07, feller@xxxxxxxxxxx <feller@xxxxxxxxxxx> wrote:
>> Ok, what does the server-side GT4 container logfile say?
>> If it's available, please post it to the list.
>> If not: Do you have the Condor's GridmanagerLog?
>> Also: please try to submit a staging job with globusrun-ws
>> (instead of condor-g). What's the output of the client and
>> what does the server-log say (if this fails too)?
>> Martin
>>
>
>