Mailing List Archives
Public Access
|
|
![[Computer Systems Lab]](https://www-auth.cs.wisc.edu/pics/csl_logo.gif)
|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Condor-users] HoldReason = "Globus error: Staging error for RSL element fileStageIn."
- Date: Mon, 7 Jul 2008 14:24:25 -0700 (PDT)
- From: Sean Manning <seangwm@xxxxxxx>
- Subject: [Condor-users] HoldReason = "Globus error: Staging error for RSL element fileStageIn."
Hello all,
I'm still working on a Java Web Services interface to submit jobs to
our condor-metascheduled Globus 4 grid. I have another problem for
you: right now, jobs which we submit through the Web Services code go
into hold mode after being idle for several minutes, with HoldReason =
"Globus error: Staging error for RSL element fileStageIn." We can see
the JDL, executable, and proxy certificate all being staged over
correctly into the spool folder. Why is this happening? Has anyone
here faced a similar problem before? We aren't using a RSL file ... is
Condor-G trying to generate one based on the JDL and failing?
I am attaching part of our code, the JDL, and an excerpt from
GridmanagerLog.<username> for a job 5334.0 which failed in this way.
Back in 2006, Jaime Frey mentioned that this log was useful in
debugging such errors.
Any advice would be appreciated,
Sean Manning
PS. I solved the problem with HoldReason = "Streaming not supported"
which I posted about a week ago. The problem was with the code in the
condor package which parsed the JDL into a Java object.
org.glite.jdl.Ad.fromFile () or org.glite.jdl.Ad.getType (String Attr)
interprets all JDL attributes as STRING-ATTRS, even those which should
be BOOLEAN-ATTRS or INT-ATTRS. Therefore, StreamOut = False was being
ignored because FALSE was interpreted as a string and not a boolean,
and the code was defaulting to StreamOut = True. Since I don't have
the source code or documentation for the Condor package, I can't debug
this further.
/*
* CondorJobSubmitter.java
*
* Created on December 1, 2007, 5:38 PM
*
* To change this template, choose Tools | Template Manager
* and open the template in the editor.
*/
package condorwsgui;
import org.globus.tools.proxy.GridProxyInit;
import org.globus.gsi.GlobusCredential;
import org.globus.gsi.GlobusCredentialException;
import org.globus.tools.proxy.ProxyListener;
import condor.*;
//import condor.CondorCollectorLocator; // SM Not used
//import org.apache.axis.constants.Style; // SM Not used
import birdbath.*;
import java.io.*;
import java.net.URL;
//import java.util.*; // SM Not used
import java.util.regex.*;
import javax.swing.JOptionPane;
//import org.glite.jdl.JobAd;
//import org.glite.jdl.*; // SM Not used
//import condor.classad.*; // SM Not used
//import java.util.Iterator; // SM Not used
import java.util.Vector;
import condor.ClassAdStructAttr;
//import condor.ClassAdAttrType; // SM Not used
import java.nio.charset.Charset;
/**
*
* To implement proxyListener, a public void proxyCreated(GlobusCredential proxy) is required.
*
* @author Core by David Gong; updated and expanded by Sean Manning
*/
public class CondorJobSubmitter implements ProxyListener {
/*
* The following variables are used chiefly for the reader's convenience,
* to explicitly record implicit information such as whether the proxy certificate
* already exists:
* isFromFile, isExistingProxy, jdlFile, jobUniverse, myGlobusCredential
*
* They may be removed without effect, but can help in debugging.
*/
/*
* These variables all represent fasts about the environment and about Condor
*/
private boolean isFromFile; // SM True iff the job will be submitted from a file???
private boolean isSubmissionNeeded; // true if myProxy == null or wantNewProxy (). Also called in proxyCreated ()
private boolean isExistingProxy; // True iff the proxy certificate already exists
private Schedd schedd; // Represents the Condor schedd daemon
private String condorHome; // The location on this machine which will be treated as the root for placing other files
private String jdlFile; // Path to the JDL description of the job to be submitted.
private GlobusCredential myProxy; // A proxy certificate
private String userProxySubject; // The subject DN of the proxy certificate. See the half of each line which is enclosed in quotes in $CONDOR_LOCATION/etc/user_map
/*
* These variables all represent lines in the JDL file
*/
private String owner; // The user id of the job owner eg. "j_smith"
private String user;
private int uid; // User ID, for Unix/Linux systems only.
private String command; // The actual command or shell script which you will execute
private String arguments;
private String requirements;
private String jobUniverse;
private Vector<File> stageInFiles; // SM A vector is an array which re-sizes itself as needed. In Java 5+ it can be parameterized (SM did so).
private Vector<ClassAdStructAttr> jobAdAttrs; // SM Added parameterization
private String proxyFileLocation; // The path to the proxy certificate on the system
private GlobusCredential myGlobusCredential; // SM What is this for?
private OS myOS; // Represents the family of operating system running on the system
private static enum OS {LINUX, WINDOWS, OTHERS};
/** Creates a new instance of CondorJobSubmitter */
public CondorJobSubmitter () {
myProxy = null;
user = System.getProperty("user.name");
owner = user;
command = null;
arguments = null;
stageInFiles = new Vector<File> ();
jobAdAttrs = new Vector<ClassAdStructAttr> ();
proxyFileLocation = null;
myGlobusCredential = null;
condorHome = "/usr/local/condor-6.8.5/local.babargt4"; // Special sub-folder of $CONDOR_LOCATION
determinteOS(); // Prints out the OS name, inter alia
uid = getUID ();
}
/**
* Takes a string which contains a job description and tries to use it
* to submit a file.
*
* Creates a simple GUI to receive the password and other information
* about the certificate from the user
*
* The functions used are proxtInitGUI () and wantNewProxy ()
* @param jdlFile The job description file as a string
*/
public void submitFromFile (String jdlFile) throws Exception {
/*
* Define more of the variables of this object
* since the job to submit is now known
*/
this.jdlFile = jdlFile; // The string is a path to the file which we wish to execute.
isFromFile = true;
isSubmissionNeeded = false;
JobHelper myJobHelper = null; // Part of birdbath but rewritten by DG; converts a JDL file into an object
System.out.println("+++Submit job from JDL file:" + jdlFile); // Line 2 of output
// Convert the JDL file at the given path into an object
try { myJobHelper = new JobHelper(jdlFile);} // SM Uses getJobAttrFromJDL (String file)
catch (Exception err) {
System.out.println("+++Error on parsing jdl file");
System.out.println (err);
throw new Exception("+++Error on parsing jdl file");
}
// The A? B: C; format avoids calling a function on a null object. DG chose it.
// owner defaults to user
owner = (myJobHelper.getOwner() == null)? user:myJobHelper.getOwner().stringValue();
System.out.println ("Owner of myJobHelper: " + myJobHelper.getOwner().toString ());
// jobUniverse defaults to "GLOBUS". Was "VANILLA" in some code which DG had commented out.
jobUniverse = (myJobHelper.getJobUniverse() == null)? "GLOBUS":myJobHelper.getJobUniverse().stringValue();
command = (myJobHelper.getCommand()==null)? null:myJobHelper.getCommand().stringValue();
arguments = (myJobHelper.getArguments()==null)? null:myJobHelper.getArguments().stringValue();
requirements = (myJobHelper.getRequirements()== null)? null: myJobHelper.getRequirements().stringValue();
jobAdAttrs = myJobHelper.getJobAttrVector();
// Organize a list of files to stage in
String[] stageInFilesString = myJobHelper.getStageInFiles();
try {
for (int i=0; i<stageInFilesString.length; i++) {
stageInFiles.add(new File(stageInFilesString[i]));
}
}
catch (Exception err) {
System.out.println("+++Can not open file for stageIn");
throw new Exception("+++Error on opening stageIn files");
}
// SM myProxy is a GlobusCredential from org.globus.gsi
// SM Note that the second part of an AND is not executed
// if the first part is false
// SM What purpose does the outer if/else serve?
// Both branches have essentially the same effect.
if( (myProxy != null) && (!wantNewProxy())){
// Is there a certificate already, and does the user want to use it?
if (getExistingProxy() && (!wantNewProxy())) {
isExistingProxy = true;
System.out.println ("+++Using existing proxy.1");
submit ();
}
else {
proxyInitGUI();
System.out.println("+++Using newly created proxy.1");
isExistingProxy = false;
isSubmissionNeeded = true;
}
}
else { // SM Execution always flows here
// Is there a certificate already, and does the user want to use it?
if (getExistingProxy() && (!wantNewProxy())) {
System.out.println("+++Using existing proxy.2");
isExistingProxy = true;
isSubmissionNeeded = true;
submit ();
}
else {
proxyInitGUI();
System.out.println("+++Using newly created proxy.2");
isExistingProxy = false;
isSubmissionNeeded = true;
}
}
}
/**
* Creates a GUI to ask the user whether he or she wishes to create a new proxy.
*
* @return true if the user selects "Yes"; false if the user selects "No"
*/
private boolean wantNewProxy() {
String message = "The following proxy exists:\n" + myProxy.toString() + "\n\nDo you want to create a new proxy?\n";
int response = JOptionPane.showConfirmDialog(null, message, "Proxy Creation Option", JOptionPane.OK_CANCEL_OPTION);
// System.out.println("response is:" + response);
return (response == 0);
}
/**
* Asks the question "Does a valid proxy certificate for the current user exist?"
*
* @return true iff a valid proxy certificate for the current user exists, false otherwise.
*
* Outputs some text to the console if true.
*/
public boolean getExistingProxy() {
String username = System.getProperty("user.name");
/*
* Get the path to such a hypothetical proxy certificate
*/
// SM Why the final underscore for linux path?
switch(myOS) {
case WINDOWS:
proxyFileLocation = "C:\\Users\\" + username + "\\AppData\\Local\\Temp\\x509up_u_" + owner;
break;
case LINUX:
proxyFileLocation = "/tmp/x509up_u" + uid/*"/tmp/x509up_u_" + owner*/;
break;
default:
// Take linux as default
proxyFileLocation = "/tmp/x509up_u" + uid/*"/tmp/x509up_u_" + owner*/;
}
try {
myProxy = new GlobusCredential (proxyFileLocation);
} catch (GlobusCredentialException err) {
return false;
}
if (this.myProxy != null){
if(this.myProxy.getTimeLeft() > 0){
System.out.println("+++*****Proxy Info*****");
System.out.println("+++Subject\t:" + myProxy.getSubject());
System.out.println("+++Issuer \t:" + myProxy.getIssuer());
System.out.println("+++Strength\t:" + myProxy.getStrength() + "bits");
System.out.println("+++Timeleft\t:" + myProxy.getTimeLeft() + " sec");
System.out.println("+++Proxy type\t:" + myProxy.getProxyType());
System.out.println("+++*****End of Proxy Info*****");
// this.myProxy = myProxy; // SM No effect
this.userProxySubject = myProxy.getSubject();
return true;
}
}
return false;
}
/**
* Creates a GUI to request the user to enter his password.
*/
public void proxyInitGUI() {
GridProxyInit myProxyInit = new GridProxyInit();
myProxyInit.setSize(500, 500);
myProxyInit.setTitle("Creating proxy...");
myProxyInit.setResizable(true);
myProxyInit.setRunAsApplication(true);
myProxyInit.setCloseOnSuccess(true);
myProxyInit.addProxyListener(this);
myProxyInit.setVisible(true);
}
/**
* This function is called when a new proxy certificate has been created.
* It assigns that certificate to this object, updates userProxySubject,
* and (if necessary) tries to submit the job to the grid.
*
* Right now, it is only called by other classes (probably one of the ones
* that handles the "Create New Proxy" window).
* @param proxy The new proxy certificate
*/
public void proxyCreated(GlobusCredential proxy)
{
System.out.println("+++ In proxyCreated ()");
myProxy = proxy;
userProxySubject = proxy.getSubject();
if (isSubmissionNeeded) {
try {submit ();}
catch(Exception err){
System.out.println ("Error caught in proxyCreated: " + err); // SM Error caught here
err.printStackTrace();
}
}
}
/**
* Detetermines which OS the system uses, assigning the correct value to myOS.
*/
private void determinteOS(){
String myOSName = System.getProperty("os.name");
Pattern p1 = Pattern.compile("WIN"); // Compiles regular expression into Pattern.
Pattern p2 = Pattern.compile("Lin");
Matcher m1 = p1.matcher(myOSName);
Matcher m2 = p2.matcher(myOSName);
if (m1.find()){
myOS = OS.WINDOWS;
System.out.println("CondorJobSubmitter.determinteos (): Windows");
}
else if (m2.find()){
myOS = OS.LINUX;
System.out.println("CondorJobSubmitter.determinteos (): Linux");
}
else
myOS = OS.OTHERS;
}
/**
* Adds the proxy certificate to the jobs to be staged over from this machine
* to the Condor server.
* @param cluster The cluster which the proxy will be associated with
* @param job The job ID which the proxy will be associated with
* @throws Exception
*/
private void addProxy(int cluster, int job) throws Exception{
try {
stageInFiles.add(new File("/tmp/x509up_u" + uid));
ClassAdStructAttr subjectAttr = JobHelper.createStringAttr("X509UserProxySubject", userProxySubject);
createProxyFileLocation(cluster, job);
ClassAdStructAttr locationAttr = JobHelper.createStringAttr("X509UserProxy", proxyFileLocation);
jobAdAttrs.add(subjectAttr);
jobAdAttrs.add(locationAttr);
}
catch (Exception err) {
throw new Exception("Error on creating proxy related attributes");
}
}
/**
* Sets proxyFileLocation to the job's folder in the spool folder.
* According to DG's workterm report p. 28, this is necessary because the SOAP
* interface can only access this folder and not /tmp where the proxy
* certificates are usually stored.
*
* @param cluster The batch of jobs which this job is part of
* @param job The number of the job within that cluster. This starts at 0 and increments by one.
*/
private void createProxyFileLocation(int cluster, int job){
// SM Was condorHome + "/spool/cluster" + cluster +".proc" + job + ".subproc0/x509up_u_" + owner.substring(0, owner.length())
proxyFileLocation = (condorHome + "/spool/cluster" + cluster +".proc" + job + ".subproc0/x509up_u" + uid);
return;
}
/**
* The proxy file in the spool folder on the host should be owned by the user who
* is running this script, with permissions -rw-------.
*
* As of 3 July 2008, it fails with no output. It definitely references the correct file.
*
* The problem is that chmod tries to change the file locally (on the client) not on the server.
*/
private void changeProxyFileOwner (int cluster, int job) {
// TODO DO I need SUDO?
String[] cmd = {"/bin/bash", "-c",
"echo $USER"/*"chown -v " + owner + ":hep " + condorHome + "/spool/cluster" + job + ".proc" + cluster + ".subproc0/x509up_u" + uid*/};
System.out.println ("+++Trying command: " + cmd[2]);
try {
Process p = Runtime.getRuntime().exec(cmd);
InputStream processIn = p.getInputStream ();
InputStreamReader reader = new InputStreamReader (processIn, "US-ASCII");
String output = new String ();
char c; // Current character
int tmp = 0;
do {
tmp = reader.read ();
if (tmp != -1) {
c = (char) tmp;
output += Character.toString (c);
}
} while (tmp != -1);
System.out.println ("Output of chown: " + output);
}
catch (IOException err) {
System.out.println ("+++Failed to change owner of proxy file on host: "+ err);
}
}
/**
* Submits a job to the grid once the schedd and this instance have been initialized.
* Specifically, it submits a job ClassAd to the application's condor_schedd.
*
* See "Developer APIs to Condor + Condor Web Services" powerpoint slide 27, 28
*/
public void submit () throws Exception {
/*
* A transaction is a discrete unit of work. It has a limited lifespan.
*/
birdbath.Transaction xact = schedd.createTransaction();
try {xact.begin(3000);} // Argument is max duration in seconds
catch (java.rmi.RemoteException err)
{
System.out.println("+++Failed to start transaction");
err.printStackTrace ();
throw (new Exception ("Failed to start transaction"));
}
// Define some final parameters of the job
int cluster = xact.createCluster();
int job = xact.createJob(cluster);
addProxy(cluster, job);
ClassAdStructAttr[] jobExtraAttrArray = (ClassAdStructAttr[])jobAdAttrs.toArray(new ClassAdStructAttr[0]);
File[] files = (File[])stageInFiles.toArray(new File[0]);
// Submits the job ClassAd (as a collection of parameters) to the Schedd
/*Was UniverseType.VANILLA*/
xact.submit(cluster, job, owner, UniverseType.GLOBUS, command, arguments, requirements, jobExtraAttrArray, files );
xact.commit(); // Commits the transaction to make it take effect
System.out.println("+++Job submitted successfully. ClusterID=" + cluster + " jobId=" + job);
changeProxyFileOwner (job, cluster);
}
/**
* Assigns a new Schedd to this object
* @param mySchedd The Schedd to be assigned to this object
*/
public void setSchedd(Schedd mySchedd){
schedd = mySchedd;
}
/**
* @return The Schedd associated with this object
*/
public Schedd getSchedd(){
return schedd;
}
/**
* This requests a condor_reschedule from the condor_schedd daemon.
*
* condor_reschedule updates the information about a given machine's resources
* and jobs to the central manager. This can be used if one wants to see the
* current status of a machine. In order to do this, one would first run
* condor_reschedule, and then use the condor_status command to get specific
* information about that machine. condor_reschedule also starts a new
* negotiation cycle between resource owners and resource providers on the
* central managers, so that jobs can be matched with machines right away.
* This can be useful in situations where the time between negotiation cycles
* is somewhat long, and an administrator wants to see if a job they have
* in the queue will get matched without waiting for the next negotiation cycle.
*
* (The above was borrowed from Sections 5 and 4 of the online manual
* version 6.0.3 at http://www.cs.wisc.edu/condor/manual/v6.0/ref.html
*/
public void requestReschedule() throws Exception {
schedd.requestReschedule();
}
/**
* TODO Implement in a generic way without mangling the original file.
*
*/
public void parseJdlFile (String jdlFile) {
// Turn A = B into A = "B";
// Rename variable Error err, and variable Output out
//Turn queue A into queue = "A";
// Add an InputSandbox
}
/**
* Gets the Unix user ID of the current user. This determines where a
* pre-existing proxy certificate is stored on a Unix system. It will fail
* if myOS has not yet been set, or if the operating system is not Linux.
*
* @return The user ID (on a Unix system) or -1 (on any other OS,
* or in case of an error on a Unix system)
*/
private int getUID () {
if (myOS == OS.LINUX) {
/*
* We break the command up into several short strings to ensure
* that is is parsed correctly. It is equivalent to typing
* "/bin/bash -c 'id'" on the command line and means
* "submit the string 'id' to be processed as a bash shell command".
*
* See http://java.sun.com/developer/JDCTechTips/2003/tt0304.html
*/
String[] cmd = { "/bin/bash", "-c", "id" };
try {
Process p = Runtime.getRuntime().exec(cmd);
InputStream processIn = p.getInputStream ();
InputStreamReader reader = new InputStreamReader (processIn, "US-ASCII");
int uid = 0;
char c;
boolean equalsFound = false;
// Parse the input looking for a uid. Expected format is:
// uid=58858(seangwm) gid=34244(hep) groups=34244(hep)
do {
c = (char) reader.read ();
if (c == '=') {
equalsFound = true;
}
else if (equalsFound) {
// If the current character is a digit, multiply the old uid by 10
// and add the current digit eg. 5 -> 50 + 8 = 58 -> 580 + 8 = 588
if (Character.isDigit(c)) {
uid *= 10;
uid += Character.getNumericValue (c);
}
}
} while (c != '(');
System.out.println ("uid = " + uid);
return uid;
}
catch (IOException err) {
System.out.println ("Failed to get uid");
return -1;
}
}
else {
return -1;
}
}
/**
*
* Performs several simple tests of this class's functionality.
*
* @param args A list of command-line arguments. This method expects
* to take a single one, which defines which sample job to submit.
*
*/
public static void main(String[] args){
/*
* Set system properties (cp. slide 38 of "Developer APIs for Condor" PPT)
*/
// For Windows
// java.lang.System.setProperty("javax.net.ssl.trustStore", "c:\\Program Files\\Java\\jre1.6.0_03\\bin\\truststore");
// For Unix
java.lang.System.setProperty("javax.net.ssl.trustStore", "/hepuser/seangwm/ashokProjects/CondorWebService/CondorWSProjectRon/src/supportfiles/truststore");
// For Windows
// java.lang.System.setProperty("javax.net.ssl.keyStore", "c:\\Users\\Daobgong\\JavaProject\\CondorWSGUI\\keystore");
// java.lang.System.setProperty("javax.net.ssl.keyStore", "c:\\Users\\Daobgong\\JavaProject\\CondorWSGUI\\DavidGridKeyStore");
// For Unix
java.lang.System.setProperty("javax.net.ssl.keyStore", "/hepuser/seangwm/ashokProjects/CondorWebService/CondorWSProjectRon/src/supportfiles/keystore");
java.lang.System.setProperty("javax.net.ssl.keyStoreType", "PKCS12");
java.lang.System.setProperty("javax.net.ssl.keyStorePassword", "An5sh6An3-");
/*
* Create a CondorJobSubmitter to submit the job
*/
//System.out.println ("Creating CondorJobSubmitter");
CondorJobSubmitter mySubmitter = new CondorJobSubmitter();
/*
* Create an instance of birdbath.Schedd located at a particular port,
* and make it the schedd of this CondorJobSubmitter.
*/
Schedd mySchedd = null;
try { // Connect to a service listening on a particular port
// String tmpStr="https://ugdev01.phys.uvic.ca:1980";;
String tmpStr="https://babargt4.phys.uvic.ca:1980";;
URL scheddLocation = new URL(tmpStr);
mySchedd = new Schedd(scheddLocation);
}
catch (Exception err) {
System.out.println("+++Failed to create scheduler, System is exiting.");
System.exit(-1);
}
mySubmitter.setSchedd(mySchedd);
/*
* Submit a test job to the grid. Creates a simple graphical user interface to do so.
*/
/* The following lines assume that this is a Linux machine. */
//Ron changed this
try {
System.out.println();
String fileToSubmit = null;
if (args[0].contains ("4")) { // testjdl-gt4
fileToSubmit = "/hepuser/seangwm/ashokProjects/CondorWebService/CondorWSProjectRon/src/supportfiles/testjdl-gt4";
}
else if (args[0].contains ("2")) { // testjdl-gt2
fileToSubmit = "/hepuser/seangwm/ashokProjects/CondorWebService/CondorWSProjectRon/src/supportfiles/testjdl-gt2";
}
else if (args[0].contains ("DG")) { // David Gong's test.jdl
fileToSubmit = "/hepuser/seangwm/ashokProjects/CondorWebService/CondorWSProjectRon/src/supportfiles/test.jdl";
}
else { // Default
fileToSubmit = "/hepuser/seangwm/ashokProjects/CondorWebService/CondorWSProjectRon/src/supportfiles/test.jdl";
}
mySubmitter.submitFromFile(fileToSubmit);
}
catch (Exception err) {
System.out.println("+++Error on submit");
err.printStackTrace();
System.exit(0);
}
try {
mySubmitter.requestReschedule();
System.out.println("+++RequestReschedule succeeded");
} // Just does Schedd schedd.requestReschedule ()
catch (Exception err) {
// err.printStackTrace();
System.out.println("+++RequestReschedule failed");
}
}
}
Attachment:
testjdl-gt4
Description: Binary data
7/7 12:14:14 [32361] Using job type GT4 for job 5334.0
7/7 12:14:14 [32361] (5334.0) SetJobLeaseTimers()
7/7 12:14:14 [32361] Found job 5334.0 --- inserting
7/7 12:14:14 [32361] Fetched 1 new job ads from schedd
7/7 12:14:14 [32361] querying for removed/held jobs
7/7 12:14:14 [32361] Using constraint ((Owner=?="seangwm"&&JobUniverse==9)) && ((Managed =!= "ScheddDone")) && (JobStatus == 3 || JobStatus == 4 || (JobStatus == 5 && Managed =?= "External"))
7/7 12:14:14 [32361] Fetched 0 job ads from schedd
7/7 12:14:14 [32361] leaving doContactSchedd()
7/7 12:14:14 [32361] gahp server not up yet, delaying ping
7/7 12:14:14 [32361] *** UpdateLeases called
7/7 12:14:14 [32361] Leases not supported, cancelling timer
7/7 12:14:14 [32361] *** checkDelegation()
7/7 12:14:14 [32361] gahp server not up yet, delaying checkDelegation
7/7 12:14:14 [32361] GridftpServer: Scanning schedd for previously submitted gridftp server jobs
7/7 12:14:14 [32361] Error loading private key from file7/7 12:14:14 [32361] Error initializing client security context
7/7 12:14:14 [32361] Error creating SSL context
7/7 12:14:14 [32361] SSL Authentication fails, terminating
7/7 12:14:14 [32361] This process has a valid certificate & key
7/7 12:14:14 [32361] GridftpServer: Submitting job for proxy '/C=CA/O=Grid/OU=phys.uvic.ca/CN=Sean Manning'
7/7 12:14:14 [32361] Error loading private key from file7/7 12:14:14 [32361] Error initializing client security context
7/7 12:14:14 [32361] Error creating SSL context
7/7 12:14:14 [32361] SSL Authentication fails, terminating
7/7 12:14:14 [32361] This process has a valid certificate & key
7/7 12:14:14 [32361] Error loading private key from file7/7 12:14:14 [32361] Error initializing client security context
7/7 12:14:14 [32361] Error creating SSL context
7/7 12:14:14 [32361] SSL Authentication fails, terminating
7/7 12:14:14 [32361] This process has a valid certificate & key
7/7 12:14:14 [32361] entering FileTransfer::SimpleInit
7/7 12:14:14 [32361] entering FileTransfer::UploadFiles (final_transfer=0)
7/7 12:14:14 [32361] entering FileTransfer::Upload
7/7 12:14:14 [32361] entering FileTransfer::DoUpload
7/7 12:14:14 [32361] DoUpload: send file /tmp/condor_g_scratch.0x9070b00.32052/grid-mapfile
7/7 12:14:14 [32361] ReliSock::put_file_with_permissions(): going to send permissions 100644
7/7 12:14:14 [32361] put_file: going to send from filename /tmp/condor_g_scratch.0x9070b00.32052/grid-mapfile
7/7 12:14:14 [32361] put_file: Found file size 55
7/7 12:14:14 [32361] put_file: sending 55 bytes
7/7 12:14:14 [32361] ReliSock: put_file: sent 55 bytes
7/7 12:14:14 [32361] DoUpload: send file /tmp/condor_g_scratch.0x9070b00.32052/master_proxy.2
7/7 12:14:14 [32361] DoUpload: send file /usr/local/condor-6.8.5/libexec/gridftp_wrapper.sh
7/7 12:14:14 [32361] ReliSock::put_file_with_permissions(): going to send permissions 100755
7/7 12:14:14 [32361] put_file: going to send from filename /usr/local/condor-6.8.5/libexec/gridftp_wrapper.sh
7/7 12:14:14 [32361] put_file: Found file size 111
7/7 12:14:14 [32361] put_file: sending 111 bytes
7/7 12:14:14 [32361] ReliSock: put_file: sent 111 bytes
7/7 12:14:14 [32361] DoUpload: exiting at 2210
7/7 12:14:17 [32361] (5334.0) doEvaluateState called: gmState GM_INIT, globusState 32
7/7 12:14:17 [32361] GAHP server pid = 32379
7/7 12:14:20 [32361] GAHP server version: $GahpVersion: 1.6.0 Dec 08 2006 GT4 GAHP (GT-4.0.3) $
7/7 12:14:20 [32361] GAHP[32379] <- 'COMMANDS'
7/7 12:14:20 [32361] GAHP[32379] -> 'S' 'ASYNC_MODE_OFF' 'ASYNC_MODE_ON' 'CACHE_PROXY_FROM_FILE' 'COMMANDS' 'GASS_SERVER_INIT' 'GT4_DELEGATE_CREDENTIAL' 'GT4_DELEGATE_CREDENTIAL_2' 'GT4_GENERATE_SUBMIT_ID' 'GT4_GRAM_CALLBACK_ALLOW' 'GT4_GRAM_JOB_CALLBACK_REGISTER' 'GT4_GRAM_JOB_DESTROY' 'GT4_GRAM_JOB_START' 'GT4_GRAM_JOB_STATUS' 'GT4_GRAM_JOB_SUBMIT' 'GT4_GRAM_PING' 'GT4_REFRESH_CREDENTIAL' 'GT4_REFRESH_CREDENTIAL_2' 'GT4_SET_TERMINATION_TIME' 'INITIALIZE_FROM_FILE' 'QUIT' 'REFRESH_PROXY_FROM_FILE' 'RESPONSE_PREFIX' 'RESULTS' 'UNCACHE_PROXY' 'USE_CACHED_PROXY' 'VERSION'
7/7 12:14:20 [32361] GAHP[32379] <- 'RESPONSE_PREFIX GAHP:'
7/7 12:14:20 [32361] GAHP[32379] -> 'S'
7/7 12:14:20 [32361] GAHP[32379] <- 'ASYNC_MODE_ON'
7/7 12:14:20 [32361] GAHP[32379] -> 'S'
7/7 12:14:20 [32361] GAHP[32379] <- 'INITIALIZE_FROM_FILE /tmp/condor_g_scratch.0x9070b00.32052/master_proxy.2'
7/7 12:14:22 [32361] GAHP[32379] -> 'S'
7/7 12:14:22 [32361] GAHP[32379] <- 'CACHE_PROXY_FROM_FILE 2 /tmp/condor_g_scratch.0x9070b00.32052/master_proxy.2'
7/7 12:14:22 [32361] GAHP[32379] -> 'S'
7/7 12:14:22 [32361] GAHP[32379] <- 'CACHE_PROXY_FROM_FILE 1 /usr/local/condor-6.8.5/local.babargt4/spool/cluster5334.proc0.subproc0/x509up_u58858'
7/7 12:14:22 [32361] GAHP[32379] -> 'S'
7/7 12:14:22 [32361] GAHP[32379] <- 'GT4_GRAM_CALLBACK_ALLOW 2'
7/7 12:14:24 [32361] GAHP[32379] -> 'S' '1'
7/7 12:14:24 [32361] (5334.0) gm state change: GM_INIT -> GM_START
7/7 12:14:24 [32361] (5334.0) gm state change: GM_START -> GM_CLEAR_REQUEST
7/7 12:14:24 [32361] (5334.0) UpdateJobLeaseSent(-1)
7/7 12:14:24 [32361] (5334.0) gm state change: GM_CLEAR_REQUEST -> GM_UNSUBMITTED
7/7 12:14:24 [32361] GridftpServer: Updating job leases for gridftp server jobs
7/7 12:14:24 [32361] Error loading private key from file7/7 12:14:24 [32361] Error initializing client security context
7/7 12:14:24 [32361] Error creating SSL context
7/7 12:14:24 [32361] SSL Authentication fails, terminating
7/7 12:14:24 [32361] This process has a valid certificate & key
7/7 12:14:24 [32361] GAHP[32379] <- 'GT4_GRAM_PING 3 https://ugdev07.phys.uvic.ca:8443'
7/7 12:14:24 [32361] GAHP[32379] -> 'S'
7/7 12:14:24 [32361] *** checkDelegation()
7/7 12:14:24 [32361] in doContactSchedd()
7/7 12:14:24 [32361] Error loading private key from file7/7 12:14:24 [32361] Error initializing client security context
7/7 12:14:24 [32361] Error creating SSL context
7/7 12:14:24 [32361] SSL Authentication fails, terminating
7/7 12:14:24 [32361] This process has a valid certificate & key
7/7 12:14:24 [32361] querying for removed/held jobs
7/7 12:14:24 [32361] Using constraint ((Owner=?="seangwm"&&JobUniverse==9)) && ((Managed =!= "ScheddDone")) && (JobStatus == 3 || JobStatus == 4 || (JobStatus == 5 && Managed =?= "External"))
7/7 12:14:24 [32361] Fetched 0 job ads from schedd
7/7 12:14:24 [32361] 5335.0 job status: 2
7/7 12:14:24 [32361] leaving doContactSchedd()
7/7 12:14:24 [32361] GAHP[32379] <- 'RESULTS'
7/7 12:14:24 [32361] GAHP[32379] -> 'R'
7/7 12:14:24 [32361] GAHP[32379] -> 'S' '1'
7/7 12:14:24 [32361] GAHP[32379] -> '3' '0' 'NULL'
7/7 12:14:24 [32361] resource https://ugdev07.phys.uvic.ca:8443 is now up
7/7 12:14:24 [32361] (5334.0) doEvaluateState called: gmState GM_UNSUBMITTED, globusState 32
7/7 12:14:24 [32361] (5334.0) gm state change: GM_UNSUBMITTED -> GM_DELEGATE_PROXY
7/7 12:14:24 [32361] getDelegationError(): failed to find ProxyDelegation for proxy /usr/local/condor-6.8.5/local.babargt4/spool/cluster5334.proc0.subproc0/x509up_u58858
7/7 12:14:24 [32361] *** getDelegationURI(/usr/local/condor-6.8.5/local.babargt4/spool/cluster5334.proc0.subproc0/x509up_u58858)
7/7 12:14:24 [32361] creating new ProxyDelegation
7/7 12:14:24 [32361] *** checkDelegation()
7/7 12:14:24 [32361] new delegation
7/7 12:14:24 [32361] GAHP[32379] <- 'USE_CACHED_PROXY 1'
7/7 12:14:24 [32361] GAHP[32379] -> 'S'
7/7 12:14:24 [32361] GAHP[32379] <- 'GT4_DELEGATE_CREDENTIAL 4 https://ugdev07.phys.uvic.ca:8443/wsrf/services/DelegationFactoryService'
7/7 12:14:24 [32361] GAHP[32379] -> 'S'
7/7 12:14:25 [32361] DaemonCore::IsPidAlive(): kill returned EPERM, assuming pid 32052 is alive.
7/7 12:14:26 [32361] GAHP[32379] <- 'RESULTS'
7/7 12:14:26 [32361] GAHP[32379] -> 'R'
7/7 12:14:26 [32361] GAHP[32379] -> 'S' '1'
7/7 12:14:26 [32361] GAHP[32379] -> '4' '0' 'https://142.104.60.57:8443/wsrf/services/DelegationService?ea2e0360-4c58-11dd-95e1-8c59e81f0204' 'NULL'
7/7 12:14:26 [32361] *** checkDelegation()
7/7 12:14:26 [32361] new delegation
7/7 12:14:26 [32361] https://142.104.60.57:8443/wsrf/services/DelegationService?ea2e0360-4c58-11dd-95e1-8c59e81f0204
7/7 12:14:26 [32361] signalling jobs for https://142.104.60.57:8443/wsrf/services/DelegationService?ea2e0360-4c58-11dd-95e1-8c59e81f0204
7/7 12:14:26 [32361] (5334.0) doEvaluateState called: gmState GM_DELEGATE_PROXY, globusState 32
7/7 12:14:26 [32361] *** getDelegationURI(/usr/local/condor-6.8.5/local.babargt4/spool/cluster5334.proc0.subproc0/x509up_u58858)
7/7 12:14:26 [32361] found ProxyDelegation
7/7 12:14:26 [32361] (5334.0) gm state change: GM_DELEGATE_PROXY -> GM_GENERATE_ID
7/7 12:14:26 [32361] GAHP[32379] <- 'GT4_GENERATE_SUBMIT_ID 5 '
7/7 12:14:26 [32361] GAHP[32379] -> 'S'
7/7 12:14:26 [32361] GAHP[32379] <- 'RESULTS'
7/7 12:14:26 [32361] GAHP[32379] -> 'R'
7/7 12:14:26 [32361] GAHP[32379] -> 'S' '1'
7/7 12:14:26 [32361] GAHP[32379] -> '5' 'uuid:ea39ea40-4c58-11dd-898e-fbb5210b5f9c'
7/7 12:14:26 [32361] (5334.0) doEvaluateState called: gmState GM_GENERATE_ID, globusState 32
7/7 12:14:26 [32361] (5334.0) gm state change: GM_GENERATE_ID -> GM_SUBMIT_ID_SAVE
7/7 12:14:29 [32361] in doContactSchedd()
7/7 12:14:29 [32361] Error loading private key from file7/7 12:14:29 [32361] Error initializing client security context
7/7 12:14:29 [32361] Error creating SSL context
7/7 12:14:29 [32361] SSL Authentication fails, terminating
7/7 12:14:29 [32361] This process has a valid certificate & key
7/7 12:14:29 [32361] querying for removed/held jobs
7/7 12:14:39 [32361] querying for removed/held jobs
7/7 12:14:39 [32361] Using constraint ((Owner=?="seangwm"&&JobUniverse==9)) && ((Managed =!= "ScheddDone")) && (JobStatus == 3 || JobStatus == 4 || (JobStatus == 5 && Managed =?= "External"))
7/7 12:14:39 [32361] Fetched 0 job ads from schedd
7/7 12:14:39 [32361] Updating classad values for 5334.0:
7/7 12:14:39 [32361] NumGlobusSubmits = 1
7/7 12:14:39 [32361] GlobusStatus = 64
7/7 12:14:39 [32361] leaving doContactSchedd()
7/7 12:14:39 [32361] (5334.0) doEvaluateState called: gmState GM_SUBMITTED, globusState 64
7/7 12:15:10 [32361] Received CHECK_LEASES signal
7/7 12:15:10 [32361] Evaluating periodic job policy expressions.
7/7 12:15:10 [32361] in doContactSchedd()
7/7 12:15:10 [32361] Error loading private key from file7/7 12:15:10 [32361] Error initializing client security context
7/7 12:15:10 [32361] Error creating SSL context
7/7 12:15:10 [32361] SSL Authentication fails, terminating
7/7 12:15:10 [32361] This process has a valid certificate & key
7/7 12:15:10 [32361] querying for renewed leases
7/7 12:15:10 [32361] querying for removed/held jobs
7/7 12:15:10 [32361] Using constraint ((Owner=?="seangwm"&&JobUniverse==9)) && ((Managed =!= "ScheddDone")) && (JobStatus == 3 || JobStatus == 4 || (JobStatus == 5 && Managed =?= "External"))
7/7 12:15:10 [32361] Fetched 0 job ads from schedd
7/7 12:15:10 [32361] leaving doContactSchedd()
7/7 12:15:20 [32361] GAHP[32379] <- 'RESULTS'
7/7 12:15:20 [32361] GAHP[32379] -> 'S' '0'
7/7 12:15:24 [32361] in doContactSchedd()
7/7 12:15:24 [32361] Error loading private key from file7/7 12:15:24 [32361] Error initializing client security context
7/7 12:15:24 [32361] Error creating SSL context
7/7 12:15:24 [32361] SSL Authentication fails, terminating
7/7 12:15:24 [32361] This process has a valid certificate & key
7/7 12:15:24 [32361] querying for removed/held jobs
7/7 12:15:24 [32361] Using constraint ((Owner=?="seangwm"&&JobUniverse==9)) && ((Managed =!= "ScheddDone")) && (JobStatus == 3 || JobStatus == 4 || (JobStatus == 5 && Managed =?= "External"))
7/7 12:15:24 [32361] Fetched 0 job ads from schedd
7/7 12:15:24 [32361] 5335.0 job status: 2
7/7 12:15:24 [32361] leaving doContactSchedd()
7/7 12:16:10 [32361] Received CHECK_LEASES signal
7/7 12:16:10 [32361] Evaluating periodic job policy expressions.
7/7 12:16:10 [32361] in doContactSchedd()
7/7 12:16:10 [32361] Error loading private key from file7/7 12:16:10 [32361] Error initializing client security context
7/7 12:16:10 [32361] Error creating SSL context
7/7 12:16:10 [32361] SSL Authentication fails, terminating
7/7 12:16:10 [32361] This process has a valid certificate & key
7/7 12:16:10 [32361] querying for renewed leases
7/7 12:16:10 [32361] querying for removed/held jobs
7/7 12:16:10 [32361] Using constraint ((Owner=?="seangwm"&&JobUniverse==9)) && ((Managed =!= "ScheddDone")) && (JobStatus == 3 || JobStatus == 4 || (JobStatus == 5 && Managed =?= "External"))
7/7 12:16:10 [32361] Fetched 0 job ads from schedd
7/7 12:16:10 [32361] leaving doContactSchedd()
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> Full fault for job https://142.104.60.57:8443/wsrf/services/ManagedExecutableJobService?ea39ea40-4c58-11dd-898e-fbb5210b5f9c:
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> fault type: org.globus.exec.generated.StagingFaultType:
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> attribute: fileStageIn
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> description:
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> Staging error for RSL element fileStageIn.
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> faultReason:
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> faultString:
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> gt2ErrorCode: 0
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> originator: Address: https://142.104.60.57:8443/wsrf/services/ManagedJobFactoryService
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> Reference property[0]:
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> <ns1:ResourceID xmlns:ns1="http://www.globus.org/namespaces/2004/10/gram/job";>ea39ea40-4c58-11dd-898e-fbb5210b5f9c</ns1:ResourceID>
7/7 12:16:10 [32361] GAHP[32379] (stderr) ->
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> stackTrace:
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> org.globus.exec.generated.StagingFaultType: Staging error for RSL element fileStageIn.
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> Timestamp: Mon Jul 07 12:16:05 PDT 2008
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> Originator: Address: https://142.104.60.57:8443/wsrf/services/ManagedJobFactoryService
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> Reference property[0]:
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> <ns1:ResourceID xmlns:ns1="http://www.globus.org/namespaces/2004/10/gram/job";>ea39ea40-4c58-11dd-898e-fbb5210b5f9c</ns1:ResourceID>
7/7 12:16:10 [32361] GAHP[32379] (stderr) ->
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at java.lang.Class.newInstance0(Class.java:350)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at java.lang.Class.newInstance(Class.java:303)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.exec.utils.FaultUtils.makeFault(FaultUtils.java:485)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.exec.utils.FaultUtils.createStagingFault(FaultUtils.java:363)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.exec.service.exec.StateMachine.processStageInResponseState(StateMachine.java:995)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at java.lang.reflect.Method.invoke(Method.java:585)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.exec.service.exec.StateMachine.processState(StateMachine.java:367)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.exec.service.exec.RunThread.run(RunThread.java:93)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> Transient transfer error
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> Server refused performing the request. Custom message: (error code 1) [Nested exception message: Custom message: Unexpected reply: 500-Command failed. : globus_l_gfs_file_open failed.
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500-globus_xio: Unable to open file /usr/local/condor-6.8.5/local.babargt4/spool/cluster5334.proc0.subproc0/testjdl-gt4
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500-globus_xio: System error in open: Permission denied
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500-globus_xio: A system call failed: Permission denied
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500-globus_xio: A system call failed: Permission denied
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500 End.]. Caused by org.globus.ftp.exception.ServerException: Server refused performing the request. Custom message: (error code 1) [Nested exception message: Custom message: Unexpected reply: 500-Command failed. : globus_l_gfs_file_open failed.
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500-globus_xio: Unable to open file /usr/local/condor-6.8.5/local.babargt4/spool/cluster5334.proc0.subproc0/testjdl-gt4
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500-globus_xio: System error in open: Permission denied
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500-globus_xio: A system call failed: Permission denied
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500 End.]. Nested exception is org.globus.ftp.exception.UnexpectedReplyCodeException: Custom message: Unexpected reply: 500-Command failed. : globus_l_gfs_file_open failed.
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500-globus_xio: Unable to open file /usr/local/condor-6.8.5/local.babargt4/spool/cluster5334.proc0.subproc0/testjdl-gt4
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500-globus_xio: System error in open: Permission denied
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500-globus_xio: A system call failed: Permission denied
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> 500 End.
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:191)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.ftp.vanilla.TransferMonitor.start(TransferMonitor.java:105)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.ftp.FTPClient.transferRunSingleThread(FTPClient.java:1451)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.ftp.FTPClient.transfer(FTPClient.java:1350)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.transfer.reliable.service.TransferClient.normalNonExtendedTransfer(TransferClient.java:866)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.transfer.reliable.service.TransferClient.transfer(TransferClient.java:679)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.transfer.reliable.service.TransferWork.run(TransferWork.java:714)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.wsrf.impl.work.WorkManagerImpl$WorkWrapper.run(WorkManagerImpl.java:345)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at java.lang.Thread.run(Thread.java:595)
7/7 12:16:10 [32361] GAHP[32379] (stderr) ->
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at java.lang.Class.newInstance0(Class.java:350)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at java.lang.Class.newInstance(Class.java:303)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.encoding.ser.BeanDeserializer.<init>(BeanDeserializer.java:90)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.encoding.ser.BeanDeserializer.<init>(BeanDeserializer.java:76)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.exec.generated.StagingFaultType.getDeserializer(StagingFaultType.java:152)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at java.lang.reflect.Method.invoke(Method.java:585)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.encoding.DeserializationContext.getDeserializerForClass(DeserializationContext.java:510)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.encoding.ser.BeanDeserializer.onStartChild(BeanDeserializer.java:250)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.encoding.DeserializationContext.startElement(DeserializationContext.java:1035)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at javax.xml.parsers.SAXParser.parse(SAXParser.java:375)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.wsrf.encoding.ObjectDeserializer.toObject(ObjectDeserializer.java:59)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at condor.gahp.gt4.JobListener.deliver(JobListener.java:165)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.wsrf.impl.notification.NotificationConsumerProvider.notify(NotificationConsumerProvider.java:109)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at java.lang.reflect.Method.invoke(Method.java:585)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:384)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:281)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:319)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:450)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.apache.axis.server.AxisServer.invoke(AxisServer.java:285)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.wsrf.container.ServiceThread.doPost(ServiceThread.java:676)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.wsrf.container.ServiceThread.process(ServiceThread.java:397)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.wsrf.container.GSIServiceThread.process(GSIServiceThread.java:151)
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> at org.globus.wsrf.container.ServiceThread.run(ServiceThread.java:302)
7/7 12:16:10 [32361] GAHP[32379] (stderr) ->
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> stateWhenFailureOccurred: StageIn
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> timestamp: java.util.GregorianCalendar[time=1215458165427,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2008,MONTH=6,WEEK_OF_YEAR=28,WEEK_OF_MONTH=2,DAY_OF_MONTH=7,DAY_OF_YEAR=189,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=1,AM_PM=1,HOUR=7,HOUR_OF_DAY=19,MINUTE=16,SECOND=5,MILLISECOND=427,ZONE_OFFSET=0,DST_OFFSET=0]
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> Message:
7/7 12:16:10 [32361] GAHP[32379] (stderr) -> org.globus.exec.generated.StagingFaultType: Staging error for RSL element fileStageIn.
7/7 12:16:10 [32361] GAHP[32379] <- 'RESULTS'
7/7 12:16:10 [32361] GAHP[32379] -> 'R'
7/7 12:16:10 [32361] GAHP[32379] -> 'S' '1'
7/7 12:16:10 [32361] GAHP[32379] -> '2' 'https://142.104.60.57:8443/wsrf/services/ManagedExecutableJobService?ea39ea40-4c58-11dd-898e-fbb5210b5f9c' 'Failed' 'Staging error for RSL element fileStageIn.' '0'
7/7 12:16:10 [32361] (5334.0) gram callback: state Failed, fault Staging error for RSL element fileStageIn., exit code 0
7/7 12:16:10 [32361] (5334.0) doEvaluateState called: gmState GM_SUBMITTED, globusState 64
7/7 12:16:10 [32361] (5334.0) globus state change: StageIn -> Failed
7/7 12:16:10 [32361] (5334.0) gm state change: GM_SUBMITTED -> GM_FAILED
7/7 12:16:10 [32361] GAHP[32379] <- 'GT4_GRAM_JOB_DESTROY 9 https://142.104.60.57:8443/wsrf/services/ManagedExecutableJobService?ea39ea40-4c58-11dd-898e-fbb5210b5f9c'
7/7 12:16:10 [32361] GAHP[32379] -> 'S'
7/7 12:16:10 [32361] (5334.0) doEvaluateState called: gmState GM_FAILED, globusState 4
7/7 12:16:10 [32361] GAHP[32379] <- 'RESULTS'
7/7 12:16:10 [32361] GAHP[32379] -> 'R'
7/7 12:16:10 [32361] GAHP[32379] -> 'S' '1'
7/7 12:16:10 [32361] GAHP[32379] -> '9' '0' 'NULL'
7/7 12:16:10 [32361] (5334.0) doEvaluateState called: gmState GM_FAILED, globusState 4
7/7 12:16:10 [32361] (5334.0) gm state change: GM_FAILED -> GM_HOLD
7/7 12:16:10 [32361] (5334.0) gm state change: GM_HOLD -> GM_DELETE
7/7 12:16:15 [32361] in doContactSchedd()
7/7 12:16:15 [32361] Error loading private key from file7/7 12:16:15 [32361] Error initializing client security context
7/7 12:16:15 [32361] Error creating SSL context
7/7 12:16:15 [32361] SSL Authentication fails, terminating
7/7 12:16:15 [32361] This process has a valid certificate & key
7/7 12:16:15 [32361] querying for removed/held jobs
7/7 12:16:15 [32361] Using constraint ((Owner=?="seangwm"&&JobUniverse==9)) && ((Managed =!= "ScheddDone")) && (JobStatus == 3 || JobStatus == 4 || (JobStatus == 5 && Managed =?= "External"))
7/7 12:16:15 [32361] Fetched 0 job ads from schedd
7/7 12:16:15 [32361] Updating classad values for 5334.0:
7/7 12:16:15 [32361] GlobusDelegationUri = UNDEFINED
7/7 12:16:15 [32361] GridftpUrlBase = UNDEFINED
7/7 12:16:15 [32361] GlobusSubmitId = UNDEFINED
7/7 12:16:15 [32361] GridJobId = UNDEFINED
7/7 12:16:15 [32361] GlobusStatus = 32
7/7 12:16:15 [32361] JobStatus = 5
7/7 12:16:15 [32361] EnteredCurrentStatus = 1215458170
7/7 12:16:15 [32361] HoldReason = "Globus error: Staging error for RSL element fileStageIn."
7/7 12:16:15 [32361] HoldReasonCode = 0
7/7 12:16:15 [32361] HoldReasonSubCode = 0
7/7 12:16:15 [32361] ReleaseReason = UNDEFINED
7/7 12:16:15 [32361] NumSystemHolds = 1
7/7 12:16:15 [32361] Managed = "Schedd"
7/7 12:16:15 [32361] No jobs left, shutting down
7/7 12:16:15 [32361] leaving doContactSchedd()
7/7 12:16:15 [32361] Got SIGTERM. Performing graceful shutdown.
7/7 12:16:15 [32361] Started timer to call main_shutdown_fast in 1800 seconds
7/7 12:16:15 [32361] **** condor_gridmanager (condor_GRIDMANAGER) EXITING WITH STATUS 0