[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[HTCondor-users] Condor SOAP API and queue jobs.



Hello.

I'm getting the first steps using Python and HTCondor SOAP API (condor v7.8.5). I'm sending single jobs with the SOAP API cycle with success.

It's easy to send jobs with a simple submit description file like this:

Universe   = vanilla
Executable = /bin/sleep
Arguments  = 30
Log        = simple.log
Output     = simple.out
Error      = simple.error
Queue

However, the problem I've run into is when I want to send a number of queued programs, for example:

Universe   = vanilla
Executable = /bin/sleep
Arguments  = 30
Log        = simple.log
Output     = simple.out
Error      = simple.error
Queue 150

I'm using the following Python code in order to send this 150 jobs:

########## begin code ############
from suds.client import Client
import logging
import sys

CONDOR_HOST      = "server"
SCHEDD_PORT      = "8080"
SCHEDD_LOCATION  = "http://" + CONDOR_HOST + ":" + SCHEDD_PORT
WSDL_SCHEDD_FILE = "file:condorSchedd.wsdl"

def updateAdProperty(job, name, type=None, value=None):
    for i in range(len(job[1][0])):
        if (job[1][0][i].name == name):
            if type:
                job[1][0][i].type = type
            if value:
                job[1][0][i].value = value
            return True
    return False

condor_schedd = Client(WSDL_SCHEDD_FILE, location=SCHEDD_LOCATION)

# Job submission cycle:
# 1. beginTransaction.
# 2. newCluster
# 3. newJob
# 4. createJobTemplate
# 5. submit
# 6. commitTransaction
#
transaction = condor_schedd.service.beginTransaction(10);
transactionId = transaction[1]
print "TransactionId: %s" % transactionId

cluster = condor_schedd.service.newCluster(transactionId)
clusterId=cluster[1]
print "ClusterId: %s" % clusterId

for i in xrange(150):
        print "newJob"
        job = condor_schedd.service.newJob(transactionId, clusterId)
        jobId = job[1]
        print "createJobTemplate"
        job = condor_schedd.service.createJobTemplate(clusterId, jobId, "condor", 5, "/bin/sleep", "30", "")
        updateAdProperty(job, "LeaveJobInQueue", value="FALSE")
        jobAd = job[1]
        print "submit jobId -> %s" % jobId
        print "submit"
        result = condor_schedd.service.submit(transactionId, clusterId, jobId, jobAd)

result = condor_schedd.service.commitTransaction(transactionId)
condor_schedd.service.requestReschedule();
#res = condor_schedd.service.closeSpool(transaction, clusterId, jobId)
print result

############ end code #####################


The job is sent 150 times for the same clusterID, however the submit loop is quite slow, probably due to the HTTP connections.

Please, Is this the correct way to send a set of jobs?
Is there any way to speed up this submission cycle, like "condor_submi" do it?

Best Regards.

--
Javi Roman