Re: [HTCondor-users] thoughts on HTCondor python bindings submit improvements

[John M Knoeller <johnkn@xxxxxxxxxxx>]

This may be more pythonic, but unfortunately it doesn't actually capture all of the capabilities of the queue statement.  there is also a count.  N jobs for each item in the list.  also, it really MUST be an iterator, not an explicit list because you also need to express this:

  executable = /bin/echo
  argument = file is $(filename)
  queue 10 filename matching *.py

or

  queue a,b,c from csvfile

where csvfile can have hundreds of thousands of lines, millions even. 

The queue statement is NOT an item in the hashtable, it's an operation that we apply to the hash table.    (The submit file syntax allows that keyword to appear multiple times.)

Of course you can always supply the submit hashtable as a python dict,  Adding the ability to supply it as a newline delimited string is just a convenience for people who are used to working that way (or who are getting the submit files from somewhere else).  

so you can do this

with open("filename.sub") as fh :
     subfile = fh.read()

with schedd.transaction() as txn : 
    htcondor.Submit(subfile).queue(txn)

and get exactly the same jobs queued as if you did 
    condor_submit filename.sub

Siince it's so easy to slurp a file into a string in python, I don't see any value in having the submit binding take a filename or a file handle rather than a string.   But having it accept a string rather than just a dict is the ONLY  way you can express full submit file capabilities.

-tj

-----Original Message-----

And besides, if you want more pythonic, "explicit is better than implicit".

hcondor.Submit( { "executable" : "/bin/echo",
		  "queue" : [
			{ "args" : "one" },
			{ "args" : "two" },
			{ "args" : "three" },
			{ "args" : "one" },
			{ "args" : "two" },
			{ "args" : "three" }
		  ] } )

or whatever your queue statement expands to. (Hardcore pythonistae can
creatively zip() a dict() there or something.)

Then add htcondor.SubmitFileToKwargs() that will read submit string/file
and return the above structure.

$.02
-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu