Re: [HTCondor-users] condor_history using the python bindings

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

Hi Jose,

By default, the condor_history command parses the local files from the filesystem.  The python bindings contact the schedd, which streams the response over the network.  condor_history can similarly contact the schedd over the network if you use the -pool/-name options.

To prevent a DoS, the network-facing query has an internal limit (10,000 ads) of how much history it will read.  I suspect that you are hitting that limit via the python bindings.  You can verify this by doing the remote condor_history.

Brian

> On Jun 24, 2015, at 1:20 PM, Jose Caballero <jcaballero.hep@xxxxxxxxx> wrote:
> 
> Hmm. I have a followed up question.
> I understand that I can use 0 as last parameter to get the entire list
> of jobs, is that correct?
> Running from python a given query I get 91 jobs. Running the same
> command from command line I get 167.
> All items in first output (python) are also listed in the second
> output (command line).
> I wonder why the command line tool gives more results than the python one...
> 
> Jose
> 
> 
> 2015-06-24 13:16 GMT-04:00 Jose Caballero <jcaballero.hep@xxxxxxxxx>:
>> you are right, I removed the actual "-constraint" string and now it works.
>> Thanks a lot !!!
>> 
>> Cheers,
>> Jose
>> 
>> 2015-06-24 12:00 GMT-04:00 Iain Bradford Steers <iain.steers@xxxxxxx>:
>>> Hi Jose,
>>> 
>>> It's been a while since i looked at our usage of history in python and I'm not at a connected pc atm.
>>> 
>>> However, I believe you have a redudant -constraint in there.
>>> 
>>> It should be:
>>> 
>>> hist = schedd.history( "'x509UserProxyVoName =?= "{0}"", ['ClusterId'], 4)
>>> 
>>> Cheers, Iain
>>> ________________________________________
>>> From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of Jose Caballero [jcaballero.hep@xxxxxxxxx]
>>> Sent: 24 June 2015 16:42
>>> To: Condor-Users Mail List
>>> Subject: [HTCondor-users] condor_history using the python bindings
>>> 
>>> Hi,
>>> 
>>> context:
>>> 
>>>    # uname -r
>>>    2.6.32-504.el6.x86_64
>>> 
>>>    # condor_version
>>>    $CondorVersion: 8.2.3 Sep 30 2014 BuildID: 274619 $
>>>    $CondorPlatform: x86_64_RedHat6 $
>>> 
>>> 
>>> I am trying to run a command like this (*) using the python bindings.
>>> That command runs perfectly on the command line.
>>> However, I am not sure I totally understand the meaning of the
>>> description for the parameters provided by the inline documentation
>>> (**).
>>> I have tried something like this (***), but the output is an empty
>>> iterator. Looping over it gives nothing.
>>> Is this (****) the only way at the time to run condor_history within python?
>>> 
>>> Any tip is more than welcome.
>>> Thanks a lot in advance.
>>> Cheers,
>>> Jose
>>> 
>>> (*)
>>> 
>>> # condor_history -format "%s \n" ClusterId -constraint 'MY_OWN_CLASSAD
>>> == "value"' -match 4
>>> 
>>> (**)
>>> 
>>>>>> help(htcondor.Schedd.history)
>>> Help on method history:
>>> 
>>> history(...) unbound htcondor.Schedd method
>>>    history( (Schedd)arg1, (object)arg2, (list)arg3, (int)arg4) ->
>>> HistoryIterator :
>>>        Request records from schedd's history
>>>        :param requirements: Either a ExprTree or a string that can be
>>> parsed as an expression; requirements all returned jobs should match.
>>>        :param projection: The attributes to return; an empty list
>>> signifies all attributes.
>>>        :param match: Number of matches to return.
>>>        :return: An iterator for the matching job ads
>>> (END)
>>> 
>>> (***)
>>> 
>>>>>> import htcondor
>>>>>> schedd = htcondor.Schedd()
>>>>>> out = schedd.history("-constraint 'MY_OWN_CLASSADD == \"value\"'", ['ClusterId'], 4)
>>> 
>>> (****)
>>> 
>>>>>> fd = os.popen("condor_history -l -constraint 'MY_OWN_CLASSAD == \"value\"' -match 4")
>>>>>> ads = classad.parseOldAds(fd)
>>>>>> print [ad["ClusterId"] for ad in ads]
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>> 
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>> 
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/