Re: [HTCondor-users] condor_history using the python bindings

[Jose Caballero <jcaballero.hep@xxxxxxxxx>]

Hi Brian,

that is one possible explanation.
Another one, which is what I think happened in that case, is that there was a file rotation exactly after I ran the command and I tried the python one. So ~half of the information was lost between tests. Makes sense?

Cheers,
Jose



On Jun 29, 2015, at 17:53, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:

> Hi Jose,
> 
> By default, the condor_history command parses the local files from the filesystem.  The python bindings contact the schedd, which streams the response over the network.  condor_history can similarly contact the schedd over the network if you use the -pool/-name options.
> 
> To prevent a DoS, the network-facing query has an internal limit (10,000 ads) of how much history it will read.  I suspect that you are hitting that limit via the python bindings.  You can verify this by doing the remote condor_history.
> 
> Brian
> 
>> On Jun 24, 2015, at 1:20 PM, Jose Caballero <jcaballero.hep@xxxxxxxxx> wrote:
>> 
>> Hmm. I have a followed up question.
>> I understand that I can use 0 as last parameter to get the entire list
>> of jobs, is that correct?
>> Running from python a given query I get 91 jobs. Running the same
>> command from command line I get 167.
>> All items in first output (python) are also listed in the second
>> output (command line).
>> I wonder why the command line tool gives more results than the python one...
>> 
>> Jose
>> 
>> 
>> 2015-06-24 13:16 GMT-04:00 Jose Caballero <jcaballero.hep@xxxxxxxxx>:
>>> you are right, I removed the actual "-constraint" string and now it works.
>>> Thanks a lot !!!
>>> 
>>> Cheers,
>>> Jose
>>> 
>>> 2015-06-24 12:00 GMT-04:00 Iain Bradford Steers <iain.steers@xxxxxxx>:
>>>> Hi Jose,
>>>> 
>>>> It's been a while since i looked at our usage of history in python and I'm not at a connected pc atm.
>>>> 
>>>> However, I believe you have a redudant -constraint in there.
>>>> 
>>>> It should be:
>>>> 
>>>> hist = schedd.history( "'x509UserProxyVoName =?= "{0}"", ['ClusterId'], 4)
>>>> 
>>>> Cheers, Iain
>>>> ________________________________________
>>>> From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of Jose Caballero [jcaballero.hep@xxxxxxxxx]
>>>> Sent: 24 June 2015 16:42
>>>> To: Condor-Users Mail List
>>>> Subject: [HTCondor-users] condor_history using the python bindings
>>>> 
>>>> Hi,
>>>> 
>>>> context:
>>>> 
>>>>   # uname -r
>>>>   2.6.32-504.el6.x86_64
>>>> 
>>>>   # condor_version
>>>>   $CondorVersion: 8.2.3 Sep 30 2014 BuildID: 274619 $
>>>>   $CondorPlatform: x86_64_RedHat6 $
>>>> 
>>>> 
>>>> I am trying to run a command like this (*) using the python bindings.
>>>> That command runs perfectly on the command line.
>>>> However, I am not sure I totally understand the meaning of the
>>>> description for the parameters provided by the inline documentation
>>>> (**).
>>>> I have tried something like this (***), but the output is an empty
>>>> iterator. Looping over it gives nothing.
>>>> Is this (****) the only way at the time to run condor_history within python?
>>>> 
>>>> Any tip is more than welcome.
>>>> Thanks a lot in advance.
>>>> Cheers,
>>>> Jose
>>>> 
>>>> (*)
>>>> 
>>>> # condor_history -format "%s \n" ClusterId -constraint 'MY_OWN_CLASSAD
>>>> == "value"' -match 4
>>>> 
>>>> (**)
>>>> 
>>>>>>> help(htcondor.Schedd.history)
>>>> Help on method history:
>>>> 
>>>> history(...) unbound htcondor.Schedd method
>>>>   history( (Schedd)arg1, (object)arg2, (list)arg3, (int)arg4) ->
>>>> HistoryIterator :
>>>>       Request records from schedd's history
>>>>       :param requirements: Either a ExprTree or a string that can be
>>>> parsed as an expression; requirements all returned jobs should match.
>>>>       :param projection: The attributes to return; an empty list
>>>> signifies all attributes.
>>>>       :param match: Number of matches to return.
>>>>       :return: An iterator for the matching job ads
>>>> (END)
>>>> 
>>>> (***)
>>>> 
>>>>>>> import htcondor
>>>>>>> schedd = htcondor.Schedd()
>>>>>>> out = schedd.history("-constraint 'MY_OWN_CLASSADD == \"value\"'", ['ClusterId'], 4)
>>>> 
>>>> (****)
>>>> 
>>>>>>> fd = os.popen("condor_history -l -constraint 'MY_OWN_CLASSAD == \"value\"' -match 4")
>>>>>>> ads = classad.parseOldAds(fd)
>>>>>>> print [ad["ClusterId"] for ad in ads]
>>>> _______________________________________________
>>>> HTCondor-users mailing list
>>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>>> subject: Unsubscribe
>>>> You can also unsubscribe by visiting
>>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>> 
>>>> The archives can be found at:
>>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>>> _______________________________________________
>>>> HTCondor-users mailing list
>>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>>> subject: Unsubscribe
>>>> You can also unsubscribe by visiting
>>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>> 
>>>> The archives can be found at:
>>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>> 
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/