Re: [HTCondor-users] condor_history using the python bindings

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

This certainly makes sense.

Unfortunately, the situation (history file rotation) is very hard to detect.  Perhaps itâs something for the HTCondor team to ponder for future releases.

Brian

> On Jun 29, 2015, at 2:04 PM, Jose Caballero <jcaballero.hep@xxxxxxxxx> wrote:
> 
> Hi Brian,
> 
> that is one possible explanation.
> Another one, which is what I think happened in that case, is that there was a file rotation exactly after I ran the command and I tried the python one. So ~half of the information was lost between tests. Makes sense?
> 
> Cheers,
> Jose
> 
> 
> 
> On Jun 29, 2015, at 17:53, Brian Bockelman <bbockelm@xxxxxxxxxxx> wrote:
> 
>> Hi Jose,
>> 
>> By default, the condor_history command parses the local files from the filesystem.  The python bindings contact the schedd, which streams the response over the network.  condor_history can similarly contact the schedd over the network if you use the -pool/-name options.
>> 
>> To prevent a DoS, the network-facing query has an internal limit (10,000 ads) of how much history it will read.  I suspect that you are hitting that limit via the python bindings.  You can verify this by doing the remote condor_history.
>> 
>> Brian
>> 
>>> On Jun 24, 2015, at 1:20 PM, Jose Caballero <jcaballero.hep@xxxxxxxxx> wrote:
>>> 
>>> Hmm. I have a followed up question.
>>> I understand that I can use 0 as last parameter to get the entire list
>>> of jobs, is that correct?
>>> Running from python a given query I get 91 jobs. Running the same
>>> command from command line I get 167.
>>> All items in first output (python) are also listed in the second
>>> output (command line).
>>> I wonder why the command line tool gives more results than the python one...
>>> 
>>> Jose
>>> 
>>> 
>>> 2015-06-24 13:16 GMT-04:00 Jose Caballero <jcaballero.hep@xxxxxxxxx>:
>>>> you are right, I removed the actual "-constraint" string and now it works.
>>>> Thanks a lot !!!
>>>> 
>>>> Cheers,
>>>> Jose
>>>> 
>>>> 2015-06-24 12:00 GMT-04:00 Iain Bradford Steers <iain.steers@xxxxxxx>:
>>>>> Hi Jose,
>>>>> 
>>>>> It's been a while since i looked at our usage of history in python and I'm not at a connected pc atm.
>>>>> 
>>>>> However, I believe you have a redudant -constraint in there.
>>>>> 
>>>>> It should be:
>>>>> 
>>>>> hist = schedd.history( "'x509UserProxyVoName =?= "{0}"", ['ClusterId'], 4)
>>>>> 
>>>>> Cheers, Iain
>>>>> ________________________________________
>>>>> From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of Jose Caballero [jcaballero.hep@xxxxxxxxx]
>>>>> Sent: 24 June 2015 16:42
>>>>> To: Condor-Users Mail List
>>>>> Subject: [HTCondor-users] condor_history using the python bindings
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> context:
>>>>> 
>>>>>  # uname -r
>>>>>  2.6.32-504.el6.x86_64
>>>>> 
>>>>>  # condor_version
>>>>>  $CondorVersion: 8.2.3 Sep 30 2014 BuildID: 274619 $
>>>>>  $CondorPlatform: x86_64_RedHat6 $
>>>>> 
>>>>> 
>>>>> I am trying to run a command like this (*) using the python bindings.
>>>>> That command runs perfectly on the command line.
>>>>> However, I am not sure I totally understand the meaning of the
>>>>> description for the parameters provided by the inline documentation
>>>>> (**).
>>>>> I have tried something like this (***), but the output is an empty
>>>>> iterator. Looping over it gives nothing.
>>>>> Is this (****) the only way at the time to run condor_history within python?
>>>>> 
>>>>> Any tip is more than welcome.
>>>>> Thanks a lot in advance.
>>>>> Cheers,
>>>>> Jose
>>>>> 
>>>>> (*)
>>>>> 
>>>>> # condor_history -format "%s \n" ClusterId -constraint 'MY_OWN_CLASSAD
>>>>> == "value"' -match 4
>>>>> 
>>>>> (**)
>>>>> 
>>>>>>>> help(htcondor.Schedd.history)
>>>>> Help on method history:
>>>>> 
>>>>> history(...) unbound htcondor.Schedd method
>>>>>  history( (Schedd)arg1, (object)arg2, (list)arg3, (int)arg4) ->
>>>>> HistoryIterator :
>>>>>      Request records from schedd's history
>>>>>      :param requirements: Either a ExprTree or a string that can be
>>>>> parsed as an expression; requirements all returned jobs should match.
>>>>>      :param projection: The attributes to return; an empty list
>>>>> signifies all attributes.
>>>>>      :param match: Number of matches to return.
>>>>>      :return: An iterator for the matching job ads
>>>>> (END)
>>>>> 
>>>>> (***)
>>>>> 
>>>>>>>> import htcondor
>>>>>>>> schedd = htcondor.Schedd()
>>>>>>>> out = schedd.history("-constraint 'MY_OWN_CLASSADD == \"value\"'", ['ClusterId'], 4)
>>>>> 
>>>>> (****)
>>>>> 
>>>>>>>> fd = os.popen("condor_history -l -constraint 'MY_OWN_CLASSAD == \"value\"' -match 4")
>>>>>>>> ads = classad.parseOldAds(fd)
>>>>>>>> print [ad["ClusterId"] for ad in ads]
>>>>> _______________________________________________
>>>>> HTCondor-users mailing list
>>>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>>>> subject: Unsubscribe
>>>>> You can also unsubscribe by visiting
>>>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>>> 
>>>>> The archives can be found at:
>>>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>>>> _______________________________________________
>>>>> HTCondor-users mailing list
>>>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>>>> subject: Unsubscribe
>>>>> You can also unsubscribe by visiting
>>>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>>>> 
>>>>> The archives can be found at:
>>>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>>> _______________________________________________
>>> HTCondor-users mailing list
>>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting
>>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>> 
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/htcondor-users/
>> 
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>> 
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/