Re: [HTCondor-users] python api about history

Mailing List Archives Public Access	UW Madison Computer Sciences Department Computer Systems Lab

What version of HTCondor are you running?

There was a bug regarding sorting of history information that was fixed in 8.2.10.

https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=5305

...Tim

On 11/17/2016 07:44 PM, jiangxw@xxxxxxxxxxxxxxx wrote:

Dear Biranï

  Thanks for the advice about job history.

I tryed to limite the amount of jobs from history file.

Condor_history doesn't looks over history files in time ordor,

so I can not get the latest jobs from 10K jobs limited sometimes.

Whether in our environment there's something wrong?

Cheers,

Jiang Xiaowei

NAMEïJiang Xiaowei

MAILïjiangxw@xxxxxxxxxxxxxxx

TELï010 8823 6024

DEPARTMENTïComputing Center of IHEP

From: Brian Bockelman

Date: 2016-11-18 09:01

To: HTCondor-Users Mail List

Subject: Re: [HTCondor-users] python api about history

Also worth noting that the remote history option is limited to 10k jobs by default. So, regardless of how much data is on disk, no more than 10k ads should be parsed.

Sent from my iPhone

On Nov 15, 2016, at 5:31 AM, Fischer, Max (SCC) <max.fischer@xxxxxxx> wrote:

Hi,

since history apparently is implemented as a generator, you should break by *yourself* in python code. The history itself has no way of knowing that an arbitrary constraint is satisfied at a certain point.

Pseudocode:

   max_age = 60*30 # only care for jobs in the last 30 minutes
   new_jobs = []
   for job in htcondor.Schedd.history():
       if job['EnteredCurrentStatus'] < time.time() - max_age:
           break # exit early, only older jobs after this point
       new_jobs.append(job)

Cheers,
Max

Am 15.11.2016 um 11:09 schrieb jiangxw@xxxxxxxxxxxxxxx:

Thank you very mach.

Hope history() can provide a parameter setting history file like "-file" in condor_history command.

NAMEïJiang Xiaowei

MAILïjiangxw@mail.ihep.ac.cn

TELï010 8823 6024

DEPARTMENTïComputing Center of IHEP

From: Iain Steers

Date: 2016-11-15 17:56

To: HTCondor-Users Mail List

Subject: Re: [HTCondor-users] python api about history

Ah I was thinking of the since and stop constraints.

https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=5642,56

Looks like they haven't been added to the python bindings yet.

________________________________________

From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of jiangxw@xxxxxxxxxxxxxxx[jiangxw@xxxxxxxxxxxxxxx]

Sent: 15 November 2016 10:53

To: htcondor-users

Subject: Re: [HTCondor-users] python api about history

Hi Iain,

   Thanks for your advice.

   I can catch your meaning.

   I implemented that function like yours.

   And the space of history files in our environments is 132G totally.

   When I execute the script with the lines

       requirements = "EnteredCurrentStatus <= %s && EnteredCurrentStatus > %s" % (current_time,pre_time)

       job_history = self.schedd.history(requirements, projection, 0),

   the script will take too long times. I think self.schedd.history() will look over all 132G history files.

   How can I optimize it?

   Thanks.

Cheers,

Jiang Xiaowei

________________________________

NAMEïJiang Xiaowei

MAILïjiangxw@mail.ihep.ac.cn

TELï010 8823 6024

DEPARTMENTïComputing Center of IHEP

From: Iain Steers<mailto:iain.steers@xxxxxxx>

Date: 2016-11-15 17:16

To: HTCondor-Users Mail List<mailto:htcondor-users@xxxxxxxxxxx>

Subject: Re: [HTCondor-users] python api about history

Hi,

history() can take a Requirement constraint as one of its arguments.

In order to do this efficiently i would suggest something like the following.

start_dt = datetime.datetime.now() - datetime.timedelta(minutes=30)

start_stamp = calendar.timegm(d.timetuple())

schedd = htcondor.Schedd()

jobs = schedd.history('EnteredCurrentStatus >= {0}'.format(start_stamp))

This should return an iterator of all the jobs which were added to the history in the last half hour.

Cheers, Iain

________________________________________

From: HTCondor-users [htcondor-users-bounces@xxxxxxxxxxx] on behalf of jiangxw@xxxxxxxxxxxxxxx[jiangxw@xxxxxxxxxxxxxxx]

Sent: 15 November 2016 10:08

To: htcondor-users

Subject: [HTCondor-users] python api about history

Hi all,

   I used python api to get job history information then insert job information into database.

      The function is htcondor.Schedd.history().

   But when the amount of the history files in "/var/lib/condor/spool/" became large,

   htcondor.Schedd.history() will be executed slowly.

   I want to gain the job history information every half hour. So I just need to look over the history file generated last half hour.

   But the htcondor.Schedd.history() may look over all history files in "/var/lib/condor/spool" every time.

   I can not find the parameter from htcondor.Schedd.history() like "condor_history -file".

   Are there some ways to solve it?

   Thanks for help.

Best regards,

Jiang XiaoWei

________________________________

NAMEïJiang Xiaowei

MAILïjiangxw@mail.ihep.ac.cn

TELï010 8823 6024

DEPARTMENTïComputing Center of IHEP

_______________________________________________

HTCondor-users mailing list

To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a

subject: Unsubscribe

You can also unsubscribe by visiting

https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:

https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________

HTCondor-users mailing list

To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a

subject: Unsubscribe

You can also unsubscribe by visiting

https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:

https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________

HTCondor-users mailing list

To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a

subject: Unsubscribe

You can also unsubscribe by visiting

https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:

https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

-- 
Tim Theisen
Release Manager
HTCondor & Open Science Grid
Center for High Throughput Computing
Department of Computer Sciences
University of Wisconsin - Madison
4261 Computer Sciences and Statistics
1210 W Dayton St
Madison, WI 53706-1685
+1 608 265 5736

Mailing List Archives

Public Access

Re: [HTCondor-users] python api about history