Re: [HTCondor-users] Splitting condor_history by user/task?

[Peter MacKinnon <pmackinn@xxxxxxxxxx>]

On 01/28/2013 05:04 AM, Max Fischer wrote:
> Hi,
>
> our job-manager tool's Condor wrapper is showing increasing delays 
> when extracting information from condor_history (and to a lesser 
> extend, condor_q). This is especially the case with user pools that do 
> not split the history file regularly and have lots of user jobs, 
> preventing our wrapper from partitioning its search. The main problem 
> seems to be that regardless of constraints, condor_history will go 
> through the entire history file merely suppressing lines that do not 
> match the constraint.
>
> What I'm looking for is a way to tell Condor to split its history/q 
> file (or make a copy of the relevant parts) either for each user or 
> preferably for a dynamic key of the job/task. Something akin to job 
> Log in combination with EVENT_LOG_JOB_AD_INFORMATION_ATTR, but the 
> important thing is that we need to be able to define it solely through 
> the job so that we do not require the pool to be configured 
> specifically for us.
>
> Cheers,
> Max
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx 
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

Hi Max,

Not sure it's exactly what you're looking for but I've developed a 
contrib that (among other things) stores job history ads into mongodb. 
There's an existing tool to query the db much like condor_history based 
on various parameters, including a user name (which is indexed). 
Unfortunately there is no full ClassAd constraint expression parsing 
capability so your queries are really what you can do from a mongodb 
client such as the provided tool (written in pymongo).

http://htcondor-git.cs.wisc.edu/?p=condor.git;a=blob;f=src/condor_contrib/plumage/README 
(scroll to bottom for history tool details)

Let me know if you have further questions if any of that sounds 
potentially useful.

\Pete

--
Peter MacKinnon
Cloud BU/MRG Grid
Red Hat Inc.
Raleigh, NC