Re: [HTCondor-users] Accessing historical condor usage statistics

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 12/30/2015 4:35 AM, Hermann Fuchs wrote:
> Seasons greetings!
>
> New years eve is a very good time to look back and generate statistics
> of our cluster.
>
> I would like to access historical usage statistics of our condor cluster
> (for the last 5 years).
> Especially I am interested in the amount of CPU hours used, the number
> of jobs (idle, running etc) and the active users on a monthly basis.
> Another granularity would be a nice bonus
>
> In principle I want to get information similar to condor_userprio, but
> on a monthly basis.
>
> For example:
> TimeFrame    User            RessourcesInUse/RunningJobs IdleJobs
> MonthylUsage(h)
> Feb 2011 a@xxxxx    17           255 253.2
>
> How could I get this data? Since start of the cluster
>
>      POOL_HISTORY_DIR = /full/path/to/directory/to/store/historical/data
>      KEEP_POOL_HISTORY = True
>
> were set correctly.
>
> I've been looking for hours and so far have found nothing useful for
> historical usage.

Well, since you set KEEP_POOL_HISTORY to True since the start, in theory 
you should be able to use the "condor_stats" command line tool. 
Something like
  condor_stats -orgformat -lastmonth -usergroupquery a@xxxxx
should at least be able to give you info about how many jobs were 
running and idle for given user for a given period of time.  I don't 
have KEEP_POOL_HISTORY set to True, so I cannot actually try the above 
command, but see the condor_stats man page at
    http://research.cs.wisc.edu/htcondor/manual/v8.4/condor_stats.html
You can specify a specific date range.

Above I say "in theory" because

1. the default for POOL_HISTORY_MAX_STORAGE looks very small (10 MB?), 
so unless you increased that I don't know how far back your data will 
actually go...

2. I have not personally used the condor_stats command for years; these 
days here at UW we instead use the condor_gangliad to send usage info to 
Ganglia and also use the condor_pool_job_report script to generate daily 
reports.  See

http://research.cs.wisc.edu/htcondor/manual/current/condor_pool_job_report.html

http://research.cs.wisc.edu/htcondor/manual/current/3_10Monitoring.html#42455

Although it won't help you to look backwards, you may be pleased to hear 
we have been working on a replacement for condor_stats and friends.  The 
idea is to have basic aggregate historical usage information stored on 
the central manager simple JSON files, and pointing your web browser at 
the central manager will give you charts and tables you can download. We 
are aiming for this functionality to debut in the currently development 
series sometime this winter if time permits. You can follow our progress at
  https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=5351



Hope the above helps and happy new year,
Todd