[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Accessing historical condor usage statistics
- Date: Mon, 04 Jan 2016 13:30:51 -0600
- From: Todd Tannenbaum <tannenba@xxxxxxxxxxx>
- Subject: Re: [HTCondor-users] Accessing historical condor usage statistics
On 12/30/2015 4:35 AM, Hermann Fuchs wrote:
New years eve is a very good time to look back and generate statistics
of our cluster.
I would like to access historical usage statistics of our condor cluster
(for the last 5 years).
Especially I am interested in the amount of CPU hours used, the number
of jobs (idle, running etc) and the active users on a monthly basis.
Another granularity would be a nice bonus
In principle I want to get information similar to condor_userprio, but
on a monthly basis.
TimeFrame User RessourcesInUse/RunningJobs IdleJobs
Feb 2011 a@xxxxx 17 255 253.2
How could I get this data? Since start of the cluster
POOL_HISTORY_DIR = /full/path/to/directory/to/store/historical/data
KEEP_POOL_HISTORY = True
were set correctly.
I've been looking for hours and so far have found nothing useful for
Well, since you set KEEP_POOL_HISTORY to True since the start, in theory
you should be able to use the "condor_stats" command line tool.
condor_stats -orgformat -lastmonth -usergroupquery a@xxxxx
should at least be able to give you info about how many jobs were
running and idle for given user for a given period of time. I don't
have KEEP_POOL_HISTORY set to True, so I cannot actually try the above
command, but see the condor_stats man page at
You can specify a specific date range.
Above I say "in theory" because
1. the default for POOL_HISTORY_MAX_STORAGE looks very small (10 MB?),
so unless you increased that I don't know how far back your data will
2. I have not personally used the condor_stats command for years; these
days here at UW we instead use the condor_gangliad to send usage info to
Ganglia and also use the condor_pool_job_report script to generate daily
Although it won't help you to look backwards, you may be pleased to hear
we have been working on a replacement for condor_stats and friends. The
idea is to have basic aggregate historical usage information stored on
the central manager simple JSON files, and pointing your web browser at
the central manager will give you charts and tables you can download. We
are aiming for this functionality to debut in the currently development
series sometime this winter if time permits. You can follow our progress at
Hope the above helps and happy new year,