[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] condor rrd



We gave up on Quill but wrote some perl scripts that run every few minutes, parse the output from various condor_status commands and then update rrd files appropriately. It is all pretty easy to do using condor_status and the -format command. Perl has a module that makes updating rrd files straightforward, search on cpan for "RRDs". Although Perl lends itself towards text parsing, the condor_status -format command can make your output so easy to parse that you should have no issues doing it in any language.

If all you want is a basic view of the pool showing you machines in the pool and how many of them are running jobs then you would only need to pull out Machine, State, Activity, VirtualMachineID. If you are doing this then you might as well record info on Idle jobs as well as graphing this may help you decide whether you need an increase in capacity or just to try and get users to spread their jobs out. We went a bit further and grab information for each user as well so that our graphs are shaded in multiple colours indicating the proportion of execute nodes each user is running on.

The beauty of the rrd files is that we can keep masses of historical information but at much lower granularity. We keep 3 months of data at 5min intervals, 18 months at 2hr intervals and 60 months at 6hr intervals. All of that is handled by rrd and requires no additional work on our part after the rrd has been created. With 60 months of data at these levels of granularity, the rrd files are only about 300KB per attribute you are monitoring. I would recommend only putting one attribute in to each RRD file . This means that whenever you come across something else you want to record, you just need to get it to create additional rrd files as adding additional attributes to RRD files that already contain data is a pain.


Mike

-----Original Message-----
From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Henning Fehrmann
Sent: 07 September 2009 07:39
To: Condor-Users Mail List
Subject: Re: [Condor-users] condor rrd



Hi,

> Are there any tutorials or howtos to integrate condor and rrd? I would
> like to get a graph of total Pool usage. I am using quill and
> generating graphs but would like to see if anyone did any work with
> rrd -> quill -> condor.

We have been looking for similar things but endend up with writing our own view for 
the quill database:

cat <<EOF > /tmp/users_slot.sql
CREATE OR REPLACE VIEW jobs_status_full AS
select c.scheddname, c.owner, count(*) as count, p.jobstatus, c.jobuniverse
FROM clusterads_horizontal c
JOIN procads_horizontal p USING (cluster_id)
GROUP BY c.scheddname, c.owner, p. jobstatus, c.jobuniverse
order by scheddname, owner,count,jobstatus DESC;
grant select on jobs_status_full to quillreader;
EOF


psql quill quillwriter < /tmp/users_slot.sql

This view can be used to read out the pool usage.

A script is generating the rrd graphs using 

psql -h $quill_ip -U quillreader quill -c "select owner,count,jobstatus  from jobs_status_full where jobstatus=5 or (jobstatus=2 and not jobuniverse=8)"


Cheers,
Henning
_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at: 
https://lists.cs.wisc.edu/archive/condor-users/

----
Gloucester Research Limited believes the information provided herein is reliable. While every care has been taken to ensure accuracy, the information is furnished to the recipients with no warranty as to the completeness and accuracy of its contents and on condition that any errors or omissions shall not be made the basis for any claim, demand or cause for action.
The information in this email is intended only for the named recipient.  If you are not the intended recipient please notify us immediately and do not copy, distribute or take action based on this e-mail.
All messages sent to and from this email address will be logged by Gloucester Research Ltd and are subject to archival storage, monitoring, review and disclosure.
Gloucester Research Limited, 5th Floor, Whittington House, 19-30 Alfred Place, London WC1E 7EA.
Gloucester Research Limited is a company registered in England and Wales with company number 04267560.
----