[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] life without quill

On 08/16/2010 11:29 PM, Mag Gam wrote:
I just setup a simple instance without quill and I was wondering how
people get around the condor_history slowness problem? I created
100,000 test jobs and I want to get information about job, 99,999 and
it takes a while however on quill its instant because of INDEXing. Is
there anything like that for non-quill users?


How are you running your history queries? I run one query with -format to generate a data set that is then process that subset of data repeatedly with other tools.

BTW, you pay the initial -format query cost in all cases. It's just a matter of if you pay it a little at a time or all at once. You should avoid paying it more than once.

If you like SQL -

$ condor_history -format "%d," ClusterId -format "%d," ProcId -format "%d," JobStatus -format "%d," QDate -format "%d," CompletionDate -format "%s," Owner -format "%s\n" Cmd > data.csv

$ sqlite3 data
SQLite version 3.6.22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> CREATE TABLE data(cluster int, proc int, status int, qdate int, completiondate int, owner varchar(16), cmd varchar(32));
sqlite> .separator ","
sqlite> .import data.csv data
sqlite> SELECT cluster, proc, owner FROM data WHERE status != 3;

JobStatus == 3 means the job was removed and the CompletionData is probably 0. So all the jobs were removed, and just a small set for this example,

sqlite> SELECT COUNT(*) FROM data WHERE owner = 'matt';