Re: [Condor-users] life without quill

[Matthew Farrellee <matt@xxxxxxxxxx>]

On 08/16/2010 11:29 PM, Mag Gam wrote:
> I just setup a simple instance without quill and I was wondering how
> people get around the condor_history slowness problem? I created
> 100,000 test jobs and I want to get information about job, 99,999 and
> it takes a while however on quill its instant because of INDEXing. Is
> there anything like that for non-quill users?
>
> TIA

How are you running your history queries? I run one query with -format 
to generate a data set that is then process that subset of data 
repeatedly with other tools.

BTW, you pay the initial -format query cost in all cases. It's just a 
matter of if you pay it a little at a time or all at once. You should 
avoid paying it more than once.

If you like SQL -

$ condor_history -format "%d," ClusterId -format "%d," ProcId -format 
"%d," JobStatus -format "%d," QDate -format "%d," CompletionDate -format 
"%s," Owner -format "%s\n" Cmd > data.csv

$ sqlite3 data
SQLite version 3.6.22
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> CREATE TABLE data(cluster int, proc int, status int, qdate int, 
completiondate int, owner varchar(16), cmd varchar(32));
sqlite> .separator ","
sqlite> .import data.csv data
sqlite> SELECT cluster, proc, owner FROM data WHERE status != 3;

JobStatus == 3 means the job was removed and the CompletionData is 
probably 0. So all the jobs were removed, and just a small set for this 
example,

sqlite> SELECT COUNT(*) FROM data WHERE owner = 'matt';
77

Best,


matt