[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] is there some kind of caching for condor_q requests ?

Hi all,

I'm the developer working on condor_watch_q, as you can perhaps tell by my name being on the GitHub account :)

If you're interesting in trying it out yourself, it can be installed via pip:

  pip install git+https://github.com/JoshKarpel/condor_watch_q

or you can simply copy condor_watch_q.py into your PATH (and remove the .py, if you'd like). After that, you should be able to run condor_watch_q to get the help message. We plan to eventually deploy it like any other condor CLI tool.

Our primary goal is to provide a pretty, streamlined tool for users to look at "live" job status without hammering the schedd. It's not going to cover the full range of things that condor_q can do (like look at full job ads) because it only knows what's in the event log.

Jin: watch_q can do an initial query to pick up event log paths. We can certainlyÂlet you query a non-local scheduler, but keep in mind that you'll need to be able to read those paths, which could be a problem if you don't have a shared filesystem. I've added it to our to-do list.

Jose: definitely interesting! I'll poke around.

Josh Karpel

On Fri, Feb 28, 2020 at 1:02 PM Jose Caballero <jcaballero.hep@xxxxxxxxx> wrote:
El vie., 28 feb. 2020 a las 17:29, Todd Tannenbaum
(<tannenba@xxxxxxxxxxx>) escribiÃ:
> On 2/28/2020 11:09 AM, Jin Mao wrote:
> > Users are always right. :)
> >
> > Maybe you can try to approach customers to introduce something like condor_wait to get job status from a user log file
> > instead of condor_q, if this is the reason to run condor_q more frequently than enough?
> >
> Hi Christoph and Jin,
> Jin had the same ideas we had :)
> We have developed a new tool called "condor_watch_q", which is similar to doing "watch condor_q", except is uses the job
> event log (i.e. the file created when you do 'log=<some_file>' in your submit file) instead of hitting the schedd. This
> tool should appear in the next 8.9 release. The tool is written in python on top of the HTCondor python bindings, so it
> should be easy for you to customize if you need to, and/or make pull requests back to us if you have improvements.
> Also, be aware of something similar that has existed in HTCondor for a long time (in HTCondor v8.6, v8.8, ...): the
> condor_q tool already has a "-userlog <file>" command line option. From the man page on condor_q:
> > -userlog file
> >Â Â Â(general option) Display jobs, with job information coming from a job event log, instead of from the real ClassAds from the condor_schedd daemon. This is most useful for automated testing of the status of jobs known to be in the given job event log, because it reduces the load on the condor_schedd. A job event log does not contain all of the job information, so some fields in the normal output of condor_q will be blank.

Hi Todd,

is this https://github.com/jose-caballero/libfactory/blob/master/libfactory/htcondorlib.py
Âsomething that could help here?
It's based on the python bindings, so no help for users running
condor_q from command line, but otherwise it caches the output of


HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at: