Re: [HTCondor-users] Programmatic access to match analysis?

[John M Knoeller <johnkn@xxxxxxxxxxx>]

Yes, that will get all of the Machine classad from the collector.   It's an EXPENSIVE query, so don't do it often
but this is essentially what the Negotiator does at the start of each negotiation cycle.  

If you are using partitionable slots and your pool doesn't allow pre-emption, you can ignore the dynamic slots.
This will result in a LOT less data (50% to 90% less data to transfer?).  

coll.query(htcondor.AdTypes.Startd, 'DynamicSlot =!= true')

And if you only want to know about slots you can have *now*, then you should fetch only idle slots, which
is what the negotiator does when the NEGOTIATOR_ALLOW_PREEMPTION knob is false.

-tj

-----Original Message-----
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Jose Caballero
Sent: Wednesday, January 30, 2019 3:08 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Programmatic access to match analysis?

Hi

this thread catch my attention.
Question for the experts: is not possible to get the classads
advertised by all the startd's to the collector using the python
bindings?
I see in the docs a line like this

         startd_ads = coll.query(htcondor.AdTypes.Startd)

Would that be the entire list of machine classads?
If yes, then you can do matchmaking for a job ad against that list,
without submitting, right?

Cheers,
Jose


El miÃ., 30 ene. 2019 a las 15:49, Ian McEwen (<mian@xxxxxxxxxxx>) escribiÃ:
>
> This is a decent start! We don't have C++ expertise in-house, so ideally
> this would be accessible some other fairly sane way -- python bindings
> or similar -- rather than making any attempt at parsing condor_q output
> or reimplementing the analysis code ourselves in one language or
> another.
>
> Similarly to what Michael said, it'd also be nice if there's a way to
> query the negotiator for concurrency limits, rather than having
> out-of-band knowledge of what the limits are set to and extracting it
> from the running jobs. But in both of these cases I do at least realize
> the officially-blessed way of doing this would be to use condor_q
> interactively and our system's way of doing things is out of the norm.
>
> Thanks for the response and the pointer to the condor_q code for this!
>
> On Tue, Jan 29, 2019 at 05:43:45PM +0000, John M Knoeller wrote:
> > If you are writing C++ code,  the better-analyze code is in HTCondor source at src\condor_q.V6\jobmatch.cpp. look at the doJobRunAnalysis function.
> >
> > Were you looking for programmatic access in a different language?
> >
> > By the way this code does matchmaking analysis only.  There is currently no code in HTCondor outside of the Negotiator itself that checks concurrency limits -  only the Negotiator knows what limits have been handed out already.
> >
> > -tj
> >
> > -----Original Message-----
> > From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Ian McEwen
> > Sent: Monday, January 28, 2019 6:34 PM
> > To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> > Subject: [HTCondor-users] Programmatic access to match analysis?
> >
> > Hi;
> >
> > This isn't something I'm expecting to have time to do soon, so I'm just
> > gathering information for now, but I'm wondering if there's an
> > appropriate way to get programmatic access to what `condor_q -analyze`
> > and `-better-analyze` output, especially for jobs that have not yet been
> > submitted.
> >
> > That is, given a job ad (constructed however is appropriate), is there a
> > good way to test, non-interactively and without submitting the ad,
> > whether there is a slot that meets the requirements (or, perhaps, if
> > there could be one with partitionable slot defragmentation?), whether
> > concurrency limits are met, etc.?
> >
> > As many of you probably know, we run a system where the direct
> > interaction with HTCondor (including building job ads and submitting
> > them) is handled by us, so users are using our web interface without
> > needing to know HTCondor itself, but lately we've been adding more
> > features that allow users to select their requirements more
> > specifically, which may increase the possibility that jobs will be
> > unable to match (or wait a long time to do so). I'd like to build
> > something that can give users an idea of whether their job is likely to
> > run soon, and why or why not.
> >
> > Thanks in advance for any help!
> > _______________________________________________
> > HTCondor-users mailing list
> > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/htcondor-users/
> >
> > _______________________________________________
> > HTCondor-users mailing list
> > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/htcondor-users/
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/