[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Efficiency & centralization of global information gathering?
- Date: Sat, 07 Jan 2017 00:49:43 +0000
- From: John M Knoeller <johnkn@xxxxxxxxxxx>
- Subject: Re: [HTCondor-users] Efficiency & centralization of global information gathering?
Generic ads are actually at specific type of ad. What you want to use is
What you actually want is condor_status -subsytem 'mytype'
This will make a generic query to the collector for ads of type 'mytype'
On the other hand, if your ads actually ARE of type generic, then condor_status -generic is the correct query to use.
Hint: you can look at the collector log to see what it actually sees for any condor_status query, grep for 'Query info'
From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Fischer, Max (SCC)
Sent: Thursday, January 5, 2017 4:00 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Efficiency & centralization of global information gathering?
1. Nice! I have to try that myself. ^^
Can you pull your custom classad also with condor_status -generic (instead of -any)?
Not entirely sure how condor does the filtering, but it may be more efficient.
2. Concerning the "modifying startd ClassAds" approach:
You can push values to any startd using condor_update_machine_ad . I can confirm this is working persistently as long as the startd lives (yes, it did hurt a lot in our case).
Scalability wise, it shouldn't matter whether you push or pull information: That blob of meta-data seems negligible compared to what jobs consume.
If your workers go up and down a lot (e.g. VMs), pulling is easier to keep synchronised. If the value changes infrequently, pushing is more responsive.
Personally, I prefer every startd pulling its own information. Makes it easier with heterogeneous resources.
> Am 05.01.2017 um 01:11 schrieb Michael Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>:
> Thanks for that suggestion! For my FlexLM problem, I just wrote a quick Perl script which you call like so:
> flexlm2classad lmstat -a
> This runs the "lmstat -a" command (or you can feed it flexlm data on stdin), and converts it into a ClassAd that looks like so:
> MyType = "Generic"
> Name = "FlexLM"
> FlexLM_Available_a_spaceclaim_dirmod = 6
> FlexLM_Available_acfd = 3
> FlexLM_Available_acfd_flo = 1
> FlexLM_Available_agppi = 8
> The identifier is the feature name for each license, and it's derived from the "Users of" lines like so:
> Users of a_spaceclaim_dirmod: (Total of 8 licenses issued; Total of 2 licenses in use)
> Then this can be pulled out for use by a startd_cron job or what have you with:
> condor_status -any -constraint 'Name == "FlexLM"' -af:lrng FlexLM_Available_a_spaceclaim_dirmod
> I noticed that in the help output for condor_advertise, there's a "MERGE_STARTD_AD" option, but it's not mentioned in the man page and it doesn't seem to let me add an attribute to an existing startd ad even if I structure it as a Query type like the invalidate commands. Maybe someone from the CHTC pantheon can enlighten us on this point.
> -Michael Pelletier.
> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Fischer, Max (SCC)
> Sent: Wednesday, January 04, 2017 1:59 PM
> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
> Subject: Re: [HTCondor-users] Efficiency & centralization of global information gathering?
> Hi Michael,
> I've found this to be best solved outside of Condor.
> 1. Have a regular cron job *somewhere* fetch the data once.
> 2. Provide that data via files on shared filesystems.
> 3. Have startd_cron read from the file.
> 4. ???
> 5. Profit
> The trick is just to have 1. and 3. separate. There's no problem having 1. create proper classad already, and 3. just using cat.
> Note that using files for 2. is historical laziness on my part:
> You can just as well publish this information via custom ClassAds. I think condor_advertise with UPDATE_AD_GENERIC should do the trick.
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> The archives can be found at: