Re: [HTCondor-users] condor_adstash hard limit

[Jason Patton <jpatton@xxxxxxxxxxx>]

Robin,

We definitely would welcome any and all PRs to improve condor_adstash. It was originally a set of scripts that were used internally to put ads in Elasticsearch, and only then after was modified and packaged to work for "generic" HTCondor pools, and we certainly could have overlooked some finer points!

Thanks

Jason

On Wed, Aug 4, 2021 at 9:56 AM Hofsaess, Robin (ETP) <robin.hofsaess@xxxxxxx> wrote:

Hi Jason,

thank you! This should do the job.
Obviously, we already changed this at some point to 50000 (but weren't aware of this ð) so that this became our hard limit.


On top of that, I found another little issue (not worth a pull request I guess) while searching for a solution:
https://github.com/htcondor/htcondor/blob/db44d1948122d10b98bf101acfbd67c3f109d8f1/src/condor_scripts/adstash/config.py#L86

This never evaluates to None, even if {name}_ES_PASSWORD_FILE  is not set, since "es_password_file": p.get(f"{name}_ES_PASSWORD_FILE") becomes an empty string in this case:

$ condor_adstash --standalone
ERROR:root:Fatal error while trying to read password file .
Traceback (most recent call last):
  File "/usr/libexec/condor/adstash/config.py", line 92, in get_htcondor_config
    with passwd.open() as f:
  File "/usr/lib64/python3.6/pathlib.py", line 1183, in open
    opener=self._opener)
IsADirectoryError: [Errno 21] Is a directory: '.'

I fixed this for me by adding:

if conf.get("es_password_file") is not None and conf.get("es_password_file") != "":

To make sure, the string is not empty.

Thanks again and have a nice day everyone!

Cheers,
Robin

---

Von: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> im Auftrag von Jason Patton <jpatton@xxxxxxxxxxx>
Gesendet: Mittwoch, 4. August 2021 16:22:54
An: HTCondor-Users Mail List
Betreff: Re: [HTCondor-users] condor_adstash hard limit

 

Hi Robin,

When using the Python bindings, history queries are considered remote queries. I *think* the limit is set by the value of HISTORY_HELPER_MAX_HISTORY (which has a default of 10,000) in the schedd's config, but I haven't tested this myself. See if changing that allows condor_adstash to query for more ads?

Jason Patton

On Wed, Aug 4, 2021 at 9:11 AM Hofsaess, Robin (ETP) <robin.hofsaess@xxxxxxx> wrote:

-------------------------------------------

$ condor_version
$CondorVersion: 9.1.2 Jul 29 2021 BuildID: 552044 PackageID: 9.1.2-1 $
$CondorPlatform: x86_64_CentOS7 $
-------------------------------------------

Hi everyone,

I have a question concerning condor_adstash:

We want to put our complete condor history into an elasticsearch DB with condor_adstash.

Therefore, we changed: https://github.com/htcondor/htcondor/blob/db44d1948122d10b98bf101acfbd67c3f109d8f1/src/condor_scripts/adstash/history.py#L68

history_iter = schedd.history(
                constraint="true",
                projection=[],
                match = -1,  # changed
                #match=min(10000, args.schedd_history_max_ads),
                since=since_str
            )

to get all entries, according to: https://htcondor.readthedocs.io/en/v8_8/apis/python-bindings/api/htcondor.html

But then, we experience a hard limit of 50000 entries per Schedd.

Does anyone know, where this limit comes from?

Can this be changed?

It seems that the HistoryIterator (https://github.com/htcondor/htcondor/blob/6809756bb368f8c77c4e1d17730addb0a0f9940d/src/python-bindings/schedd.cpp#L435) is the reason, as it only has 50000 entries. But I don't see any obvious limitation.

Thanks for your help!

Cheers,

Robin

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/