[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Adding new ClassAd functions to HTC
- Date: Tue, 11 Nov 2014 11:55:14 -0600
- From: Brian Bockelman <bbockelm@xxxxxxxxxxx>
- Subject: Re: [HTCondor-users] Adding new ClassAd functions to HTC
> On Nov 10, 2014, at 3:56 AM, Fischer, Max (SCC) <max.fischer@xxxxxxx> wrote:
> Hi all,
> I’m looking for a way to query a REST service on our worker nodes for job matching. Dumping the information into the job/worker ClassAds as lists is not possible due to the data volume.
> I’m constrained by having to execute the query on the worker node, but with information from the job, during START/RANK/… evaluation.
> It seems the only realistic solution is a custom ClassAd function, similar to the HDFS plugin . I’m lost at how to include a new ClassAd function into HTC itself, though. Is there any tutorial, guide or other information on this?
I think Dan Bradley wrote this up (very well) once - but I also think the tutorial disappeared when he moved on to a new job. :(
Here's a perhaps-simpler example custom function to start with:
Basically, you compile your function into a shared library and add the shared library to CLASSAD_USER_LIB.
One thing to consider is that the negotiator may evaluate the function many, many times (note - this means that having side-effects can be very bad!). If you want an example of a custom function whose results are cached, look here:
(Very similar idea to Stephen's HDFS function; I don't recall which code came first)
> Alternatively, is there a simpler way to execute an arbitrary query from within a ClassAd?
Nothing comes to mind.
PS - I'm working on a patch that allows you to write custom function in python, but it's not done / released yet.