Re: [HTCondor-users] Managing dynamic services

[Stuart Anderson <anderson@xxxxxxxxxxxxxxxx>]

> On Nov 21, 2018, at 6:13 PM, Stuart Anderson <anderson@xxxxxxxxxxxxxxxx> wrote:
> 
>> On Nov 21, 2018, at 12:52 PM, Greg Thain <gthain@xxxxxxxxxxx> wrote:
>> 
>> On 11/16/18 6:57 PM, Stuart Anderson wrote:
>>> Is there a standard way for Condor to manage local privileged dynamic services on an execute machine depending on whether any currently matched jobs have requested it?
...
>> 
>> This is an interesting question, and I think there's at least one way to do this with HTCondor, but I have a couple of questions to put a finer point on potential solutions
> 
> The background/use case for this is Kafka. In particular, if there are Condor jobs running that want to access a popular set of messages from Kafka then a per-machine service should start to receive those data and perform some common processing for all jobs on the machine. Problems to avoid: 1) do not load down the network and Kafka brokers with thousands of extra copies of messages that will not be used; 2) Avoid the inefficiency of multiple jobs on the same execute machine requesting the same messages independent of each other.

Greg,
	Any further thoughts on this?

Thanks.

--
Stuart Anderson  anderson@xxxxxxxxxxxxxxxx
http://www.ligo.caltech.edu/~anderson