[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Event log to kafka-like system


If you're a python person, you can additionally use the python bindings to pull events from the event log or the job queue.  Since it's inotify-based, it would work well with event loops and give relatively near-time information about job status.

Now, that's useful if you already have a python-based framework you'd like to integrate job state into.  Logstash is a bit more of a prepackaged / general-purpose tool.


> On Jul 4, 2017, at 3:34 PM, Kevin Retzke <kretzke@xxxxxxxx> wrote:
> We currently use Logstash to tail the global event log and send it directly to Elasticsearch [1]. Logstash can easily send events to Kafka instead, or just about anywhere else (e.g. we're putting RabbitMQ into our pipeline). We've found having the event log in Elasticseach has been hugely useful for system analytics and troubleshooting.
> [1] https://github.com/fifemon/logstash-config/blob/master/condor.logstash.conf
> /Kevin
> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Samik Raychaudhuri <samikr@xxxxxxxxx>
> Sent: Saturday, July 1, 2017 9:39 AM
> To: HTCondor-Users Mail List
> Subject: [HTCondor-users] Event log to kafka-like system
> Hello,
> Wanted to process the event logs from a submitted job as they come in. Is there a way to get the logs in an event queue like kafka (http://kafka.apache.org/)? Is there a socket from where I can directly listen to, and read the events using an external program?
> Apache Kafka
> kafka.apache.org
> Apache Kafka: A Distributed Streaming Platform. ... Kafkaâ is used for building real-time data pipelines and streaming apps.
> Just exploring!
> Regards,
> -Samik
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/