Re: [HTCondor-users] Event log to kafka-like system

[Samik Raychaudhuri <samikr@xxxxxxxxx>]

Hi Brian,
Thanks - not in the python world - mostly in JVM :-) Will keep the python-based approach parked for now.
-Samik

On 05-Jul-17 6:25 AM, Brian Bockelman wrote:

Hi,

If you're a python person, you can additionally use the python bindings to pull events from the event log or the job queue.  Since it's inotify-based, it would work well with event loops and give relatively near-time information about job status.

Now, that's useful if you already have a python-based framework you'd like to integrate job state into.  Logstash is a bit more of a prepackaged / general-purpose tool.

Brian

On Jul 4, 2017, at 3:34 PM, Kevin Retzke <kretzke@xxxxxxxx> wrote:

We currently use Logstash to tail the global event log and send it directly to Elasticsearch [1]. Logstash can easily send events to Kafka instead, or just about anywhere else (e.g. we're putting RabbitMQ into our pipeline). We've found having the event log in Elasticseach has been hugely useful for system analytics and troubleshooting.

[1] https://github.com/fifemon/logstash-config/blob/master/condor.logstash.conf

/Kevin

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Samik Raychaudhuri <samikr@xxxxxxxxx>
Sent: Saturday, July 1, 2017 9:39 AM
To: HTCondor-Users Mail List
Subject: [HTCondor-users] Event log to kafka-like system
 
Hello,

Wanted to process the event logs from a submitted job as they come in. Is there a way to get the logs in an event queue like kafka (http://kafka.apache.org/)? Is there a socket from where I can directly listen to, and read the events using an external program?
Apache Kafka
kafka.apache.org
Apache Kafka: A Distributed Streaming Platform. ... Kafkaâ is used for building real-time data pipelines and streaming apps.



Just exploring!

Regards,
-Samik
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/