[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] Event log to kafka-like system

Logstash works independently from Elasticsearch; it's really a "swiss-army-knife" for data, with the large number of plugins available for moving and manipulating data, which is why I tend to reach for it first. That said there's no shortage of log shipping tools, so yeah, pick what you're comfortable with and fits best into your pipeline.

We use both Grafana and Kibana; Kibana is great for ad-hoc analytics and visualizations, but Grafana is better for presenting pre-defined dashboards for our users, and lets us mix Elasticseach and Graphite data.


From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Samik Raychaudhuri <samikr@xxxxxxxxx>
Sent: Wednesday, July 5, 2017 1:50 AM
To: htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] Event log to kafka-like system
Hi Kevin,

Thanks for the input. Couple of questions/comments:
  • I am stumbling at the very beginning: I am not able to find the "global event log" file, i.e. I do not see /var/log/condor/EventLog. Is there a setting to enable this? I could not find something similar in: [http://research.cs.wisc.edu/htcondor/manual/v8.6/3_5Configuration_Macros.html]. I am using 8.6.4 - the most recent version on a Debian VM.

  • Given the "condor" patterns file, looks like you are using the text version of the log file, rather than the xml version.  I am assuming that those are the two options, right? (the secret question is, there is no JSON option, right? :-))
  • The way you are doing this indicates that, I can roll out my own kafka producer by tailing the files myself (e.g., by using Tailer from Apache Commons IO), if I do not want Elasticsearch (ES) in the loop (although I am pretty familiar with these systems, and completely see the value of ES/Graphana). Will probably try that out.
    • Curious from that perspective and slightly off-topic: since you already have the data in ES, why are you using graphana instead of kibana?


On 05-Jul-17 2:04 AM, Kevin Retzke wrote:

We currently use Logstash to tail the global event log and send it directly to Elasticsearch [1]. Logstash can easily send events to Kafka instead, or just about anywhere else (e.g. we're putting RabbitMQ into our pipeline). We've found having the event log in Elasticseach has been hugely useful for system analytics and troubleshooting.


From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Samik Raychaudhuri <samikr@xxxxxxxxx>
Sent: Saturday, July 1, 2017 9:39 AM
To: HTCondor-Users Mail List
Subject: [HTCondor-users] Event log to kafka-like system

Wanted to process the event logs from a submitted job as they come in. Is there a way to get the logs in an event queue like kafka (http://kafka.apache.org/)? Is there a socket from where I can directly listen to, and read the events using an external program?
Apache Kafka: A Distributed Streaming Platform. ... Kafka™ is used for building real-time data pipelines and streaming apps.

Just exploring!


HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting

The archives can be found at: