Got answer to mt first question here:
Will check this out.|
On 05-Jul-17 12:20 PM, Samik
Thanks for the input. Couple of questions/comments:
- I am stumbling at the very beginning: I am not able to find
the "global event log" file, i.e. I do not see
/var/log/condor/EventLog. Is there a setting to enable this? I
could not find something similar in:
I am using 8.6.4 - the most recent version on a Debian VM.
- Given the "condor" patterns file, looks
like you are using the text version of the log file, rather
than the xml version. I am assuming that those are the two
options, right? (the secret question is, there is no JSON
option, right? :-))
- The way you are doing this indicates
that, I can roll out my own kafka producer by tailing the
files myself (e.g., by using Tailer from Apache Commons IO), if I do not want
Elasticsearch (ES) in the loop (although I am pretty
familiar with these systems, and completely see the value of
ES/Graphana). Will probably try that out.
- Curious from that perspective and
slightly off-topic: since you already have the data in ES,
why are you using graphana instead of kibana?
On 05-Jul-17 2:04 AM, Kevin Retzke
We currently use Logstash to tail the global event log and
send it directly to Elasticsearch . Logstash can easily
send events to Kafka instead, or just about anywhere else
(e.g. we're putting RabbitMQ into our pipeline). We've found
having the event log in
Elasticseach has been hugely useful for system analytics
Wanted to process the event logs from a submitted job as
they come in. Is there a way to get the logs in an event
queue like kafka (http://kafka.apache.org/)?
Is there a socket from where I can directly listen to,
and read the events using an external program?
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
You can also unsubscribe by visiting
The archives can be found at: