We currently use Logstash to tail the global event log and send it directly to Elasticsearch . Logstash can easily send events to Kafka instead, or just about anywhere else (e.g. we're putting RabbitMQ into our pipeline). We've found having the event log in Elasticseach has been hugely useful for system analytics and troubleshooting.
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Samik Raychaudhuri <samikr@xxxxxxxxx>
Sent: Saturday, July 1, 2017 9:39 AM
To: HTCondor-Users Mail List
Subject: [HTCondor-users] Event log to kafka-like system
Wanted to process the event logs from a submitted job as they come in. Is there a way to get the logs in an event queue like kafka (http://kafka.apache.org/)? Is there a socket from where I can directly listen to, and read the events using an external program?