Kafka Logger - Writes Bro Logs to Kafka

Description

As part of the Apache Metron project, we needed a way to send Bro logs to Kafka. From my research it seems like this is a common request. I'd rather give this code back to the Bro community than maintain it as part of Apache Metron.

This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as simple as adding the following Bro script.

{{
@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "bro";
redef Kafka::kafka_conf = table(
["metadata.broker.list"] = "localhost:9092"
);
}}

This plugin has the following features.

  • The user can specify a subset of all logs that should be sent to kafka. For example, to only send conn, http, and dns logs, specify the following.

{{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
}}

  • Full configurability of Kafka connectivity. Any configuration setting accepted by the librdkafka library can be passed to the plugin to tune how the logs are sent to Kafka.

{{redef Kafka::kafka_conf = table(
["metadata.broker.list"] = "localhost:9092",
["client.id"] = "bro"
);
}}

  • The plugin will wait a configurable period of time (for example, 3 seconds) after shutdown to attempt to send any queued messages to Kafka.

{{redef Kafka::max_wait_on_shutdown = 3000;
}}

  • There are two message formats to choose from. By default, the standard Bro JSON format is used. There is an alternative 'tagged JSON' format that is provided by the plugin. Currently, all messages are sent to a single Bro topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log stream the message originated from. This format prepends the log stream identifier to the JSON message.

{{{'conn': { ... }}
{'http': { ... }}
{'dns': { ... }}}}

To enable this alternative format, simply specify the following.

redef Kafka::tag_json = T;

Environment

None

Assignee

Robin Sommer

Reporter

Nick Allen

Labels

None

External issue ID

None

Components

Priority

Normal
Configure