Memory leak in Bro Intel framework

Description

The policy/frameworks/intel/seen bro scripts have a memory leak. On my moderately busy Bro installation I am leaking about a gig of memory a day per worker process with the Intel framework enabled. I can replicate by adding the following to the local.bro default script and then running through a small PCAP with primarily dns, dhcp and syslog traffic.

{{
@load policy/frameworks/intel/seen

redef Intel::read_files += {
"/usr/local/bro/spool/domain_suspicious.txt",
};
}}

The intel file is in the following format, here's a few sample lines. It is generated automatically by CIF:

{{
#fields indicator indicator_type meta.source meta.desc meta.url meta.cif_impact meta.cif_severity meta.cif_confidence
mete-tools.biz Intel:OMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=mete-tools.biz (public) - - 95
rttvxygkmwlqmq.net Intel:OMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=rttvxygkmwlqmq.net (public) - - 95
podserveruho.com Intel:OMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=podserveruho.com (public) - - 95
wwfcogdgntlxw.biz Intel:OMAIN CIF - need-to-know spammed domain http://www.spamhaus.org/query/dbl?domain=wwfcogdgntlxw.biz (public) - - 95
}}

I compiled bro with gperftool debug support and followed the instructions here: http://www.bro.org/development/howtos/leaks.html. (Note, the instructions are wrong on the flags for ./configure, you need to add --enable-perftools-debug to get the -m option for bro)

Here's the output from pprof top after running a PCAP trace with 10,000 packets. Running traces with more packets show a greater number of lost objects in the same code locations.

{{

  1. pprof bin/bro "/tmp/bro.24541.net_run-end.heap" --inuse_objects --lines --heapcheck --edgefraction=1e-10 --nodefraction=1e-10
    Using local file bin/bro.
    Using local file /tmp/bro.24541.net_run-end.heap.
    Welcome to pprof! For help, type 'help'.
    (pprof) top
    Total: 4295 objects
    2150 50.1% 50.1% 2150 50.1% AsciiFormatter:arseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:186
    2141 49.8% 99.9% 2141 49.8% copy_string /usr/src/bro-2.2/src/util.cc:155
    2 0.0% 100.0% 2 0.0% re_alloc /usr/src/bro-2.2/build/src/re-scan.cc:2287
    1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:110
    1 0.0% 100.0% 1 0.0% RE_parse /usr/src/bro-2.2/build/src/re-parse.y:133
    0 0.0% 100.0% 2141 49.8% AsciiFormatter:arseValue /usr/src/bro-2.2/src/threading/AsciiFormatter.cc:195
    0 0.0% 100.0% 4 0.1% Connection::NextPacket /usr/src/bro-2.2/src/Conn.cc:259
    0 0.0% 100.0% 4 0.1% NetSessions:ispatchPacket /usr/src/bro-2.2/src/Sessions.cc:189
    0 0.0% 100.0% 4 0.1% NetSessions:oNextPacket /usr/src/bro-2.2/src/Sessions.cc:709
    0 0.0% 100.0% 4 0.1% NetSessions::NextPacket /usr/src/bro-2.2/src/Sessions.cc:247
    }}

Environment

Red Hat Enterprise Linux Server release 6.5

Activity

Show:
Andrew Hoying
December 5, 2013, 3:16 PM

I ran some more checks and the memory leak is only based on the size and frequency of the files loaded through Intel::read_files, the size and content of the trace files tested doesn't seem to make much difference in the final number of lost objects.

Seth Hall
December 6, 2013, 12:31 AM

I have confirmed this leak, it seems to be an issue with the input framework. We'll be fixing it ASAP. Thanks for tracking it down.

amannb
December 6, 2013, 2:01 AM

Seth and me tracked down the root of the leak - the input manager uses the length of the root-record while deleting values instead of the length of the unrolled record.

I will fix this in a few hours (well, after re-checking it really fixes it), thank you very much for the very thorough bug-report.

amannb
December 6, 2013, 6:00 AM

verified patch is in topic/bernhard/ticket1103

Andrew Hoying
December 6, 2013, 6:37 AM

I applied the patch against 2.2 and verified that it fixed the memory leak on my system. Thank you!

Merged

Assignee

Johanna Amann

Reporter

Andrew Hoying

Labels

External issue ID

None

Components

Affects versions

Priority

High