While tracking down a possibly memory leak in master, I dumped a core file from a worker and ran strings against it. The top strings were:
and that's with a bunch that show up as iSMB:RINT_READ,NSMB::FILE_READ, etc, I didn't bother to normalize those.
Probably unrelated to the memory usage I was seeing, but this is very odd because enum values like SMB::UNKNOWN_CLOSE and HTTP::COOKIE_SQLI are literally never used. The are defined, but otherwise never referenced.
It's likely that this is in previously allocated, but not currently referenced memory, but even if that is the case, the fact that bro even did anything at all with UNKNOWN_CLOSE 5 million times is kinda strange. That's an order of magnitude more than the number of smb related log lines we've had.
If the strings ARE in allocated memory, then 5752357 strings of SMB::UNKNOWN_WRITE alone account for almost 100MB of ram. all the SMB strings would total 1.5G of ram.
It seems to be related to how enums are serialized - that's the only thing that would touch unreferenced values.
I think I remember noticing this years ago, but never dug too deep into things.
So far all I've been able to determine is that EnumType:oSerialize gets called more than it should.
has bursts of:
And this is DoSerialize:
so, something is serializing the entire enum type... often.
I tracked that down to a
call I am doing in one of our local scripts that looks up the hostnames of external dns servers.
I fixed that similar to how the MHR script was fixed in d3f88ba9d19814076b02bde7ea81e70f4ef4f7c9. I'm not sure why a when statement triggered from a dns_request would be cloning SMB::Action, but it was...
This occurs elsewhere as well:
Why would a when statement triggered by an ssh connection clone SMB::Action or HTTP::Tags? I'd expect that c$http and c$smb would be unset..
It is odd that it does appear to be only the tunnel, smb, and http types that show up.
Bro currently only uses when in a few places, but as I understand it, many of the broker features are async, so it's possibly that an increase in when usage could have negative affects. I haven't been able to write a standalone script that shows increased memory use. when + big records did use more ram on OS X, but not when I ran it on linux.
Unrelated to the memory issue I also noticed one weird thing. Val.cc has this code in it:
The first thing I noticed is that rt->Lookup(rv) performs a linear search across the enum mapping. It looks like bro does this linear search for every single enum it receives. I don't see any reason EnumType couldn't maintain a vector of names along side the map of string -> int that it currently uses. Some enums are small, but the notice type enum has 60+ values in it.
What I don't get about that block is that it is all ran locally, so how could it detect a mismatch? It takes the id, looks up the name, then takes the name and looks it back up to get the id, and then compares them. But as far as I can tell all of this is all ran against the same lookup table. What am I missing?
'this' is 0x7f53a4490b00 for both lookups, and in the map: