topic/jsiwek/tcp-improvements

Description

This branch is in the bro, bro-testing, and bro-testing-private repos and has a few changes to improve reporting of TCP connection sizes and gaps (commit messages explain in more detail).

The baseline changes in the external repos all seemed reasonable/explainable (or actually fix a problem). There's too much changed to go through case-by-case and actually check things, but I did do closer examinations of unique differences as I came across them (e.g. try to corroborate Bro results via wireshark). Then for those that seem to follow the same trend as something I already inspected, I wouldn't manually check.

Environment

None

Activity

Show:
Robin Sommer
January 28, 2014, 11:34 PM

I'm going ahead merging this but I'm wondering about the new detect_filtered_trace flag. It's pretty common (in the research world, anyways to run Bro on a SYN/FIN/RST trace and I imagine having this by default off can add a lot for warnings in that case. Can we add some other heuristic to detect such a trace (i.e., guess whether detect_filtered_trace should be on) ? A (very) coarse approach would simply be a global variable recording if we've ever seen anything else than a TCP control packet. Thoughts?

Jon Siwek
January 29, 2014, 4:14 PM

I'm going ahead merging this but I'm wondering about the new detect_filtered_trace flag. It's pretty common (in the research world, anyways to run Bro on a SYN/FIN/RST trace and I imagine having this by default off can add a lot for warnings in that case. Can we add some other heuristic to detect such a trace (i.e., guess whether detect_filtered_trace should be on) ? A (very) coarse approach would simply be a global variable recording if we've ever seen anything else than a TCP control packet. Thoughts?

If a person found out that Bro automatically switched modes part way through the trace, they will probably just re-run after manually toggling the option, right? Maybe treat it in a similar way to checksums – have a FAQ and/or have some script warn if all TCP connections are missing 100% of content and suggest toggling detect_filtered_trace if the person would like to trade off correctness for minimized output. But if it's actually not that important for a person using filtered traces to minimize output, I think it's fine enough as is?

Robin Sommer
January 29, 2014, 4:24 PM

have some script warn if all TCP connections are missing 100% of content and suggest toggling detect_filtered_trace

I like that, is that something we can do efficiently?

But if it's actually not that important for a person using filtered traces to minimize output, I think it's fine enough as is?

it's less the volume of output but the potential for confusion: one sees it and starts wondering what's wrong. It's easy to forget that TCP analysis gets confused because the trace is filtered. So if there was some way to point that out, that's all it would need.

It's not a biggie but it's indeed in the same category like the checksums: something easy to get wrong without realizing what's going on, in particular because we're changing the default here.

Seth Hall
January 29, 2014, 5:01 PM

We could probably do it similarly to how we're doing the detection of invalid checksums by sampling weirds for a little bit. I also like this approach a lot. I think that keeping the default settings of Bro working "correctly" in the normal case is good, but it's awesome to be able to notify people when things are failing and how they could fix it.

Jon Siwek
January 29, 2014, 5:46 PM

it's less the volume of output but the potential for confusion: one sees it and starts wondering what's wrong. It's easy to forget that TCP analysis gets confused because the trace is filtered.

I might be misremembering (or repressed the details of the TCP code), but isn't the TCP analysis less confused in the face of filtered traces with the change? i.e. things are now most correct and it actually reports content gaps so e.g. missing_bytes fields for connections can be populated.

but it's awesome to be able to notify people when things are failing and how they could fix it.

I wouldn't say filtered traces fail due to the change, you just get more, possibly unexpected but not incorrect, output.

(I'm just trying to clarify perspective, not really against idea of sampling weirds to issue suggestion/warning)

Jon Siwek
January 31, 2014, 11:11 PM

Added a new commit on the branch to add a script which auto-detects/warns about running on filtered trace.

Merged

Assignee

Unassigned

Reporter

Jon Siwek

Labels

None

External issue ID

None

Components

Fix versions

Affects versions

Priority

Normal
Configure