Then there are also some smaller quick-fix tasks, that would be really
helpful to have if a rework of the code is not possible.
A) The smaller items:
IPv6 support. Currently IPv4 is hard-coded in the TM. However, adding
support for IPv6 should not be too hard.
TM restart. This is probably the most pressing issue!
Currently when the TM restarts (or crashes) it cannot use
the data it still has on its disk. It would be great if the restart
could take this data into account. There are several options to do
+ Re-read the stored files (they are in pcap format) and/or the index
files and rebuild the full internal state and continue after the
last file. This enables queries for stored data. However, when
large disk-storage is used re-reading the files might well take
+ Do not re-read the stored files, but learn about them and include
them in buffer management. I.e., the TM starts building its internal
state only from data that newly arrives, but it knows that there are
old files lying around and it will delete the old files in order to
stay within the buffer budget. Then the old files can be searched
manually with tcpdump/tcpslice/whatever and restart is pretty much
TM-cluster mode. This should be fairly easy. We would need a TM
cluster front-end. Bro would then communicate with the front-end. The
front-end sends the request to its workers (maybe with some
intelligence to only query the workers that see the traffic according
to the load balancing scheme) and gathers the results from the works,
sorts them (by time) and delivers them to the requesting Bro.
Use a directory/inventory for disk searches. Currently disk queries
are done using pcapnav to try to find the "right" location in a
file (probabilistically jump to an offset and try to see if the
offset is a valid start of a pcap-record).
It would be good if the TM could store a directory for each pcap file
it writes. The directory could then contain the file offset of
each n-th packet + timestamp. A query can then just check the
directory for the best location to jump to. No probabilist search.
(Maybe this should be part of (B) though)
B) TM code rework
In general some of the biggest problems of the TM are IMHO:
poor write performance. The memory buffer is not really used as
elastic storage. The disk can block the capturing thread and thus
lead to packet loss.
inflexible, hard-coded indexes. Slow lookup performance for on-disk
All possible key combinations (e.g.,
2-tuple, 5-tuple, etc.) have to be specified at compile-time. It would
be great if the TM could support queries for any combination of
Using fastbit and an indirection could help here. I have some
early ideas on how this could be done.
Keep flow records in addition to packet data and keep it longer.
The TM pretty implicitly keeps "flow" data for the connections it
has in its storage. We could extend this to actually write the flow
records to disk and assign a separate disk budget for such flow
records. This would allow us to store flow records for significantly
longer than just packet data. So it would increase the amount of time
we can "travel back", but with less information.
IPv6 deluxe, Tunnel, non-IP support. In addition to basic IPv6
support, it would be nice if the TM could handle / de-capsulate
tunnels (e.g., 6in4, L2TP) and handle non-IP protocols (e.g., ARP,
ESP, AH, etc.)
I'm happy to elaborate or discuss these topics some more if there's