TimeMachine TODO Meta Ticket

Description

Then there are also some smaller quick-fix tasks, that would be really
helpful to have if a rework of the code is not possible.

A) The smaller items:
=====================

  • IPv6 support. Currently IPv4 is hard-coded in the TM. However, adding

    support for IPv6 should not be too hard.

  • TM restart. This is probably the most pressing issue!

    Currently when the TM restarts (or crashes) it cannot use

    the data it still has on its disk. It would be great if the restart

    could take this data into account. There are several options to do

    this:

+ Re-read the stored files (they are in pcap format) and/or the index

files and rebuild the full internal state and continue after the
last file. This enables queries for stored data. However, when
large disk-storage is used re-reading the files might well take
several hours

+ Do not re-read the stored files, but learn about them and include

them in buffer management. I.e., the TM starts building its internal
state only from data that newly arrives, but it knows that there are
old files lying around and it will delete the old files in order to
stay within the buffer budget. Then the old files can be searched
manually with tcpdump/tcpslice/whatever and restart is pretty much
instantaneous.

  • TM-cluster mode. This should be fairly easy. We would need a TM

    cluster front-end. Bro would then communicate with the front-end. The

    front-end sends the request to its workers (maybe with some

    intelligence to only query the workers that see the traffic according

    to the load balancing scheme) and gathers the results from the works,

    sorts them (by time) and delivers them to the requesting Bro.

  • Use a directory/inventory for disk searches. Currently disk queries

    are done using pcapnav to try to find the "right" location in a

    file (probabilistically jump to an offset and try to see if the

    offset is a valid start of a pcap-record).

    It would be good if the TM could store a directory for each pcap file

    it writes. The directory could then contain the file offset of

    each n-th packet + timestamp. A query can then just check the

    directory for the best location to jump to. No probabilist search.

    (Maybe this should be part of (B) though)

B) TM code rework
=================

In general some of the biggest problems of the TM are IMHO:

  • poor write performance. The memory buffer is not really used as

    elastic storage. The disk can block the capturing thread and thus

    lead to packet loss.

  • inflexible, hard-coded indexes. Slow lookup performance for on-disk

    queries.

    All possible key combinations (e.g.,

    2-tuple, 5-tuple, etc.) have to be specified at compile-time. It would

    be great if the TM could support queries for any combination of

    IP,IP,port,port,transport.

    Using fastbit and an indirection could help here. I have some

    early ideas on how this could be done.

  • Keep flow records in addition to packet data and keep it longer.

    The TM pretty implicitly keeps "flow" data for the connections it

    has in its storage. We could extend this to actually write the flow

    records to disk and assign a separate disk budget for such flow

    records. This would allow us to store flow records for significantly

    longer than just packet data. So it would increase the amount of time

    we can "travel back", but with less information.

  • IPv6 deluxe, Tunnel, non-IP support. In addition to basic IPv6

    support, it would be nice if the TM could handle / de-capsulate

    tunnels (e.g., 6in4, L2TP) and handle non-IP protocols (e.g., ARP,

    ESP, AH, etc.)

I'm happy to elaborate or discuss these topics some more if there's
interest.

cu
gregor

Environment

None

Assignee

Unassigned

Reporter

gregor

Labels

None

Priority

High
Configure