Enable bro-cut to read JSON logs

Description

Here is a proposal for adding the ability for bro-cut to read JSON logs.

The first line of the log determines the log type (JSON or regular Bro Ascii),
so the user never needs to specify the log type. The output format remains
the same regardless of the log type (a couple exceptions are listed below).

JSON logs have two differences from the normal Bro ascii logs
that cause problems for bro-cut:
1) There are no header metadata lines in Bro JSON logs.
2) Any log fields with the &optional attribute will not appear at all (not
even the field name) in a JSON log record when that field doesn't have
a value. This means that the number of fields per line in a JSON log
can vary from one line to the next.

Therefore, when processing a JSON log, the behavior of bro-cut changes in the
following ways:

1) If the user doesn't supply any field names on the bro-cut cmdline,
then bro-cut will assume that the field names on the first line are all the
ones available. If any additional field names are encountered, then
they are added to the list of known field names. This means the number
of columns in the output can increase when no field names are given on the
command line.

2) Likewise, if the user specifies the "-n" option, then bro-cut assumes
that the field names on the first line are all the ones available.
Any new field names encountered will be added to the list. This means the
number of columns in the output can increase when the "-n" option is used.

3) If the bro-cut "-c" or "-C" options are specified, then bro-cut exits
immediately with an error message, because we cannot reliably create
a format header block since we don't even know all the field names until we
read the entire log.

Environment

None

Assignee

Unassigned

Reporter

Daniel Thayer

Labels

None

External issue ID

None

Components

Priority

Normal
Configure