Input::add_table is not properly reading in sets

Description

I’m reading a table into a script. The table includes two sets in the values fields. When executing the script, I’m getting the error message ”Did not find requested field service in input data file model2.log”

Following the example in bro/testing/btest/scripts/base/frameworks/input/setseparator.bro, I’ve redefined the set separator as ‘|’ (redef InputAscii::set_separator = "|".

The

The table key consists of two addresses, node_A and node_B.
My value inputs consist of two sets, which can consist of just a single value; all fields are separated by tabs. The first two lines of my input file are:
#fields node_A node_B layer_3_4 service
xxx.yyy.zzz.30 xxx.yyy.255.255 udp dns

xxx.yyy.zzz are valid IP address values.
It appears that the strings ‘udp’ and ‘dns’ are both being read as part of the layer_3_4 set. Since they are separated by a tab instead of ‘|’, they should be interpreted as separate fields.

Environment

ArchLinux on VMware

Activity

Show:
earl eiland
August 13, 2015, 10:58 AM

I'm running version 2.4-20, Johanna.

Earl

Daniel Thayer
August 13, 2015, 2:53 PM

I can reproduce those errors. The problem is that the file "model2.log.txt"
contains newline characters that do not match those used in Linux (perhaps
you created the file on another OS, and then copied it over to your Linux
machine?). If you look at the file using the "vi" editor in Linux, you
can see "^M" characters at the end of each line. If you remove those, then
the errors disappear.

earl eiland
August 13, 2015, 3:02 PM

Thanks, Daniel.

I'm using python's csv writer – in Linux, but clearly, it's adding a newline. Thanks for the analysis! When I've figured out the correct csv.writer() parameter, I'll share it with the community.

Earl

Johanna Amann
August 13, 2015, 3:53 PM

Great, thank you – I just copied it over and forgot to look at the newlines in the original file.

earl eiland
August 13, 2015, 3:54 PM

If you're writing bro input files with python and using csv, then csv.writer must have the correct parameters. For example,
write_model = csv.writer(model_file, delimiter='\x09', lineterminator = '\n')

These settings place tabs between the columns and terminate lines with the newline character.

Earl

Solved

Assignee

Johanna Amann

Reporter

earl eiland

Labels

External issue ID

None

Components

Affects versions

Priority

Normal
Configure