Bro 2.3 - 2.3.1 manager dieing on Bivio hardware

Description

After starting bro up, the bro manager crashes in less than 60 seconds.
Thanks for any help you can give.

Sent stack trace to vendor (at bottom), and here was their response:

Comment(s): Hello Larry,

We have duplicated a crash in our lab setup that seems to be identical to that experienced by you. The code has changed quite a bit from 2.1 to 2.3.1, and we suspect a bug was introduced.

What is going on, seems to be that a writer thread is being terminated, and the destructor for the Ascii writer is called eventually. However, the destructor code does some checks and finds out that proper cleanup has not been done, so it aborts. This does not seem to be due to any library incompatibility, and looks more like maybe a race condition was introduced.

Since you knows the Bro developers, can you please ask them to take a look this and get back to us? We think it requires their expertise at this point.

Thank You,

Hassan.

Bivio Case Information:
Bivio Case #: 4566243
Date Created: 9/02/2014 08:02 AM PDT

Stack trace below:
GNU gdb (GDB) Fedora (6.8.50.20090302-40.fc11) Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "ppc-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
backtrace
[New Thread 25501]
[New Thread 25328]
[New Thread 25378]
[New Thread 25379]
[New Thread 25380]
[New Thread 25381]
[New Thread 25382]
[New Thread 25383]
[New Thread 25384]
[New Thread 25385]
[New Thread 25386]
[New Thread 25389]
[New Thread 25442]

warning: Can't read pathname for load map: Input/output error.
Missing separate debuginfo for /usr/local/lib/libz.so.1
Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a2/0a0d1fc0d48c2a303af1417ccc03308b9de04a
Missing separate debuginfo for /usr/local/lib/libtcmalloc.so.4
Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/27/eaf56bc64810920d55b9530156c1e8ffbfd43e
Missing separate debuginfo for /usr/local/lib/libcurl.so.4
Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a7/9a2cebb4abc156495ec0806b1c18015c8eba01
Reading symbols from /usr/lib/libpcap.so.1...done.
Loaded symbols for /usr/lib/libpcap.so.1 Reading symbols from /usr/lib/libssl.so.10...done.
Loaded symbols for /usr/lib/libssl.so.10 Reading symbols from /usr/lib/libcrypto.so.10...done.
Loaded symbols for /usr/lib/libcrypto.so.10 Reading symbols from /usr/lib/libbind.so.4...done.
Loaded symbols for /usr/lib/libbind.so.4 Reading symbols from /usr/local/lib/libz.so.1...done.
Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from /usr/local/lib/libtcmalloc.so.4...done.
Loaded symbols for /usr/local/lib/libtcmalloc.so.4 Reading symbols from /usr/local/lib/libcurl.so.4...done.
Loaded symbols for /usr/local/lib/libcurl.so.4 Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/lib/libzcp.so...done.
Loaded symbols for /usr/lib/libzcp.so
Reading symbols from /lib/libgssapi_krb5.so.2...done.
Loaded symbols for /lib/libgssapi_krb5.so.2 Reading symbols from /lib/libkrb5.so.3...done.
Loaded symbols for /lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /lib/libk5crypto.so.3...done.
Loaded symbols for /lib/libk5crypto.so.3 Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Reading symbols from /lib/libbvsp.so...done.
Loaded symbols for /lib/libbvsp.so
Reading symbols from /lib/libbcon.so...done.
Loaded symbols for /lib/libbcon.so
Reading symbols from /lib/libkrb5support.so.0...done.
Loaded symbols for /lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...done.
Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2 Reading symbols from /lib/libhmlibs.so...done.
Loaded symbols for /lib/libhmlibs.so
Reading symbols from /lib/libhmolddb.so...done.
Loaded symbols for /lib/libhmolddb.so
Reading symbols from /lib/libcf.so...done.
Loaded symbols for /lib/libcf.so
Reading symbols from /lib/libbvsep.so...done.
Loaded symbols for /lib/libbvsep.so
Reading symbols from /usr/lib/libnrddi.so...done.
Loaded symbols for /usr/lib/libnrddi.so
Reading symbols from /lib/libselinux.so.1...done.
Loaded symbols for /lib/libselinux.so.1
Core was generated by `/var/tmp/bro/spool/tmp/bro -U .status -p broctl -p broctl-live -p local -p mana'.
Program terminated with signal 6, Aborted.
#0 0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
in ../nptl/sysdeps/unix/sysv/linux/raise.c
Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.9-2.fc11.ppc glibc-2.17-4.fc11.ppc keyutils-libs-1.2-5.fc11.ppc krb5-libs-1.9.3-1.fc11.ppc libbind-6.0-1.fc11.ppc libgcc-4.4.1-2.fc11.ppc libselinux-2.0.80-1.fc11.ppc libstdc++-4.4.1-2.fc11.ppc libxml2-2.7.6-1.fc11.ppc openssl-libs-1.0.1e-37.fc11.1.ppc
(gdb) backtrace
#0 0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x0f6d0de0 in *__GI_abort () at abort.c:90
#2 0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
#3 0x10236b70 in threading::Manager:rocess (this=0x10dae180)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
#4 0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
#5 0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
Current language: auto; currently minimal
(gdb)
#0 0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x0f6d0de0 in *__GI_abort () at abort.c:90
#2 0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
#3 0x10236b70 in threading::Manager:rocess (this=0x10dae180)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
#4 0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
#5 0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
(gdb) quit

Environment

Bro 2.3 and Bro 2.3.1
bivio hardwareLinux CPU.2.6.31-45 has curl 7.36 gperftools 2.2 flex 2.5.39 bison 3.0.2 libpcap 1.1 swig 2.0.8

Activity

Show:
Larry Leviton
September 17, 2014, 3:46 PM

Seems to run okay with debug mode on. No crashes.

The machine I put it on originally had 2.3, and I had DNS logging turned
off (due to stack trace I had sent in first e-mail). After that, I got
the manager crashes on 2.3 without debug, but the box itself was running
argus at the same time, so I put it on a second standalone machine only
running bro figuring that running both could cause some sort of unique
problem. So today, I put on 2.3.1 on the machine with argus and recompiled
with debug, and everything is running without crashes.

I am going to compile it with debug on the second machine which is only
running bro. Hopefully I will get back to you before I leave today.

Johanna Amann
October 21, 2014, 3:36 PM

I just wanted to check if you managed to get more information on this?

Jon Siwek
March 16, 2015, 6:18 PM

Maybe related to BIT-1331.

Johanna Amann
March 17, 2015, 8:56 PM

I will just close this because we have not gotten any more feedback / information on it and it is currently not actionable.

If you ever have more information on this, please feel free to re-open the ticket.

Larry Leviton
April 29, 2015, 2:21 PM

I apologize for not getting back to you sooner, but due to some contractual
issues and other problems, I wasn't able to get back to this till now.
The probably appears fixed in 2.3.2. Thanks so much for fixing it.

On Tue, Mar 17, 2015 at 4:40 PM, Jon Siwek (JIRA) <

Assignee

Johanna Amann

Reporter

Larry Leviton

Labels

None

External issue ID

None

Components

Fix versions

Affects versions

Priority

Normal
Configure