Bro 2.3 - 2.3.1 manager dieing on Bivio hardware

Description

After starting bro up, the bro manager crashes in less than 60 seconds.
Thanks for any help you can give.

Sent stack trace to vendor (at bottom), and here was their response:

Comment(s): Hello Larry,

We have duplicated a crash in our lab setup that seems to be identical to that experienced by you. The code has changed quite a bit from 2.1 to 2.3.1, and we suspect a bug was introduced.

What is going on, seems to be that a writer thread is being terminated, and the destructor for the Ascii writer is called eventually. However, the destructor code does some checks and finds out that proper cleanup has not been done, so it aborts. This does not seem to be due to any library incompatibility, and looks more like maybe a race condition was introduced.

Since you knows the Bro developers, can you please ask them to take a look this and get back to us? We think it requires their expertise at this point.

Thank You,

Hassan.

Bivio Case Information:
Bivio Case #: 4566243
Date Created: 9/02/2014 08:02 AM PDT

Stack trace below:
GNU gdb (GDB) Fedora (6.8.50.20090302-40.fc11) Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "ppc-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
backtrace
[New Thread 25501]
[New Thread 25328]
[New Thread 25378]
[New Thread 25379]
[New Thread 25380]
[New Thread 25381]
[New Thread 25382]
[New Thread 25383]
[New Thread 25384]
[New Thread 25385]
[New Thread 25386]
[New Thread 25389]
[New Thread 25442]

warning: Can't read pathname for load map: Input/output error.
Missing separate debuginfo for /usr/local/lib/libz.so.1
Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a2/0a0d1fc0d48c2a303af1417ccc03308b9de04a
Missing separate debuginfo for /usr/local/lib/libtcmalloc.so.4
Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/27/eaf56bc64810920d55b9530156c1e8ffbfd43e
Missing separate debuginfo for /usr/local/lib/libcurl.so.4
Try: yum --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/a7/9a2cebb4abc156495ec0806b1c18015c8eba01
Reading symbols from /usr/lib/libpcap.so.1...done.
Loaded symbols for /usr/lib/libpcap.so.1 Reading symbols from /usr/lib/libssl.so.10...done.
Loaded symbols for /usr/lib/libssl.so.10 Reading symbols from /usr/lib/libcrypto.so.10...done.
Loaded symbols for /usr/lib/libcrypto.so.10 Reading symbols from /usr/lib/libbind.so.4...done.
Loaded symbols for /usr/lib/libbind.so.4 Reading symbols from /usr/local/lib/libz.so.1...done.
Loaded symbols for /usr/local/lib/libz.so.1 Reading symbols from /usr/local/lib/libtcmalloc.so.4...done.
Loaded symbols for /usr/local/lib/libtcmalloc.so.4 Reading symbols from /usr/local/lib/libcurl.so.4...done.
Loaded symbols for /usr/local/lib/libcurl.so.4 Reading symbols from /lib/libpthread.so.0...done.
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/lib/libzcp.so...done.
Loaded symbols for /usr/lib/libzcp.so
Reading symbols from /lib/libgssapi_krb5.so.2...done.
Loaded symbols for /lib/libgssapi_krb5.so.2 Reading symbols from /lib/libkrb5.so.3...done.
Loaded symbols for /lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /lib/libk5crypto.so.3...done.
Loaded symbols for /lib/libk5crypto.so.3 Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /lib/ld.so.1...done.
Loaded symbols for /lib/ld.so.1
Reading symbols from /lib/libbvsp.so...done.
Loaded symbols for /lib/libbvsp.so
Reading symbols from /lib/libbcon.so...done.
Loaded symbols for /lib/libbcon.so
Reading symbols from /lib/libkrb5support.so.0...done.
Loaded symbols for /lib/libkrb5support.so.0 Reading symbols from /lib/libkeyutils.so.1...done.
Loaded symbols for /lib/libkeyutils.so.1 Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2 Reading symbols from /lib/libhmlibs.so...done.
Loaded symbols for /lib/libhmlibs.so
Reading symbols from /lib/libhmolddb.so...done.
Loaded symbols for /lib/libhmolddb.so
Reading symbols from /lib/libcf.so...done.
Loaded symbols for /lib/libcf.so
Reading symbols from /lib/libbvsep.so...done.
Loaded symbols for /lib/libbvsep.so
Reading symbols from /usr/lib/libnrddi.so...done.
Loaded symbols for /usr/lib/libnrddi.so
Reading symbols from /lib/libselinux.so.1...done.
Loaded symbols for /lib/libselinux.so.1
Core was generated by `/var/tmp/bro/spool/tmp/bro -U .status -p broctl -p broctl-live -p local -p mana'.
Program terminated with signal 6, Aborted.
#0 0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
in ../nptl/sysdeps/unix/sysv/linux/raise.c
Missing separate debuginfos, use: debuginfo-install e2fsprogs-libs-1.41.9-2.fc11.ppc glibc-2.17-4.fc11.ppc keyutils-libs-1.2-5.fc11.ppc krb5-libs-1.9.3-1.fc11.ppc libbind-6.0-1.fc11.ppc libgcc-4.4.1-2.fc11.ppc libselinux-2.0.80-1.fc11.ppc libstdc++-4.4.1-2.fc11.ppc libxml2-2.7.6-1.fc11.ppc openssl-libs-1.0.1e-37.fc11.1.ppc
(gdb) backtrace
#0 0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x0f6d0de0 in *__GI_abort () at abort.c:90
#2 0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
#3 0x10236b70 in threading::Manager:rocess (this=0x10dae180)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
#4 0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
#5 0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
Current language: auto; currently minimal
(gdb)
#0 0x0f6cf01c in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x0f6d0de0 in *__GI_abort () at abort.c:90
#2 0x1024be70 in logging::writer::Ascii::~Ascii (this=0x11a87200, __in_chrg=<value optimized out>)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/logging/writers/Ascii.cc:186
#3 0x10236b70 in threading::Manager:rocess (this=0x10dae180)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/threading/Manager.cc:171
#4 0x101a5400 in net_run () at /bivio/scsi/b/levitonl/bro-2.3.1/src/Net.cc:389
#5 0x100f7554 in main (argc=<value optimized out>, argv=<value optimized out>)
at /bivio/scsi/b/levitonl/bro-2.3.1/src/main.cc:1165
(gdb) quit

Environment

Bro 2.3 and Bro 2.3.1
bivio hardwareLinux CPU.2.6.31-45 has curl 7.36 gperftools 2.2 flex 2.5.39 bison 3.0.2 libpcap 1.1 swig 2.0.8

Assignee

Johanna Amann

Reporter

Larry Leviton

Labels

None

External issue ID

None

Components

Fix versions

Affects versions

Priority

Normal
Configure