The plugin build system makes packaging bro difficult

Description

Hey!

Let's make the bro plugin building system packaging friendly.

The current situation is that if you want to build a package, it needs a ./configure-ed bro source with certain headers built, for the plugin itself to build.

This breaks some package building system, that assume 1 source = 1 package with no external dependencies. Furthermore, some of them actually enforce that and disable networking so nothing extra can be downloaded. They also generate RPM spec files, so doing tricks with unpacking additional Bro source and temporary-building it is impossible.

I spent way too many hours and finally figured out what are the requirements for the plugin to build, i.e. what is the minimum viable Bro source they actually need. Here are my observations:

cd ./bro-2.5.9271b2032 && ./configure && cd build/src && make bifcl

./bifcl /home/clear/rpmbuild/BUILD/bro-plugin-afpacket-1.3.79edee2/bro-2.5.9271b2032/src/const.bif
./bifcl /home/clear/rpmbuild/BUILD/bro-plugin-afpacket-1.3.79edee2/bro-2.5.9271b2032/src/types.bif
./bifcl /home/clear/rpmbuild/BUILD/bro-plugin-afpacket-1.3.79edee2/bro-2.5.9271b2032/src/event.bif
./bifcl /home/clear/rpmbuild/BUILD/bro-plugin-afpacket-1.3.79edee2/bro-2.5.9271b2032/src/reporter.bif
cd ../../..
./configure --bro-dist=./bro-2.5.9271b2032 --install-root=${_prefix}/usr/lib/bro/plugins --with-kernel=/tmp/linux-4.14.21 && make %{?_smp_mflags}

So it looks like the ./configure process creates something (I guess, make files or cmake files) that allow the bifcl binary to build, and the bifcl binary can be used to build those 'bif' files plugins require.

What needs to happen so that is easier? If building a plugin would not require a full Bro source to be included, that would be ideal.

BTW, this is also a requirement for the BroIDS to be (ever) included in the Clear Linux. We have already included Suricata, as it does not have similar requirements.

I don't know anything about cmake - but people in the know told me that our configure + cmake process is awkward i.e. you either use autoconf or cmake, both aren't necessary. Just passing along.

Environment

None

Activity

Show:
Michał Purzyński
April 11, 2018, 10:26 AM

Indeed, having a clear 'dev' packages that would be required by plugins to build is the best thing we would do. Same with the bifcl - it should be easy to have that packaged so it can be installed on the builder system.

By no-dependencies environment, I mean a build system that deletes and re-creates everything between builds, so when trying to build (say) plugins there is no way to depend on the bro source having been built before. Those would be built in two separate chroot() environments, removed in between.

In this case it is also not possible to fetch one giant source of bro + bro-plugins, dump everything into a directory and produce N separate packages, since this kind of build is not supported by design. That is what Johanna did with RPMs and it works really well for CentOS, Suse, etc.

It would also needs entire Bro to be rebuild to add a single plugin, which is suboptimal. A clear set of development headers + bifcl would solve that.

Bro itself is not eligible to be included, because it breaks the following design principles:

1. packages need to build cleanly and install itself into a temporary directory (for packaging) without any external dependencies. Bro itself could do it, but only if stripped of plugins.

2. Bro depends on an earlier version of CAF. That's a separate problem, though.

3. That version of CAF cannot be included upstream, because it's not the current one and Clear Linux does not support earlier (or multiple) package versions.

Jon Siwek
April 12, 2018, 1:39 AM

A clear set of development headers + bifcl would solve that.

Cool, thanks for confirming that we think that's at least the start of what should be done.

Bro itself is not eligible to be included, because it breaks the following design principles:

1. packages need to build cleanly and install itself into a temporary directory (for packaging) without any external dependencies. Bro itself could do it, but only if stripped of plugins.

I don't get that last part: which plugins are you talking about that prevent Bro from building without external dependencies?

Components of Bro like the analyzers and log writers that currently ship w/ Bro are called "plugin" because they can theoretically be split out into a separate source tree, though I don't think there's any intention to do that globally for everything that's currently there. They are actually intended to be compiled in the same source tree as Bro and statically linked in.

2. Bro depends on an earlier version of CAF. That's a separate problem, though.

3. That version of CAF cannot be included upstream, because it's not the current one and Clear Linux does not support earlier (or multiple) package versions

For the current Bro release, the CAF dependency is optional/experimental (and I probably wouldn't suggest using it). For the next Bro release, CAF will be required and it will be the latest version of CAF.

Michał Purzyński
April 12, 2018, 7:32 AM

Ha, so we're getting somewhere, cool. Let me be more precise here.

There is almost nothing in the bro source package itself that makes building it all by itself impossible on CL, if you don't depend on the CAF and don't want to use external plugins (like the af_packet one).

Some unit tests fail because they can't find a way to link but that can be solved adding -lpthread here and there, I can share more details in people are interested in fixing (myself, I have no idea about CMake, so half-randomly patched everything that moves till it built).

Adding external plugins is impossible, because those depend on Bro sources half-built. Having bifcl and headers separated would solve that.

Didn't know that libcaf is not really used by anything (except for maybe the experimental broker implementation) so that's cool as it removes one dependency.

BTW, CAF could be included, as the fresh version builds cleanly and automatically.

So the plan of action should be, for Bro 2.5.x - separate bifcl and headers into packages, get those included, do the same with Bro itself (without libcaf) and we will include CAF when 2.6 is around.

Jon Siwek
April 12, 2018, 8:54 AM

The unit test failures due to missing -lpthread sound like maybe a legit problem that's independent of project/package/dependency structures, so yeah, that might be good to share more details on if you could.

Jon Siwek
July 25, 2018, 8:19 AM

The general issues covered in this ticket should be fixed per BIT-1955.

Fixed

Assignee

Unassigned

Reporter

Michał Purzyński

Labels

None

External issue ID

None

Components

Fix versions

Affects versions

Priority

High