topic/jsiwek/broxygen

Description

This branch is in the bro and broccoli repos and improves the automated script-reference documentation generation process, broxygen.

This should address issues in and BIT-751, let me know if there's anything not covered that's still relevant.

Highlights:

  • Remove --doc-scripts and -Z options to toggle documentation mode – the parser is now always instrumented to gather documentation from comments of the form "##", "##!", or "##<".

  • Raw comments are available at runtime through several BIF functions: get_*_comments;

  • Add --broxygen and -X options to toggle generating reST-format documentation output, driven by a config file argument.

  • Add a "broxygen" Sphinx extension domain, allowing certain pieces of documentation to be generated on-the-fly via invoking a Bro process. Re-organized/cleaned up the Sphinx source tree in doc/ to use this in some places.

Environment

None

Activity

Show:
Robin Sommer
December 5, 2013, 8:56 AM

Excellent, I like the new way of feeding the Broxygen manager.

I've merged it. Some additional suggestions:

  • what would you think about moving the global type alias table into BroType as a private static member, along with a few static accessor methods to manipulate it?

  • if there's a bro binary in PATH, it seems that "make doc" picks up that one instead of the one in src/build/bro

  • "make doc" shows received termination signal right before Running Sphinx. Is that expected?

  • reST Output:

    • I'd remove the "Param" in the function descriptions (or is that a Sphinx thing?)

    • If a package has scripts both w/ and wo/ a further description, they appear separately in the package overview; see, e.g., scripts/base/frameworks/packet-filter/index.html

    • should we just omit _load_.bro in the package overviews?

    • I like how the records now show fields that are redefined somewhere else. Two further thoughts:

      • how about changing from redef in XXX to added if XXX is loaded. For a user, it might be more intuitive to talk about loading another script than just redef.

      • for the script that does the redef, can we show the new fields there as well? For example, in scripts/policy/protocols/http/software-browser-plugins.bro.html it would show not only that HTTP::Info got redef, but also repeat the fields it added right there. Same for the enums.

Jon Siwek
December 7, 2013, 2:58 AM
  • what would you think about moving the global type alias table into BroType as a private static member, along with a few static accessor methods to manipulate it?

Seems better. Done.

  • if there's a bro binary in PATH, it seems that "make doc" picks up that one instead of the one in src/build/bro

Fixed.

  • "make doc" shows received termination signal right before Running Sphinx. Is that expected?

Expected, I guess. It's just a result of the broxygen/ package calling terminate() in bro_init(). I do that so that scripts like listen.bro can be loaded during the doc-generation process. I don't want to filter all stderr to /dev/null, because that can give useful diagnostics when something goes wrong. Could grep out just that string, but I'm kind of fine with it...

– I'd remove the "Param" in the function descriptions (or is that a Sphinx thing?)

(I think) that was there in the previous Broxygen version, too. And I think it was only there to make the raw reST output look more like some Python docstring markup. But I think we're different enough that keep it isn't that beneficial. So removed.

– If a package has scripts both w/ and wo/ a further description, they appear separately in the package overview; see, e.g., scripts/base/frameworks/packet-filter/index.html

Think I fixed it, or at least made it render consistently.

– should we just omit _load_.bro in the package overviews?

Not sure, but I think it's easiest to just treat them no different than other scripts. In some cases there might actually be stuff other than @load in there that would be useful for a person browsing docs (or maybe even having quick access to the set of @load that is done is useful for quickly switching to other docs).

– I like how the records now show fields that are redefined somewhere else. Two further thoughts:
— how about changing from redef in XXX to added if XXX is loaded. For a user, it might be more intuitive to talk about loading another script than just redef.

Sounds good, changed to present if XXX is loaded, which seemed clearer that it's referring to the existence of the field.

— for the script that does the redef, can we show the new fields there as well? For example, in scripts/policy/protocols/http/software-browser-plugins.bro.html it would show not only that HTTP::Info got redef, but also repeat the fields it added right there. Same for the enums.

Should be technically possible (all the info is tracked), but there'd be some effort setting up new linkage to the doc. Also not sure how confusing it is to have pieces of the total type put in various places – right now there's always one canonical place for the full type and there's no duplication of any documentation. Holding off on doing anything for now.

Updated the /topic/jsiwek/broxygen branch with the changes.

Merged

Assignee

Unassigned

Reporter

Jon Siwek

Labels

None

External issue ID

None

Components

Fix versions

Affects versions

Priority

Normal