broctl needs to keep track of desired state

Description

On a multi-node cluster the following sequence of events can happen:

  • A cluster node (node-2) has a power problem and is shut down

  • broctl stop is ran on the manager

  • broctl then fails to stop bro on node-2

  • node-2 reboots

  • broctl cron restarts bro on node-2 because the last known state is up

The problem can happen in reverse as well, where broctl will not restart bro on a node that was down.

The problem arises because broctl stores the actual state of the nodes, but not the desired state. commands like stop and start need to set the desired start first, and then attempt to sync reality with that state information. broctl cron then just needs to attempt the similar sync.

Environment

None

Assignee

Justin Azoff

Reporter

Justin Azoff

Labels

None

External issue ID

None

Components

Fix versions

Priority

Low
Configure