[MidoNet-dev] Feature Proposal: Tunnel Health checks

Guillermo Ontañón guillermo at midokura.jp
Tue Feb 19 14:57:10 UTC 2013

Hi Galo,

Nice write up, I have a couple of questions/comments...

On Tue, Feb 19, 2013 at 2:54 PM, Navarro, Galo <galo at midokura.jp> wrote:

> Hi devs,
> Find below a proposal for the implementation of Tunnel Health checks.
> Thanks in advance for any feedback, new requirements, etc.
> # Use Case
> The goal of this feature is to watch tunnel's health with a double
> purpose: first expose problems in the virtual network that might be
> caused by the underlying network (as seen in the Netflix POC). Second,
> enable corrective actions, both manual via the API and automatic from
> inside MM (typically, recreate the tunnel).
> The solution presented here offers a way for each agent to maintain an
> updated status of the tunnel between its own local host and all its peer
> hosts. By design, this status relationship will not be symmetric because
> there is no guarantee that if A can communicate with B, B will also be
> able to communicate with A. Therefore, given any pair of nodes, each
> will need to watch the connectivity from itself to the other.
> ## API changes
> The API changes will be highly conditioned by the desired GUI to display
> tunnel health to users. I will make here a proposal that conditioned to
> reusing existing data and functionality. But of course let me know if
> you envision a different UI and I'll propose a suitable API for it.
> There are already methods to query existing port stats that provide
> rxBytes, txBytes, rxPackets, txPackets. These can be used by the backend
> to check tunnel's health, and since there are API methods to query this
> data it would be possible to offer a simple visualization of the
> connectivity status on tunnels. For example, the GUI could show the list
> of port stats (all, or filtering only tunnel ports) where the RX and TX
> values would clearly indicate whether there is connectivity or not.
> The backend proposal includes maintaining data in Cassandra to flag
> broken or suspected-to-be-broken tunnels. This data could be exposed
> through an API call such as:
> GET failing nodes
> - Returns a Collection<TunnelHealthAlert> where each TunnelHealthAlert
>   contains
>   - HostId, PeerHostId, TunnelPortId and Counter. Where Counter is the
>     number of consecutive checks without Host with id <hostId> receiving
>     data from Host with id <PeerHostId>.
>   - Only tunnels with >0 errors will be returned, all others can be
>     assumed healthy
> - If considered useful, the GET could accept optional filters such as
>   HostId (show only tunnels from/to <HostId>)
> It'd be convenient to offer users the option to take corrective actions
> on tunnels with no connectivity. This would avoid the need to restat MM
> as seen in the Netflix PoC. The simplest of these actions would be
> offered by an API call:
> POST: request recreation of tunnel betweeh hostId1 and hostId2
> - Receives both hostIds as parameters
> Some questions regarding needs for the GUI:
> * Q: Do we want to be able to enable/disable tunnel health checks
>   altogether?
> * Q: Do we care about zones?
> ## Backend changes
> The idea would be to have a component (which I'll call here
> TunnelDoc) in each MM agent that monitors the state of its tunnels
> FROM the peer to itself. This component will use the metrics published
> for each port to evaluate the connectivity from each peer to its own
> host.
> Let's say we have two peers A, B connected by a tunnel
> - TunnelPorts become active on each side of the tunnel, the TunnelDoc
>   becomes aware of local ports and starts taking care of them.
> - Regularly, for each cared-for tunnel the TunnelDoc:
>     - Sends a packet to the other peer
>     - Logs variation on RX value of the PortStats on the tunnel's local
> port.
>     - If variation = 0, increment a "no-increment" counter
>     - If "no-increment" counter > threshold, trigger alert message for
>       lack of connectivity on the REVERSE direction of the tunnel (e.g.:
>       if the TunnelDoc at A spots no RX, the alert refers to loss of
>       connectivity from B to A).
>     - Implement whatever corrective measures upon receiving the alert
>       (typically, the DatapathController could recreate the tunnel)

This is not a lot of extra traffic, but the number of tunnels does grow
quadratically with the number of MM agents. I propose a slight variation on
the above to avoid sending traffic on non-idle tunnels, along the lines of
what is done by IPsec's dead peer detection:


Basically, from the POV of view of one of the nodes, it looks like this:

   * Monitor idleness (by looking at RX as you outline above) and do
nothing and consider the tunnel healthy while idleness doesn't go above a
certain threshold.
   * When the tunnel becomes idle, send an "are-you-there" packet to the
Peer (we could just use the tunnel-key for this).
   * When an "are-you-there" packet is received, reply to it with an Ack.

> The counters described above would be stored in a common data structure
> in Cassandra so that API clients could easily retrieve the list of
> failing tunnels as described above.

I wonder why Cassandra and not Zookeeper?


> Thanks!
> /g

-- Guillermo Ontañón
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.midonet.org/pipermail/midonet-dev/attachments/20130219/869bcf2c/attachment.html>

More information about the MidoNet-dev mailing list