Networking

Just a quick announcement of a tool I wrote recently to help with debugging of Keepalived behavior in an OpenShift On-Prem IPI cluster. This is specifically intended to handle the logs from the keepalived pods running in the openshift-[platform]-infra namespace, although with a little work it could probably be generalized to work with most any Keepalived configuration.

The Problem

This is some design work I did a while back as a result of an edge case that we had not considered in the original design of the loadbalancer architecture for OpenShift on-prem networking. Our (mistaken) assumption was that apiservers would either be up or down and our healthchecks were written with that in mind. As it turns out, it is possible for a cluster to be in an unhealthy state but not completely down. This results in intermittent failures of API calls, which causes flapping of the healthchecks. One could argue that the healthchecks are correctly representing the state of the cluster, but the problem is that VIP failovers break all connections to the API which can exacerbate the instability of a flaky cluster. Each time the VIP fails over it forces every client to reconnect, and if the apiservers are already struggling to handle the load then having a huge number of connections come in at once just makes it worse.

Oh my.
-George Takei

Mostly writing this down so Google knows about it the next time I search. On a fresh VM I tried to use NMState to apply a configuration that included OVS bridges and interfaces. This failed with the error libnmstate.error.NmstateDependencyError: Open vSwitch support not properly installed or started. I had installed and started OVS, so I was very confused. I vaguely recalled that there was an integration package for NetworkManager, but a dnf search openvswitch turned up nothing.

Just a quick update on the TripleO Network Isolation Template Generator. A few new features have been added recently that may be of interest.

I've finally gotten around to recording a demo of the current iteration of my tool to generate TripleO network isolation templates. You can watch the demo video (in which I say "umm". A lot. Sorry.) or check out the tool itself. There are also some templates generated with the tool if you want examples to start from.