BLOG.info(self)

I'm working on a bug that involves functionality at a fairly low level, at least much lower than I usually deal with. One request from the experts who are helping me with this was to enable kernel bond debugging, which is not something I've ever done. At runtime this isn't too difficult, just two commands:

sysctl -w kernel.printk=8
echo 'module bonding +p' > /sys/kernel/debug/dynamic_debug/control

Over the past few years there have been quite a few changes in how day one (or perhaps more accurately, deployment-time, since they also apply to scaleout operations on day two) networking functions. This particular phase of networking is especially tricky because cluster resources are not, for the most part, available yet. This means you can't use any of the normal operators that handle network configuration later in the deployment process.

Previously the Keepalived Log Parser required manual log collection via must-gather or some other mechanism. Recently I added some functionality to allow it to read logs directly from a live cluster by taking a kubeconfig as input instead of a log directory. This may make the tool more useful to non-developer users who want to see what's going on in their cluster, which is a semi-regular request we have gotten.

Here's a quick YouTube video demonstrating this new functionality. Hope you find it useful.

Like most tech companies these days, Red Hat is encouraging everyone to brush up on their AI knowledge. For my part I have been doing a number of online training courses lately and thought I would write up my experience for posterity.

Just a quick announcement of a tool I wrote recently to help with debugging of Keepalived behavior in an OpenShift On-Prem IPI cluster. This is specifically intended to handle the logs from the keepalived pods running in the openshift-[platform]-infra namespace, although with a little work it could probably be generalized to work with most any Keepalived configuration.

The Problem

This is some design work I did a while back as a result of an edge case that we had not considered in the original design of the loadbalancer architecture for OpenShift on-prem networking. Our (mistaken) assumption was that apiservers would either be up or down and our healthchecks were written with that in mind. As it turns out, it is possible for a cluster to be in an unhealthy state but not completely down. This results in intermittent failures of API calls, which causes flapping of the healthchecks. One could argue that the healthchecks are correctly representing the state of the cluster, but the problem is that VIP failovers break all connections to the API which can exacerbate the instability of a flaky cluster. Each time the VIP fails over it forces every client to reconnect, and if the apiservers are already struggling to handle the load then having a huge number of connections come in at once just makes it worse.

Fair warning: This is gonna be looooong. Proceed at your own risk. ;-)

Introduction

Since I started working with OpenShift on baremetal one of the things I've wanted to do is deploy OpenShift using OpenStack Virtual Baremetal to provide the host VMs. The usual developer setup is dev-scripts, which uses libvirt to stand up a virtual baremetal environment. This works fine, but it has a few drawbacks:

If you open a bug with NetworkManager, there is a high probability that the first thing they will ask you is to provide trace logs from around the time whatever bad behavior you're reporting occurs. This isn't terribly complicated to do, but most people are not familiar with the NetworkManager logging configuration so when asked for trace logs their first response is: How? I'm writing this up so I can just provide a link here when I get that question.

Oh my.
-George Takei

Mostly writing this down so Google knows about it the next time I search. On a fresh VM I tried to use NMState to apply a configuration that included OVS bridges and interfaces. This failed with the error libnmstate.error.NmstateDependencyError: Open vSwitch support not properly installed or started. I had installed and started OVS, so I was very confused. I vaguely recalled that there was an integration package for NetworkManager, but a dnf search openvswitch turned up nothing.

Just writing this down so I can find it easily in the future. My C922 webcam somehow got stuck at a 640x480 resolution, which looks weird in this day of widescreen monitors. The fix was the following command: v4l2-ctl -d /dev/video2 -v width=1280,height=720 My laptop also has a (bad) integrated webcam so that's why I had to specify /dev/video2. On Fedora I also had to install the v4l-utils package to have v4l2-ctl available.