BLOG.info(self)

The Problem

This is some design work I did a while back as a result of an edge case that we had not considered in the original design of the loadbalancer architecture for OpenShift on-prem networking. Our (mistaken) assumption was that apiservers would either be up or down and our healthchecks were written with that in mind. As it turns out, it is possible for a cluster to be in an unhealthy state but not completely down. This results in intermittent failures of API calls, which causes flapping of the healthchecks. One could argue that the healthchecks are correctly representing the state of the cluster, but the problem is that VIP failovers break all connections to the API which can exacerbate the instability of a flaky cluster. Each time the VIP fails over it forces every client to reconnect, and if the apiservers are already struggling to handle the load then having a huge number of connections come in at once just makes it worse.

Fair warning: This is gonna be looooong. Proceed at your own risk. ;-)

Introduction

Since I started working with OpenShift on baremetal one of the things I've wanted to do is deploy OpenShift using OpenStack Virtual Baremetal to provide the host VMs. The usual developer setup is dev-scripts, which uses libvirt to stand up a virtual baremetal environment. This works fine, but it has a few drawbacks:

If you open a bug with NetworkManager, there is a high probability that the first thing they will ask you is to provide trace logs from around the time whatever bad behavior you're reporting occurs. This isn't terribly complicated to do, but most people are not familiar with the NetworkManager logging configuration so when asked for trace logs their first response is: How? I'm writing this up so I can just provide a link here when I get that question.

Oh my.
-George Takei

Mostly writing this down so Google knows about it the next time I search. On a fresh VM I tried to use NMState to apply a configuration that included OVS bridges and interfaces. This failed with the error libnmstate.error.NmstateDependencyError: Open vSwitch support not properly installed or started. I had installed and started OVS, so I was very confused. I vaguely recalled that there was an integration package for NetworkManager, but a dnf search openvswitch turned up nothing.

Just writing this down so I can find it easily in the future. My C922 webcam somehow got stuck at a 640x480 resolution, which looks weird in this day of widescreen monitors. The fix was the following command: v4l2-ctl -d /dev/video2 -v width=1280,height=720 My laptop also has a (bad) integrated webcam so that's why I had to specify /dev/video2. On Fedora I also had to install the v4l-utils package to have v4l2-ctl available.

I was running fstrim on a couple of old drives in anticipation of installing them in an even older RAID controller that doesn't support the trim command.[0] To do this, I formatted the entire drive as ext4, mounted it, then ran fstrim -v /mnt/temp to discard all of the unused blocks. I did this using a USB-SATA adapter, and what I noticed was that after running fstrim the adapter activity light was still blinking like crazy. I was curious if fstrim was still running, even though the command had completed already.

The Oslo team held its second virtual PTG this week. We had a number of good discussions and even ran slightly over the 2 hours we scheduled, so I think it was a successful event. The first hour was mostly topics relating to Oslo itself, while the second hour was set aside for some cross-project discussions with the Nova team. Read on for details of both hours.

This has been driving me nuts for years. The problem is that when I move the mouse to the far left edge of the screen, my mouse clicks are ignored. I can't right click to bring up the context menu, I can't use the scroll wheel to switch virtual desktops. The latter is my particular pet peeve because I always leave a gap at the left edge of the screen, and when I want to change desktop I just move the mouse over there and scroll.

Updated 2020-08-10, see edit below

I recently had to upgrade a CentOS 7 server to CentOS 8. I foolishly expected this to be easy since it wasn't a heavily customized install in the first place and most of the stuff I cared about was in my home directory. I was, as you may have guessed, wrong.

The problem was that CentOS 8 removed the driver for the Dell H700 RAID controller in my server. Setting aside the wisdom of that decision, here's how to get around it:

I've recently had occasion to do a fair amount of work with Go modules and vendoring, and I've had trouble finding simple how-tos for doing some (to me) basic things. It turns out they aren't that complicated, but they do involve multiple systems working together which may not be obvious to Go newbies (like me!). I'm writing this down both for my own reference and in the hopes that maybe it will help someone else get started with this stuff. I make no promises as to the accuracy of these steps, but they did what I needed.