Log Parsing for On-Prem OpenShift (And More)

Recently I debugged a problem where the only logs I had to use were the Kubelet logs for the pod in question. Because there is quite a lot of logging in Kubelet, this was somewhat difficult and I concluded that it would have been much easier if I had a tool to slice and dice the logs in a more elegant manner. Initially I began writing a tool similar to my Keepalived log parser tool, but pretty quickly I realized that what I was doing was not specific to Kubelet and could be used for more general log parsing.

Because of that, I reworked the tool to be (more) general purpose. The Kubelet-specific configuration became just a preset and other presets can be added for further flexibility. At the time of this writing the only other preset is for Keepalived because even with the standalone tool it is still sometimes necessary to dive into the logs directly. It is possible to add arbitrary identifiers and filters to allow use with logs from any source. Not all logs will necessarily work as well as the builtins because the timestamp parsing may not work correctly, but adding more timestamp formats is not too difficult and could potentially even be included as a configurable option at runtime. I'll put that on the todo list.

In terms of functionality, I have a demo video for the tool that shows how to use it with both presets, but here's a quick overview. In the first column of the tool is a list of identifiers. This might correspond to the pod name in Kubelet logs or the VIP name in Keepalived. Lines that do not contain at least one of these identifiers will never be displayed by the tool, so think of this as the first level of filtering. You could arguably get similar functionality from a basic grep, except for the second column which I call the filters. This is to further refine the log lines returned by the tool, so think of this as a second level of filtering after the identifiers have been selected. Importantly, the filters will never show up in the results unless they also appear on the same line as an identifier. In situations like Kubelet logs this can be immensely helpful because there may be many log messages related to a specific pod, but you may only be interested in a subset of them. You could probably do something similar with command-line tools, but it would likely result in a pretty complex command to exactly replicate the results the tool gives you, and switching what you're looking at wouldn't be as simple as clicking a button.

Once you have your identifiers and filters configured, you can press the Parse button to actually apply them to the log file in question. Then there are several buttons you can press to view the results. Each identifier has a Filtered and All button that can be used to view the log lines for that identifier only, either filtered by the specified filters or completely unfiltered. In either case, filter messages will be highlighted for easy visual identification. At the top of the screen there are also two buttons, but this time for interleaved logs. This returns log lines with all identifiers, either filtered or not depending on which button you use. This can be helpful if you're looking at the interaction between multiple pods and want to view them all at once.

That pretty much covers the entirety of the tool's functionality. It's mostly a glorified grep/sed, but I think there's some value in not having to reinvent the CLI wheel when doing something like this.