NetworkManager TRACE logging

If you open a bug with NetworkManager, there is a high probability that the first thing they will ask you is to provide trace logs from around the time whatever bad behavior you're reporting occurs. This isn't terribly complicated to do, but most people are not familiar with the NetworkManager logging configuration so when asked for trace logs their first response is: How? I'm writing this up so I can just provide a link here when I get that question.

Manual

To enable trace logging for NetworkManager you need to do two things: Create a config file specifying a log level of TRACE, and then restart/reload NetworkManager so it picks up the new configuration. For example:

# echo "[logging]" > /etc/NetworkManager/conf.d/99-trace-logging.conf
# echo "level=TRACE" >> /etc/NetworkManager/conf.d/99-trace-logging.conf
# systemctl restart NetworkManager


And voila, you have trace logging. You can verify that by running journalctl -u NetworkManager and verifying that a large number of "TRACE" log lines appear.

Note that the filename doesn't strictly matter, but putting it at the end by using a prefix of 99 should allow it to overwrite any previous log settings that may exist in your regular configuration files.

OpenShift

While you can ssh into a node and make the modifications described above, OpenShift discourages direct access to cluster nodes so the recommended method is a machine-configuration. Note that an explicit restart of NetworkManager is not necessary in this case because machine-config-operator already reboots the machine after applying the new configuration. Also note that MCO works on pools of machines, which means this configuration will be applied to all machines in the pool, not just a single one. If that's a problem for you (and it might be, given the amount of log messages you get in trace mode), you'll just have to be a bit naughty and directly modify the node or nodes you want to enable.

For the machine-config option though, you just need to apply a machine-config such as the following:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 99-trace-logging
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,W2xvZ2dpbmddCmxldmVsPVRSQUNF
        mode: 0644
        overwrite: true
        path: /etc/NetworkManager/conf.d/99-trace-logging.conf

You should really verify the base64-encoded content there instead of trusting some random guy on the internet with the health of your cluster, but that will apply the necessary configuration to all of the worker nodes in your cluster. Replace the role label as appropriate.

Initial Deployment

Update Sep. 23, 2022: Since I first published this I've gotten questions about how to do this at deployment time for problems that prevent the cluster from being created in the first place. This is possible by including the machine-config above in the initial manifests. To do so, follow the instructions for machine-config customization in the OpenShift installer docs.