Zuul Status Page Production-Ready Configuration

Submitted by bnemec on Wed, 01/10/2018 - 20:06

About a year ago I got fed up with all the dynamic Zuul status pages (like this one). They're extremely heavyweight pages that make my browser grind almost constantly and there's no way to disable the auto-refresh while you're looking through the list. At least not that I found, and when I asked on openstack-dev the responses agreed.

So I wrote my own.

It was originally designed to keep track of the check-tripleo queue for OVB jobs, but since then I've added better support for the other queues as well. I'm very happy with how it works (shocking!), with one major exception: Periodically the webapp behind it will just stop responding. I suspect it has to do with connectivity issues to the Zuul status JSON endpoint and I've tried to set some timeouts to avoid that, but nothing I've tried has worked. Once a request gets stuck the entire app stops responding.

I suspect a big part of the problem is that I was using the WSGI server built in to Python instead of a production-ready one like uWSGI. It does work, but it seems to be a single-threaded implementation with no ability to handle parallel requests. When a request gets stuck this obviously is a problem. To address this, I've switched to running the app in uWSGI fronted by nginx. This seems to be a popular configuration and it's being used by some OpenStack services as well so it seemed like a useful thing to learn about.

You can see my uWSGI configuration file in the Git repo. I'll include my nginx config file at the end of the post. The command I use to start uWSGI is sudo uwsgi --ini zuul-status.ini

There are a few things I suspect are not ideal (socket in /tmp and running as root), but this is just running in a throwaway vm so if someone does manage to hose it up through some sort of exploit it won't be the end of the world. Be aware that my configuration is not necessarily following best practices though.

I figured I would write this up since I had some trouble finding complete, working configurations for the Python/Nginx/uWSGI combination. Found lots of people with non-working configs though. :-)

A few other environmental notes:

  • I installed nginx on CentOS 7 using the repo provided by nginx. It sounds like it might also be available in EPEL.
  • I also enabled EPEL to get uWSGI.
  • The uwsgi-plugin-python package is also needed to run Python apps.
  • I create a virtualenv for the webapp in the venv directory that contains all of the dependencies for it.
  • I also shut off SELinux. Yeah, yeah, see above about throwaway VM.

nginx conf.d/default.conf:

server {
    listen       80;
    server_name  localhost;

    location / { try_files $uri @zuul-status; }
    location @zuul-status {
        uwsgi_pass unix:///tmp/uwsgi.sock;
        include uwsgi_params;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

And that's pretty much it. The app has only been running in uWSGI for a few hours so I can't say whether it will actually help with stability, but hopefully this will be helpful to someone anyway. I'll try to remember to update this post after a while to report how it worked.

Tags: