The Whys and Hows of the Oslo Namespace Change

Edit Jan. 26, 2015: After discussing this with the Oslo team some more, I discovered that almost everything I said here was already covered quite well in the drop namespaces spec (that I reviewed... ;-). You're probably just as well off going and reading that, but I'm going to leave this here just in case it happens to help get the word out.

The Why

During the Kilo cycle the Oslo team has been busy deprecating the oslo. namespace in all of our projects. The first question this probably raises for most people is: why? Unfortunately the answer to that is not simple, and I'm not sure I understand some of the deeper details of it myself. However, after having talked to some of the Python packaging folks that hang around the OpenStack project we came to the conclusion that namespace packages in Python 2 are just too broken for us to continue using them.

Now, many people may use namespaced packages in Python and never run into trouble, but part of the issue is that when you do hit one of these problems it's not at all obvious what is wrong. Let me give you an example (disclaimer, it's possible I have some aspects of this backwards, but the general scenario does demonstrate the problem):

Say you're working on a new feature in a project that uses a number of Oslo libraries. Let's also say that in the process of doing this, you hit a bug in one of the Oslo libs (which is unheard of, naturally ;-), perhaps oslo.utils, that prevents you from progressing on your feature, so you decide to fix it (thanks!). You check out the library code, find the bug, fix it, and want to verify that it gets you past the problem you were having in the other project. To do this, you pip install -e the library working tree into the other project's tox virtualenv so that you can make changes on the fly if you need to. You run the other project's unit tests, and all of a sudden oslo.db can no longer be found. What? You only made changes in oslo.utils, why is oslo.db broken now? In short: namespaces.

I won't get into the details of why this doesn't work, but suffice it to say that mixing install types, such as pypi packages and pip install -e source trees, with Python 2 namespaces is a recipe for headaches and disaster, not necessarily in that order. While this may not happen a lot in production environments, it does happen fairly often in development and has caused quite a lot of wasted time as people try to figure out what's going on when they're doing development involving Oslo libs. The oddly-named oslotest and oslosphinx libraries? Yep, that goes back to the namespace problem too. The awkward naming was an early attempt to address the issue.

We've worked around the problem in the past and thought we had a decent solution, but even that required installing Oslo libs in a non-editable way with devstack, so it caused other confusion. After investigating a number of dead ends it was decided that the only way to solve this once and for all was to eliminate the usage of Python namespaces for the Oslo libraries.

The WhoHow

That decision raised some pretty thorny backwards compatibility issues though. We couldn't just drop the namespace packages completely and switch to a namespace-free layout, because that would result in an impossible upgrade path for consumers. We also didn't want to try to maintain duplicate copies of all the code for the entire deprecation cycle needed to safely remove namespaces. Doug Hellmann (Oslo PTL) came up with a pretty slick solution though: We leave the namespace package files, but make them essentially redirects to the new, un-namespaced files. This allows people to continue importing from oslo.utils, but they actually get code from oslo_utils. After an appropriate deprecation period to allow everyone to migrate to the new names, we can remove the old oslo.* code and be done with namespaces forever! \o/

What does this mean for consumers of the libraries? For the most part it should be as simple as s/oslo./oslo_/ in the import statements. Possibly with some reordering to satisfy our alphabetical ordering hacking rule. We have run into a few issues where projects were using private symbols in a library (which are not included in the compatibility namespace modules), but honestly that's a good opportunity to fix those cases. Released Oslo libraries are intended to have a well-defined interface and no one should have to use private symbols. If they do it's an issue that should be addressed in the library.

Anyway, we've been getting a fair number of questions about this work so I figured I would put this together as a one-stop explanation of what's going on with the Oslo namespace work. I hope it's been helpful.