Taking Advantage of Squid for TripleO Development

I did a post in the past about using caching and local mirrors to speed up TripleO development, but at this point TripleO has changed so much that pretty much everything in that post is completely outdated, so I thought it was time for another one about my current caching setup.

This time around I haven't set up any local mirrors. We're no longer installing from pip, so a local mirror for that isn't going to be beneficial anymore, and I haven't been willing to take the hit to set up an entire CentOS/EPEL mirror. Fortunately, Squid can still help a bunch.

First, it's important to configure Squid correctly. If some tweaking of the default configuration is not done then it will either be less effective than it could be or, in some circumstances, flat-out break your installs. My particular Squid server is running on a Fedora VM on a local OpenStack instance, so the changes I'm suggesting are based on what I had to do with the default Fedora Squid configuration.

The main things that need to be right are cache_dir and the acls. In my case, because I run my local network on a subnet I'm not supposed to, I had to add it to the localnet acl. I simply added this line to the local networks section: acl localnet src 11.0.0.0/8. At the bottom of squid.conf, I also added the following:
acl repodata url_regex .*repodata.*
cache deny repodata

This is important because without it squid will cache repodata files from the yum repos, and because of the dynamic way our Delorean repos are built that can cause significant problems. It does mean you will lose some time downloading repodata files the first time yum runs, but since those are cached internally in yum anyway it isn't too big a hit.

Finally, make sure that disk caching is properly configured. Otherwise you may find that you aren't getting the full benefit from Squid. My disk cache configuration looks like this:
# Must come before the cache_dir line
maximum_object_size 2 GB
# Uncomment and adjust the following to add a disk cache directory.
cache_dir ufs /var/spool/squid 15000 16 256

This allows Squid to cache up to a 2 GB file (the CentOS image is nearly 1 GB, so this leaves us some wiggle room - although I always manually upload the CentOS image so it doesn't have to be downloaded separately for each undercloud anyway), and allows it to use up to 15 GB of disk space for caching. I'm running on a 20 GB VM, so it's a value that seems to allow plenty of cache space without filling up the VM.

All that remains then is to tell the TripleO tools to use your squid cache. I always point the undercloud yum.conf at Squid explicitly with a line like proxy=http://roxy:3128. To improve the caching, you also want to point at a specific mirror, and not use the mirrorlist. This can be done by swapping the mirrorlist lines in /etc/yum.repos.d/*.repo for the baseurl lines in those files (both are included, but one will be commented out). Once you've done that, your undercloud install should be using Squid to cache rpms so that future installs will be noticeably faster. I have a base CentOS image with these changes pre-configured that I always use as the basis for my undercloud.

The other major place Squid can help is in image building. Although diskimage-builder caches packages locally anyway, the first time images are built all of the packages have to be downloaded to the local cache, and since many of those packages are shared with the undercloud Squid should already have cached them. Plus, if you've already built images using Squid it may have cached all of the packages you need. To tell the image build to use Squid, export the following before running the image build:
# DIB_LOCAL_IMAGE is optional. Only set this if you have a pre-downloaded CentOS image on the undercloud already.
export DIB_LOCAL_IMAGE=CentOS-7-x86_64-GenericCloud-1602.qcow2
export DIB_DISTRIBUTION_MIRROR=http://mirror.centos.org/centos
export DIB_EPEL_MIRROR=http://dl.fedoraproject.org/pub/epel
export http_proxy=http://roxy:3128
export no_proxy=192.168.24.1,192.168.24.2,192.168.24.3

Make sure no_proxy includes all three undercloud IP addresses, namely local_ip, undercloud_public_host, and undercloud_admin_host. Now you can run an image build and it will pull any packages it can from the cache.

Important: As of this writing, there is a bug with overcloud deployment that causes issues when http_proxy is set and network isolation is being used. I suggest that you not run a deployment with http_proxy set if you are using network isolation and the proxy won't have access to all of your overcloud endpoints (public and ctlplane in particular). This should be fixed in the near future, but it's something the be aware of in the meantime.

And that should do it. There's other work going on in TripleO right now to speed some of these processes up even further, but if you are doing a lot of undercloud installs or image building this can save quite a bit of time already.