devpi For Fast Python Package Installation

Tags: devops

(I just upgraded to 1.2.1, and I’m updating this doc accordingly)

Getting Apt-Cacher NG installed with such little fanfare and such huge payoff put me in a mood to cache more stuff.

We have an embarrassment of riches when it comes to caching PyPi packages for a network of hosts. I found four contenders, and I didn’t look very hard. The long activity history as well as the well written documentation pushed me toward devpi.

It’s working so well that I feel no need to try the alternatives. It installed in a virtualenv, and it behaves under upstart management.

# /etc/init/devpi.conf
description "devpi"
 
setuid www-data
setgid www-data
 
start on runlevel [2345]
stop on runlevel [!2345]
respawn
 
exec /opt/devpi/bin/devpi-server --host 0.0.0.0 --refresh 3600 --datadir /opt/devpi/data --secretfile /opt/devpi/.secret

Then update your pip.conf file:

# ~/.pip/pip.conf
[global]
index-url=http://your_cache_server:3141/root/pypi/+simple/

And your .pydistutils.cfg file:

# ~/.pydistutils.cfg
[easy_install]
index_url = http://your_cache_server:3141/root/pypi/+simple/

(You can run this on your workstation. I have it running on a separate cache server VM because I have several hosts that make use of it.)