Juggling the development of package-rich Python projects with mxdev

mxdev is a utility that makes it easy to work with Python projects containing lots of packages, of which you only want to develop some. It works on top of stable constraints and uses the power of pip and git.

You learn

• why we need it/ what problems it solves,
• how it is used in your day to day work (based on Plone Backend),
• how it compares to zc.buildout
• how to extend it,
• the current state of development and what our future plans are.

The release manager provides us with a pip constraints file with over 300 versions. You pip install Plone with these versions. Works fine. Now try to install a different version of plone.restapi, which is one of the pinned packages. Create an own constraints, interit from the upstream constraints, add new plone.restapi version. Try it: pip does not allow it, because there now are two conflicting version pins.

Now add mxdev to the requirements with a git url, in editable mode. Change something in it, commit.  Run pip install for this again. Boom, my changes are gone, because the package is checked out again, or reset to the given upstream commit.

Manually, you could download the constraints, edit them, and then it would work. Clone checkouts manually, and use filesystem path in the requirements.txt. It works, but you do not want this manually.

So I created a bash file to do this, but then thought, isn't there a Python tool for this? Well, there is Buildout. We love this, but it is unknown in the Python community.

So, I rolled my own solution: mxdev. It is a preprocessor. It reads its own configuration, fetches requirements and constraints files, replace constraints with version overrides, directly clone or update from vcs (git, etc). We have mx.ini for configuration, with version overrides and sections for packages to clone.

Packages and documentation: https://pypi.org/project/mxdev/

You can try it for Plone with cookiecutter-plone-starter.