Alexander Loechel: Porting RestrictedPython to Python 3

Python 2 reaches end of life in 2020. Plone does not run on Python 3 yet. We started talking about that in the Bristol conference in 2014, and said it could not be done. Especially hard would be the packages AccessControl and RestrictedPython.

AccessControl is handled meanwhile. I will talk about RestrictedPython. Why should that be a blocker?

Hanno Schlichting, Zope Release Manager, once said: 'Every piece of Zope2 that was not adopted by Plone is literally dead.' So we adopted it. Problem: RestrictedPython had almost no documentation, and low test coverage.

'Where Zope leads, Python follows,' used to be a saying. There are some pieces in core Python that are there specifically for Zope, like the compiler package that is used in RestrictedPython. This RestrictedPython is used in TTW (Through The Web) code, like scripts that some authorized users can write.

In 1994 in the first PyCon at NIST, there was already talk about creating some kind of restricted Python. RestrictedPython is no sandbox: someone is always going to be smarter than you are, and break out of the sandbox. So you get a limited, safe subset of the Python language and grammar. You don't get file access, for example.

John Johnson: 'First solve the problem, then solve the code.' In other words: first understand the problem.

Python 2 had the compiler module and its ast class. But: not fully documented, and no upgrade path for Python 3 described. Compiler knowledge was necessary to port RestrictedPython. I knew something about it from my studies, so I gave it a try. We started looking into it at the Plone Open Garden 2015.

Ken Beck: 'Any fool can write code that a computer can understand. Good programmers write code that humans can understand.' So: I wanted to make the code more readable and documented and tested, so others can work on it too.

Instead of compiler.ast there is ast.AST in Python 3, so that was a start. The builtin compile function since Python 2.6 accepts ast.AST as input and compiles it to Python byte code. With Python 2.6 and 3.4 as minimum, we could get somewhere. I was happy to be able to go to a Plone conference in Japan to work on it.

I started with making it comply with our coding conventions, without changing anything else. And added tests. We started at 18 percent test coverage, and are now at 95 percent.

RestrictedPython depends only on the standard Python library, which made the test setup easier. I switched to tox so we could test multiple Python versions at the same time. Also pytest, which allows parameterisation, which means we could test the old and new implementation next to each other, so we can prove that they work the same.

People from Gocept and other Zope core developers joined in, and together we made it. In May 2017 we released version 4.0a1, and now we have 4.0b2.

Ian Hickson: 'Things that are impossible just take longer.'

My wish is to make RestrictedPython more known, and enable other projects and frameworks to use a 'safe' Python interpreter through the web. Think of projects like Jupyter, where you can write online code in a sandbox; if you are knowledgeable, you could probably kill the server, and RestrictedPython would help here. It could be interesting for Django, Pyramid, guillotina.

Lessons learned:

• impossibly to port code: probably not
• adopt modern tools and frameworks like tox and pytest

Update the best practices for Plone development.

See the slides.