I have worked in the German air force. They have rules on how to write software. For example, you start with user requirements. In open source, who is writing down user requirements? The requirements are there, but they are often implicit. Write tests that test the requirements.

'Testing leads to failure. Failure leads to understanding.'

A test is a specific set of assertions.

You can test requirements, design, interfaces, code/implementation, documentation (embedded code), conventions.

The basis for tests in Python is the unittest module. It has lots of specific assertions, like assertEqual, assertIsNotNone, etc. But why can't we simply use assert? That is what the pytest module does instead. It makes writing the test simpler. It is by now the de facto standard in the Python world. It has a pluggable add-on system.

There is unittest2, nose, nose2, but today they are mostly outdated. Use pytest.

Robot framework is used by Plone to do web testing with a real browser.

Richard Feynman: 'The first principle is that you must not fool yourself, and you are the easiest person to fool.'

For test runners we have the unittest testrunner, zope.testrunner, pytest-testrunner, gocept.pytestlayer. Such a runner collects tests for execution and shows the outcome to the user. If you use unittest, you can use these. A test runner can interact with other tools, like coverage.

On a command line you can usually run python setup.py test to run tests. Or you have scripts to run it, like buildout can install with a recipe.

You want to run your tests automatically on a test server:

• Travis CI has Linux and MacOSX machines, and are perfect for pure Python tests
• Circle CI has Docker containers for Linux and MacOSX
• Appveyor tests on Windows.

On Travis you can use travis_retry in your .travis.yml file to retry a command three times. This can help when there is a temporary network problem.

tox is a test invocation tool. You use it to run your tests on multiple environments, for example Python 2 and 3, or with an extra package installed. You can use additional helpers like pyenv for virtual environments. One of the environments can apply isort, or run zest.releaser, even if these environments are not run by default. Or you can run a linter and have it report on your code quality.

My wishes for better practices in Plone:

• Adapt tox on all packages.
• Switch to different package structure and enforce that, like bobtemplates.plone, with docs, src, Tests.

Maybe do not ship with the tests in the PyPI releases: this code will not be run in production. Tests belong in the source distribution, but not in binary packages like wheels or eggs. The unit tests should be tested within the actual package. Integration tests could live somewhere else.

See http://plone-best-practices-discussion.readthedocs.io/

See the slides.

MVP (minimal viable product) approach can be good. Start with a basic skate board, then go for the best skate board in the world. Create a set of user stories, like: login, add, edit, delete content, add images. SEO so you can be found on search engines. Get something basic where companies and individuals can jump on board. Start with a small project that can grow.

'Medium' editor can be an option, more lightweight than TinyMCE. Editors can be fine on their own, but are usually hard to extend. Just some inline styles, like bold, italic, link, can be enough as basis. We can start with any editor and try to make it work. We can base on the GraphJS editor, where Rob Gietema did a prototype. We don't want to reinvent our own editor.

Which exact editor we use, should be an implementation detail. We should still decide what the features should be, how the user experience should work for the editor.

We need to put an editor in the hands of actual users and see if they can actually use it or are confused.

We know about React and Angular, and they can battle it out. But what css framework should we choose? If we have a prototype with either React or Angular, then after that we will be more able to pick a certain css framework.

Just get something done. If we wait until we are sure, then it will automatically be the wrong choice by the time we are done.

We will not take everything we currently have in Plone and reimplement it in Javascript. Expose the backend to the frontend, and then you can do on the frontend whatever you want to do. Want other widgets? Take another javascript widget library.

Don't try to solve too many problems at the same time.

We can also just call it a proof of concept, then no one will care about the implementation. Hire someone to do it in PHP, that is fine too.

MVP:

• login
• add a page with title and text, no wysiwyg yet
• view a page
• edit a page
• mark body text as bold, italic, add a link
• add image to the body text of a page. That includes folderish content, and drag and drop, so that is a large thing that can be split up
• Take less than half a second to view the page.
• Let Google see the page.

The backend can just be current Plone, maybe with a bit of helper code.

We may need a folderish page type, where you can drop files and images. But let's not discuss folderish types for hours, let's start building. Try it and see what works.

If with plone.restapi it works on Plone 4, then that makes it far easier to show on current existing sites to actual users.

But apparently we want to store the page content as json and not html, so that is not likely on Plone 4. Well, for new content it could work, but you will not be able to edit content that was created with TinyMCE. For example, you cannot edit tables yet.

Is the user the end user, or the frontend developer?

For A/B testing, we could strip down TinyMCE to only do what our new UI will be able to do, for better comparison.

It is not up to us to decide if this goes into Plone core or not. We just want to show the Pastanaga UI.

For notifications I would normally use Redis and PubSub. Then you need an app server, a way of scaling this, centralized logging, etcetera. You end up with an overengineered system for which you need three fulltime developers. Can we do better?

I use Firebase. This is a suite of lots of things, inluding:

• free CDN and SSL
• easy OAuth
• a free tier
• good scaling

It includes the Firebase realtime database which I will use. You can register a new firebase based app only and start building quickly.

Some packages in combination with React:

• reactfire
• react-redux-firebase
• firebase

I use Pyrebase on the Python backend.

In the frontend, I use redux-saga, a redux middleware for handling side effects. It communicates directly with the API.

Demo code: https://github.com/adaschevici/firebase-demo

Twitter: @_arturdas.

At Skyscanner we use lots of microservices for hotels. Microservices talk to each other with HTTP. The languages that we use all know how to talk HTTP.

We need to know what is happening in our micro services, like: which requests take more than 20 seconds? With aioHTTP middlewares we added code to check this. We make this visible in Kibana.

We wanted to follow the code path taken by a request. aiotask-context stores information within the current asyncio.task instance. The request id is stored on the task by a middleware. That way we can follow a request through the code.

DNS in AWS with aioHTTP. DNS TTL is usually 60 seconds in AWS, so IP addresses can change, and it can be more addresses. But aioHTTP versions below 2 did not support this: they cached the DNS requests. We created code to handle this and cache this for only a short time. And we throttled the DNS events, to avoid querying the DNS a hundred times when you fire a hundred requests very quickly after each other.

We call microservices with a timeout of one second, and catch the error. Timeouts can get triggered when the reactor is saturated. We use asyncio.Future and can cancel such a future when we detect a timeout.

Desired plans:

• trace queued operations when the HTTP pool connection limit is reached
• AWS Xray support
• Back pressure at the HTTP layer. When the reactor is too busy, return a 504 error. Scale horizontally when you get lots of 504 errors.

See the slides.

Python 2 reaches end of life in 2020. Plone does not run on Python 3 yet. We started talking about that in the Bristol conference in 2014, and said it could not be done. Especially hard would be the packages AccessControl and RestrictedPython.

AccessControl is handled meanwhile. I will talk about RestrictedPython. Why should that be a blocker?

Hanno Schlichting, Zope Release Manager, once said: 'Every piece of Zope2 that was not adopted by Plone is literally dead.' So we adopted it. Problem: RestrictedPython had almost no documentation, and low test coverage.

'Where Zope leads, Python follows,' used to be a saying. There are some pieces in core Python that are there specifically for Zope, like the compiler package that is used in RestrictedPython. This RestrictedPython is used in TTW (Through The Web) code, like scripts that some authorized users can write.

In 1994 in the first PyCon at NIST, there was already talk about creating some kind of restricted Python. RestrictedPython is no sandbox: someone is always going to be smarter than you are, and break out of the sandbox. So you get a limited, safe subset of the Python language and grammar. You don't get file access, for example.

John Johnson: 'First solve the problem, then solve the code.' In other words: first understand the problem.

Python 2 had the compiler module and its ast class. But: not fully documented, and no upgrade path for Python 3 described. Compiler knowledge was necessary to port RestrictedPython. I knew something about it from my studies, so I gave it a try. We started looking into it at the Plone Open Garden 2015.

Ken Beck: 'Any fool can write code that a computer can understand. Good programmers write code that humans can understand.' So: I wanted to make the code more readable and documented and tested, so others can work on it too.

Instead of compiler.ast there is ast.AST in Python 3, so that was a start. The builtin compile function since Python 2.6 accepts ast.AST as input and compiles it to Python byte code. With Python 2.6 and 3.4 as minimum, we could get somewhere. I was happy to be able to go to a Plone conference in Japan to work on it.

I started with making it comply with our coding conventions, without changing anything else. And added tests. We started at 18 percent test coverage, and are now at 95 percent.

RestrictedPython depends only on the standard Python library, which made the test setup easier. I switched to tox so we could test multiple Python versions at the same time. Also pytest, which allows parameterisation, which means we could test the old and new implementation next to each other, so we can prove that they work the same.

People from Gocept and other Zope core developers joined in, and together we made it. In May 2017 we released version 4.0a1, and now we have 4.0b2.

Ian Hickson: 'Things that are impossible just take longer.'

My wish is to make RestrictedPython more known, and enable other projects and frameworks to use a 'safe' Python interpreter through the web. Think of projects like Jupyter, where you can write online code in a sandbox; if you are knowledgeable, you could probably kill the server, and RestrictedPython would help here. It could be interesting for Django, Pyramid, guillotina.

Lessons learned:

• impossibly to port code: probably not
• adopt modern tools and frameworks like tox and pytest

Update the best practices for Plone development.

See the slides.