mxdev is a utility that makes it easy to work with Python projects containing lots of packages, of which you only want to develop some. It works on top of stable constraints and uses the power of pip and git.

You learn

• why we need it/ what problems it solves,
• how it is used in your day to day work (based on Plone Backend),
• how it compares to zc.buildout
• how to extend it,
• the current state of development and what our future plans are.

The release manager provides us with a pip constraints file with over 300 versions. You pip install Plone with these versions. Works fine. Now try to install a different version of plone.restapi, which is one of the pinned packages. Create an own constraints, interit from the upstream constraints, add new plone.restapi version. Try it: pip does not allow it, because there now are two conflicting version pins.

Now add mxdev to the requirements with a git url, in editable mode. Change something in it, commit.  Run pip install for this again. Boom, my changes are gone, because the package is checked out again, or reset to the given upstream commit.

Manually, you could download the constraints, edit them, and then it would work. Clone checkouts manually, and use filesystem path in the requirements.txt. It works, but you do not want this manually.

So I created a bash file to do this, but then thought, isn't there a Python tool for this? Well, there is Buildout. We love this, but it is unknown in the Python community.

So, I rolled my own solution: mxdev. It is a preprocessor. It reads its own configuration, fetches requirements and constraints files, replace constraints with version overrides, directly clone or update from vcs (git, etc). We have mx.ini for configuration, with version overrides and sections for packages to clone.

Packages and documentation: https://pypi.org/project/mxdev/

You can try it for Plone with cookiecutter-plone-starter.

The average client has no idea what Plone is, and they don’t really care. They won’t trust something they’ve never heard of, and there are plenty of other solutions out there that claim to offer the same things.

Designers and developers, on the other hand, do care about Plone: an elegant, powerful, secure, scalable, flexible solution to clients’ content management dilemmas. However, clients generally don’t care about the technical stuff – they just want to be online. It’s our job to tell them why Plone is the way to go.

In this talk, we share what we’ve learned about selling Plone to people who don’t care over the last 15 years.

When planning the trip here from South Africa, I was looking for a backpack. Why did I buy this one?

1. It has some specific features.
2. It looks good.
3. It feels good.

I actually started at the bottom of the list. The brand is known to care about environment and their workers. Then number two, I like this black one. And only then I looked at the first item om the list.

We work at Juizi: websites worth having. We use Plone, and need to convince clients that Plone is a good choice. Mostly, they don't care.

Why don't people care about Plone? They don't know the name. Similar functionality is offered by other names that are more familiar.

What is Plone? You are here, so you know it. Plone is not so unique: most features are offered by most other CMSes as well. Some features we like: easy user management, content workflow and rules, many different content types. But the client is after a solution. What are benefits to the client? Explain the Plone features in terms of the customer. Will it get me online? Can my website grow? Is my website safe?

How do we sell Plone to people who don't care? Focus on the benefits, like: presence, freedom, safety.

Solve problems, rather than build new things. Clients think they are unique, but try to see the general feature that the really need. Think inside the box that Plone arrives in. Listen to wants and address needs. Never say never. Help the client funnel their budget to where it is needed.

Our team is our unique selling proposition. It sets us apart from anyone else. We don't just say we know how to use Plone. We highlight use cases. Clients stick around for years. A lot of business comes in through word of mouth.

In the backpack I bought, I could see the maker cared about it. His name was even in the label. We care about Plone.

Remember what clients are buying. They buy presence, freedom, and safety. Ensure they get this and they'll buy you.

Audience:

These are calls to your emotions, offering presence, freedom, safety. Same as years earlier we said: "Plone gives you peace of mind." Can we prove that we are better at this than others? You can put up testimonials that highlight these things. Marketing in general is an ongoing conversation. Sales is story telling. Get them to listen, and they fall deeper and deeper in love with Plone.

If you enable your client to nicely and easily tell their story to their clients, then you both win.

Send them to the demo sites to try it out.

[Maurits: idea for my own company that I am starting up: have a few pages or paragraphs with features, and link them to a demo page where they can try it out themselves.]

Find out how to level up recommendations to your website based on machine learning open source Python library scikit-learn. Now that we’ve tried its simple and efficient tools ourselves, we will show you hands-on how you can benefit from them. We developed a useful add-on for both Plone Classic and Plone Volto. Get smart content recommendations by using basic Natural Language Processing to integrate this content recommendation system which is accessible for everybody.

Vectorisation: create a vocabulary of all unique words, a bag of words. With countvectorizer you calculate the amount of appearances of each word. This leads to a vector, think of a line in a graph. Then for an unknown text you can compare its vector with the list of known vectors, and see what the nearest neighbour is, using sklearn.neighbors.

interaktiv.recommendations is a Plone package for Classic and Volto. It provides a behaviour which can be added to the content type of your choice, plus a control panel. Say you add this for the Page content type. Then on each page you can show a list of other pages that are similar, so are recommended for the user to read.

Future plans:

• auto tagging
• dimensionality reduction with part-of-speech-filtering and lemmatisation, which breaks the words down to their base form.

Slow tests suck. They are annoying and slow down you and your team. Hours and hours of engineer time is lost waiting for the CI to finish. I'll go through a number of approaches and principles to help you write fast tests.

I started plone.api way back, now in core. I have organised about twenty sprints. Next up is the Nix(OS) sprint 21 to 25 November in Lanzarote.

I developed an app for security on Mac, see https://paretosecurity.com/. We had tests for this, they were fast, but after a while I checked again and it took 3 minutes for only 300 unit tests. What? We need to bring this down. Long tests can cost money, and at least they cost developer time. If the CI pipeline takes twenty minutes, then you cannot wait for it, and you cannot start anything new in that time. There are a few things you can try.

Try one approach, measure it locally, measure it on CI, merge when it is an improvement, wait a few days before trying the next thing.

Use hyperfine to compare results of commands.

Running tests

Throw money at the problem: make sure your team has fast laptops and that you fast have CI runners. Buy a Mac mini for less than 1000 euros and use it as a fast CI runner in GitHub or GitLab.

Try pytest --collect-only. Ballpark: should take 1 second for 1000 tests. If it takes longer, there is something wrong. You are probably telling it to look in too many directories.

Try pytest --collect-only --noconftest. Does this differ much with the previous result? 

export PYTHONDONTWRITEBYTECODE=1

Usually we know a few tests that are slow. You can mark them as slow, and do not run them by default, only if you run them explicitly. Then do run them on CI.

Try pytest-incremental or pytest-testmon.

Writing tests

Make sure tests are not doing any network I/O, like to GitHub or gravatar. Use pytest-socket to get them to fail, and then fix them by mocking the network access.

Use pyfakefs to use a fake filesystem in memory.

Do all tests need a database? If tests only do simple things, like doing some calculations, do not setup a database. Or maybe not all tables are needed for this test.

Can I create the database only once? At the end of a test, you can truncate tables, so you don't need to recreate them.

Can I populate the database only once?

Parallellisation

The big wins are here, but it is more involved, so start with the others first.

pytest-xdist helps here. Make sure your parallel runs don't influence each other, like writing to the database when another process truncates it.

pytest-split is good for the CI, not for local runs.

Now that your tests are fast, how do you keep them fast? We made BlueRacer.io. Free for per personal use and open source software.

Extra tip: run pytest --lastfailed to only run the tests that failed in the last run.

See https://github.com/zupo/awesome-pytest-speedup for the full list with more information.

Audience: gocept.pytestlayer helps for running Plone tests in pytest.

iA.Délib is used by more than 250 public authorities in Belgium: cities, public social core, fire fighters, etc. It is a list of packages to build applications. Plone has many features to support this, but it needs to be optimised. We have a theme, dashboards, office document production, contacts management. When all is setup, you can build a new application within weeks.

We use collective.contact.* and collective.eeafaceted.* packages. We have special workflows for documents. When a user is not allowed to do a transition we do show a grey button for this. When they hover over it, they see the reason why they are not allowed. We use dexterity.localrolesfield, linking roles to an organisation.

We worked on caching and performance. We are never using varnish; plone.app.caching gives us enough power.

When you reindex an object in Plone, it will update all metadata, also when you only update one index. There are 36 metadata columns in a fresh Plone site, and we have 59.

Our biggest organisation creates 500 too 1000 new items every week. In peak hours, there are several transactions per second. 1150 users, hundreds concurrently. Only using 30 GB of RAM. Still performant out of the box. We optimised to use a standard Plone ZODB to manage one million portal_catalog records. Optimizations for big installations will benefit smaller ones: they use the same setup. Some actions in Plone, like in z3c.form are being done two or three times, so we are looking into optimizing that.

We have configured the default Plone RAM cache to 100,000 entries. We have improved the cache so that "hot", recently used cache item will not be deleted.

We use the ZCA for low level adaptations, but we moved the adaptability to TTW configuration. This fit our audience better.

We have used in-place migration since Plone 2.1.2 until Plone 4.3.20. Most content types have been moved to Dexterity, and use GenericSetup. We will probably evaluate collective.exportimport to go to Plone 6, but will try first the in-place migration. This is in Plone and lets users feel confident.

Second site: www.deliberations.be. This brings the politics back to the people. It is a sharing tool between municipalities and citizens. One single portal to find them all.

Using Plone 5.2, Py 3, dexterity, eea.facetednavigation. This fetches data from iA.Délib through the Rest API.

Conclusions, Plone needs some optimisations, but then it works quite performant.