Lessons from integrating Pyramid, React, and Plone (and elasticsearch, and Salesforce, and a payment gateway, and...) Quoting from the Plone conference site: "The Volunteer Management System (VMS) is a tool that Jazkarta built for Washington Trails Association to manage volunteer sign-ups for helping maintain hiking trails in the state of Washington. It allows staff to schedule work parties, volunteers to find and register for them, crew leaders to access information about their crew, and land managers to report on work that was done in their region."

The tools you use matter. So use the correct tools.

So at the start, we had the wta.org site, and Salesforce, and vols.wta.org, which was basically a bunch of ten year old scripts.

We could have built everything in Salesforce. It has all kinds of options for doing this, but there are all kinds of restrictions.

We considered Plone. All user accounts were already in here. But we did not need a lot of content management.

On the front-end we used ReactJS. This communicates with Plone for content and with Pyramid for the volunteer management. Pyramid interfaces with Redis, ElasticSearch and Salesforce.

With Pyramid, the view configuration is very close to the actual code, so you don't need a separate zcml file like in Plone. There are no globals, which makes it easier to write isolated tests. Views are easier to test because they return a dictionary instead of a complete html response.

Pyramid is a framework that is good for writing frameworks. We needed to write some infrastructure ourselves, which is nice to do in Pyramid. For example, with config.add_request_method we made a method available on the request to access Salesforce. We registered an exception view to gracefully handle exceptions for either html or json.

With Pyramid, batteries are available for cheap:

• pyramid_chameleon
• pyramid_layout
• pyramid_cachebust
• pyramid_mailer

As special deployment bonus: we were not using a ZODB, so no ZODB cache was needed, so memory requirements were low, so we could use lots of threads to handle moments of high load.

It is not perfect, for example there is no automatic csrf protection like in Plone. Some assembly is required: Plone has lots built-in and in Pyramid you need to do more yourself, for example setting up logging.

We used the py.test framework, which makes it really easy to define fixtures. There is some magic, which not everyone likes. Assertions are easier to debug, giving you more feedback. We used factory_boy to provide fixtures. [See a talk at PyGrunn earlier this year, by Oleg Pidsanyi. Maurits]

We used http://locust.io for load testing.

React was our chosen Javascript framework. I like that it is declarative. You can focus on what is displayed, not how. A lot of changes, like toggling a checkbox, happen automatically.

We have shared navigation between Plone and Pyramid. The navigation is done in Plone, and we created a json endpoint in Plone that Pyramid could call.

We needed to think about Cross Origin Resource Sharing (CORS), with all kinds of headers. These days I would look at plone.rest and how it does this.

Single sign-on. Pyramid redirects you to Plone, where you can login, and with a token you are then logged in to Pyramid.

I am in Berlin via a live video feed. Hello!

In several recent projects with Plone I had a common challenge. The CMS was just one of many parts of the stack. People may view our content on a large computer screen, a mobile phone, or virtual reality glasses, which all need to be handled differently. There are several solutions, like contentful. Some common traits:

• Everything is through the web.
• APIs for various languages

Why was I using Plone for that?

• No more custom PHP development please.
• Lots of things included.
• I am still quite productive with Plone.

Biggest reason:

• I did not look for hosted solutions.

If I build it in Plone, I may be the only one at the company who can maintain it. Not so with a hosted solution.

But: I still love Plone, and want to use it, for good reasons:

• One API endpoint.
• Better control over performance issues.
• Integration with front-end development is essential.

And now I actually have four Plone developers in the team, so that is great. The team is happy with Plone. We use as building blocks:

• plone.restapi. This saved a lot of time.
• plone.app.multilingual
• a policy package
• PyCharm IDE for writing Python
• AWS/Kubernetes/Deis
• Thumbor. Image server over Tornado. Cropping, filters, transforms. We have integrated this in Plone using event subscribers.

Lessons for Plone:

• documentation, documentation, documentation. Everything I did was possible due to documentation.
• More documentation, more documentation, more documentation.
• We need more projects like plone.client.
• We need more commercial offering of headless hosted Plone.

Docker is what virtualenv is for Python. Isolated environment. You have the same environment on Linux/Mac/Windows and development/test/production.

For Plone without Docker you need to install all kinds of libraries, run a buildout, ignore some SyntaxErrors.

For Plone with Docker you can run:

docker run -p 8080:8080 plone

This uses the new official Plone Docker image at https://hub.docker.com/_/plone See information on that page, also on including add-ons or developing. In that case buildout gets run so it can fetch the add-ons.

Always label your Docker volumes to avoid data loss.

You can use a zeoserver:

docker run --name=zeo plone zeoserver

In production you should create your own Docker images, including all add-ons you need. You can use extended_buildout.cfg. For composing your own Dockers, see https://docs.docker.com/compose

When using multiple hosts, you want orchestration. There are several options. We use Rancher. You can use the official Plone Docker image and set several options.

The plan is to use this for continuous delivery, like in the talk by Nejc Zupan.

See various repositories with Docker for the EEA.

See https://github.com/plone/plone.docker for the source of the Plone Docker image.

Discussions in the hallway are the best part of this conference. You don't have slides there, so I won't use them here either. I do have them by the way.

Have you deployed anything to production this month? Week? Today? If code only sits on your local computer, it does not give value to your customer.

What would deploying look like if it was easy? You fix a big, push the change, the tests are run, they pass, production gets updated automatically or maybe after one push of the button. Instead of saying 'the fix has been committed and will be in the next release', you want to say 'it is fixed'.

The bigger the change, the harder it is to deploy. One fix, with one data migration, is fast to deploy, and easy to undo if needed. With three fixes: it gets harder. For one fix, do you really need a developer, a QA person, a project manager, a sysadmin?

When you push to a git repository, you can hook up actions to start testing and deploying.

Tools are interchangeable. You are smart people, you can figure out how to do this. More important is why you should do this.

When a pull request is made, we copy production data to staging. This may be a lot of data, so in case of SQL database we only copy a subset. The pull request is deployed to staging, so a reviewer can immediately check it there. We have 100 percent test coverage. We have regression tests. For new employees, I want them to deploy something to production the first day, going through the whole process. This is very empowering. It keeps developers happy.

Code ownership. If you know that your pull request may be deployed within the next twenty minutes, then you will be careful about what you put in the pull request.

Tip: if you don't have this yet, then start doing this with a very small project.

Give your customers access to the staging machine that has this change, so they can check it.

We host everything on Heroku. For the highest paying levels, you have a button to revert the code change and the database change. Heroku does this well. If you are not in the business of hosting, then don't do hosting.

We have about ten people in the company. We do have a few small Plone Sites.

For us, the only way to get code into the master branch, is via pull requests.

We spend more money on the staging servers than on production.

You can automate stuff like: if a new tag of a package is generated, push it to your internal PyPI.

What about big features instead of bug fixes? You can use feature flagging, showing for example a new UI to only a few people. But anyway, it is rolled out to staging, you test it, if it works you can deploy it. For big features it can take longer to get everything correct, so you just don't merge the pull request yet.

I will never write code without writing a test. For an emergency fix, it is still better to spend twenty minutes to write a test, instead of spending two hours later reverting and debugging.

Disclaimer: I do not call myself a super integrator, but others do, and I humbly accept it.

Three years ago I had not heard about Plone, though I had used various other CMS packages. Now I work at Penn State university doing websites, particularly Plone websites, moving static content into Plone. When I started, there were 120 websites, and other a few months my only other colleague left. We got other people into the team. We support 23 departments, 10 programs, 16 centers and institutes, and more. Not all websites are Plone, but about forty to sixty are.

I was showing a site to Eric Steele, and he looked like: you do this through the web? Yes, I do. Theming, Diazo, works fine.

I learned through trial and error. It builds character... and eats up time.

If it exists, don't recreate it. Take an existing solution and improve on it.

I am using Diazo to it's fullest. Using xpath and xsl to keep original classes and ids, so that an editor who is looking at a page source can still see that a part of the page is for example a portlet, so that he knows where to start editing it.

Some stuff I need to do in page templates.

If I need a similar block three times, then I make it reusable.

Portal types. We were using News Items for slider images. That is hard for editors to remember, they think: I want an image, so I can't use a news item. For those cases I simply copy the News Item type and give it a name that makes sense to them, like Home Images.

I talked to Cris Ewing last year about how I did all this through the web, and I saw his enthusiasm, so I knew: I am not crazy to do everything through the web.

I want my end users to feel empowered. It is their website.

Questions?

What would make my life as integrator better? Plone 5 actually solves several things. It would help with caching of our custom css, and defining the colors for css in the resource registries. Debugging diazo could maybe be easier, saying clearer what the error is. Diazo snippets would be nice. David: there is a diazo snippets library, available as chrome extension.

If your Diazo rules are slow, make your rules more specific.

Our policy is to train the users. If they don't get training, they don't get an account. And I say: if you mess things up, like accidentally removing an entire folder, don't try to fix it yourself, but immediately call us. Not an email, call us. We have good relations with our users.