Weblog
Nathan Van Gheem: Introduction to Python Asyncio
Talk by Nathan Van Gheem at the Plone Conference 2017 in Barcelona.
This is about the Python 3 core asyncio library. "This module provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives."
The first time I read that I was like: what?
Asynchronous programming using async and await syntax. Any network activity should not block other code, that is the main idea. This is useful because web applications use TCP sockets. It is a way to improve performance and scale web applications. Also think of microservices.
The optimised event loop allows you to handle a larger number of requests per second. You can have long running requests with very little performance impact. With standard Plone that is impossible.
Requirements: Python 3.4.
How are typical web servers designed like Flask, and Django? Each request is tied to a thread, so you are limited to handling number of threads and processes you run. Threads are expensive (GIL, context switching, CPU). If no threads are available, further requests are blocked, waiting for an open thread. Threads are blocked by network traffic, for example to a database server.
With asyncio, requests can be tied to tasks. You can have lots of tasks per thread, and if a task needs to wait for network traffic, it does not hurt you. But be careful: if anywhere in your code you use the requests library instead of asyncio, that will block your network traffic.
We have Futures`. ``asyncio.run_until_complete with ensure_future wraps your asynchronous call in a Future object.
You can have long running Tasks. Tasks, futures and coroutines are very similar, in the beginning you don't need to worry about that.
Gotcha: everything must be async. Async functions need to be run by the event loop. If you run it manually, it will not do anything. If you don't call an async function using await it will never be run either.
asyncio is single threaded: only one event loop can run in a thread at a time. Running multi threaded code in asyncio is unsafe. You can have multiple threads, each having their own event loop. You can get the feel of multiprocessing by using asyncio.gather
With an 'executor' you can make synchronous code asynchronous. Typically it is a thread executor. Try to avoid it, but it is a tool that you can use if needed. See concurrent.futures.
asyncio comes with an amazing subprocess module, so you can await the result of executing a command on the terminal.
The event loop is pluggable, for example tokio.
More and more libraries are popping up using asyncio:
- aiohttp: client and server library
- aioes for elastic search
- asyncpg for postgres
- aioredis
- aiobotocore
- aiosmtpd for smtp
[See https://github.com/aio-libs for more.]
Debugging is more difficult than regulare sequential programs, the pdb is tricky. aioconsole allows you to have a Python prompt with an asyncio loop already setup for you.
guillotina uses asyncio.
In Python 3.7 you have an execution context, which is going to be nice.
Questions and answers:
- You cannot do WSGI with asyncio. But Tornado uses asyncio.
- What was hardest? Wrapping your head around it all.
- Is this only for network calls? Or also useful for disk access? There is an add-on for that. I tried it and then it was kind of a hack.
- Do you have profiler tools, like seeing if code is blocking too long? See an earlier talk. There is `aiomonitor <https://github.com/aio-libs/aiomonitor>`_.
Twitter: @vangheezy
Bert JW Regeer: I broke what? Taking over maintenance on existing (well loved) projects.
Talk by Bert JW Regeer at the Plone Conference 2017 in Barcelona.
Existing code needs love too! Look at the truth behind open source apps on commitstrip. You can help open source projects by becoming a maintainer. I became maintainer of WebOb.
What I was told: don't mess it up. There was no existing maintainer at the time. It was handed over to the Pylons project for maintenance, but no single person was responsible for it. They all stepped back.
Side-track: imposter syndrome. See Denys Mishunov's talk yesterday. Usually you get extra responsibility gradually: for example first commit rights, then release rights. You may think you are not good enough for extra responsibility, but probably you are.
You have all these nice ideas and good intentions. You push out some alphas and betas, all seems okay. You make a full release. Then a bug report comes in because you removed something that you did not expect anyone to use.
Code grows over time. All kinds of code is there because it fixed an actual problem for an actual user. So you are faced with backwards compatibility. How much of it do you do? It depends on whether you are using a library or a tool. Libraries need to maintain more backwards compatibility. For a command line tool, the only API is the tool, not its internals.
Can you afford to lose your users? Someone can fork your code and maybe rename it and create an alternative tool.
Testing: if you have 100 percent test coverage, and you change something and a test breaks, then you can more easily see what is wrong. Does the test simply need to be rewritten for the new situation? Or is the test breakage a hint that something will break for a user, letting an old bug resurface?
You sometimes have to make a breaking change. Give a deprecation warning then.
Joel Spolsky: 'Single worst strategic mistake: rewrite the code from scratch.' This was about NetScape 6, and allowed Internet Explorer to catch up and take over. It is an option, but probably not the best.
So. You took over. You are now the gate keeper. You are a temporary guardian. Eventually someone else is going to take over. You should start looking at mentoring opportunities. Find ways to engage others, engage the community. Create pull requests instead of pushing to master.
Reach out to other communities, consumers of your code. Can you help them?
People may ask or tell you to just accept this pull request. Just push out a new version. Just is a bad word. Push back and insist on them following the standards of your project. If you require 100 percent test coverage, don't review the pull request.
I have received bad bug reports, so now I myself write better bug reports. I push myself to do better. Maybe even try to provide a fix for a bug upstream.
Be friendly when someone does crazy or seemingly stupid things in a pull request. A good question to get clarity is: 'What are you trying to accomplish?'
Twitter: @bertjwregeer.
Mark Pieszak: Rendering JavaScript on the Server? Welcome to Angular Universal.
Keynote talk by Mark Pieszak at the Plone Conference 2017 in Barcelona.
[Sorry Mark, I came in late.]
SSR = server side rendering
Create an app.module.ts and an app.server.module.ts
- Static SSR is done at build time.
- Dynamic SSR is done at run time.
SSR gotchas:
- If you use window or document, the server does not know what to do: this only lives in the browser. If you must, create a WindowService and use dependency injection to provide different versions. Use isPlatformBrowser() as much as possible. Hide things from node. Not all parts need a server version.
- Be careful with timeouts, because they will let your server wait.
Conclusion:
- Universal makes SEO possible.
- Universal gives really fast initial painting of your app, and you keep the interactivity. Can be two to three times faster on mobile.
- Be mindful of browser-specific things you might be using in your code.
- Choose third party libraries carefully, as they need to be mindful of the pitfalls as well.
- It takes a bit of work, but it is worth it
Further reading:
- Code: https://github.com/angular/universal
- Twitter: @MarkPieszak, which will get a link to the slides soon.
Annual membership meeting Plone Foundation
Annual membership meeting of the Plone Foundation at the Plone Conference 2017 in Barcelona.
Plone Foundation president Paul Roeland presents the report of the past board year. Documents can be found on plone.org.
There were lots of sprints, most of them sponsored by the Foundation.
Eric Steele was interviewed by podcast init, which you should listen to.
At http://smile.amazon.com you can by stuff from Amazon and have Amazon give a percentage to the Plone Foundation, at no cost to you.
Sad is that long time Python organiser Jean Ferri from Brasil passed away.
Financials. Summary: we are doing fine, and can afford to spend a bit more. We would like more sponsorships, like providers on plone.org.
The entire current board nominate themselves for the new board, and there are unfortunately no other candidates, so we can have an easy vote.
Erico motions to approve the candidates. Maurits seconds this motion. Philip and Alexander abstain. Otherwise everyone says aye. The old board is hereby the new board.
Erico: Is the Foundation supporting bitcoin donations? The Free Software Foundation does.
Paul: Not at the moment. Depends on our bank account.
Matthew: Does the Foundation have an environmental policy?
Paul: We have recommendations. For this conference it does not really work, also because we have a caterer.
Paul: We may want to open up the Plone Foundation to family and friends, like guillotina and Pyramid. Pyramid lacks a legal framework currently. This needs careful reflection before we do anything, but the board is initially open and positive to it.
Paul: We are sometimes in difficult discussion with the Zope Foundation, which does not technically exist anymore, linked to the Zope Corporation, which technically does not exist. So the situation is unclear. We are working on it.
Philip: At Zope sprint in Halle there was consensus to unwind the Zope Foundation and incorporate everything in the Plone Foundation.
Alexander motions to adjourn the meeting and go party. Fred seconds. All say aye.
Thank you and have a great party. Party responsibly and be there tomorrow at nine for the keynote speaker.
Oh, there were two proposals for the next Plone conference. The board did due diligence and found that only one was viable. It will be announced tomorrow.
Lightning talks Thursday
Lightning talks on Thursday at the Plone Conference 2017 in Barcelona.
Andreas Jung: Plone and the blockchain
Blockchain is the base technology behind bitcoin, but it is not bound to crypto currencies. It is a distributed data structure, usually based on peer to peer. No central entity of control.
Each block has a hash of its previous block, timestamp, transaction root, and a nonce.
Use cases: auditing, financial transactions, logistics, QA, legal, automotive, others.
What does this have to do with Plone and CMS? Some ideas:
- revision safety
- audit trail
- verification of content integrity and authenticity
Our use case: collaborative editing environment. So we created SmashDocs. Using Plone and BigChainDb.
Erico Andrei: Websauna
Websauna is a web framework based on Pyramid. https://tokenmarket.net is created with that, also with blockchain BTW. Miko says hi!
I am using it too, and helping him. We want you on board as well, and we can sprint on it. Also we want to improve Pyramid. Move Websauna to Pyramid 1.9. Documentation. User testing.
See https://websauna.org.
Alexander, Anton: Ploneconf and PyconWeb
Why don't we make a PyconWeb conference? We did that last year. Next one starting on 9 June 1918. See you at https://pyconweb.com
Fred van Dijk: Music to your ears
Confession: I am a bit of an audiophile. When I talk with people about how they listen to music, I get sad. Lot of people use a ten euro headphone. "I don't hear a difference with more expensive ones." I am convinced that a better audio setup helps you work better.
What is the weakest link? Music source (up the settings), D/A converter (underestimated component), cables (spend twenty euros, that's the sweet spot), headphones (ten euro and you expect quality?).
If you divide the costs over the number of hours you listen music: I came at six cents an hour.
Especially the D/A converter (USB) really helped me.
Nejc Zupan: A few Pyramid goodies by Niteo
- pyramid_force_https
- pyramid_redirect
- pyramid_heroku
Releases are on PyPI, enjoy!
Manuel Reinhardt: Giesing 2060
Science fiction writing project about an area of Munich in the year 2060. Using Plone, two content types, snippets that give links to other story snippets, different story lines, you can read through it in various ways.
If you like reading or writing science fiction or Python code, or both, have a look at http://giesing2060.de
Alexander Pilz: Ten years of Euphorie
A Plone success story.
This is a software to guide employers and employees for mandatory health and safety risk assessment. In 2007 Euphorie was created by Wichert Akkerman and Cornelis Kolbach, with the NuPlone interface (currently still working on Plone 5 actually). In 2008 adopted by Europe. In 2016 interest by an industry client of ours.
Why was this successful with Plone? Customisation and enhancement was made easy. Good security. Open source. Now they no longer need to cary kilos of paperwork to factories.
Jens Klein: Alpine City Sprint
I invite you to come to Innsbruck to work with us on the next Plone. Today on an open space we discussed that we may be using Zope 4 on Plone 5.2, and we can work on that.
Welcome January 2018 in Austria. We always visit a special place as well, now a space lab, with simulation for Mars missions.