Weblog
Phil Lombardi - An Illustrated Guide to Microservices
Talk by Phil Lombardi at the Plone Conference 2016 in Boston.
Datawire is a first-class Python shop. Python and microservices complement each other really well.
What is a microservice? Small service, self contained, narrow in scope. It is like a lego block. Simpler: it is a unit of business logic. It could send a mail, or fill out a row in a database, anything. You wire them together by combining them in a topology, like a ring, map, star, fully connected, line. Fully connected topology is what I a focusing on in this talk. Netflix five years ago had more than 500 microservices. Twitter too.
Developers are the ones who should define the topology. Business logic is distributed, rather than central. It can help when you need to scale up, integrate various systems, or have really high reliability. If one part of the topology fails, the rest should still remain functioning as much as possible.
Linear topology lends itself for scaling and optimising each part separately: ingest data (network heavy), have one source of truth for data (disk heavy), transform data (disk and cpu), present data (network and cpu).
Microservices are about service oriented development. Thinking about architecture up front can sound nice, but when this takes months without actually building anything, you are probably out of date before you get started. An architecture should think less up front, and do more experimentation, and enable developers.
You need to move away from DNS to service discovery. And from central load balancers to smart end points, to update routing tables in real time. A microservices client should be a smart endpoint that can do the routing, that knows how to discover working and available services.
A microservice is a node in an agile service topology. And it is service oriented development. Needed mindset: from architecture to experimentation.
See the company at https://datawire.io and the microservices development kit at https://github.com/datawire/mdk
Twitter: @TheBigLombowski
Nathan Vangheem - Building a Castle
Talk by Nathan Vangheem at the Plone Conference 2016 in Boston.
I work at Wildcard, mostly on highly secure websites. I am on the Plone Security Team.
CastleCMS is an opinionated version of Plone. It packages all kinds of things up for Plone that we have been doing for security.
It is not a fork and it won't ever be. We want to work with the Plone community and want to continue to innovate with Plone. It gives us a place to innovate. Performance is important for us, and we have integrations like CloudFlare for that. ElasticSearch indexes all your content, with an asynchronous implementation of the search api. We heavily use Redis, using it as a cache that is shared between clients or threads. ZRS (Zope Replication Services) to share the load over databases. A lot of ReactJs. z3c.unconfigure for some adapters. And sometimes just monkey patches if there is no better way.
Security is integrated 2-factor authentication. Too many login attempts lock you out. The root Zope user can only login at the root of Zope, not in Plone.
Adding content is done slightly differently, and everything is Mosaic. No display menu, no default pages. All videos are compiled to a web compatible format asynchronously when uploaded. We have a Map tile for OpenStreetmap. Focal point image tile, where you can indicate what the main focus point of an image is, which is then using during image cropping. Social media tiles. All tiles are integrated with patternslib. We have a preview for different screen sizes.
We have a plugin for ElasticSearch to let the search results order be impacted by Google Analytics and social media popularity, for anonymous users.
You can audit what users have been doing. You can login as a different user. We have additional integrations with Google Analytics, archiving of content in S3, sms support, twitter. Recycle bin. Celery integration for asynchronous tasks like video conversion or pushing large files or moving lots of documents.
What is missing? Diazo (you could use it for theming, but we don't, and you don't need it for moving html content around), portlets and viewlets (just use tiles, also possibly inheriting from parent pages), default pages and display menu (we use Mosaic).
When stuff is not open sourced yet, bug me about it.
Roadmap: chat (rocketchat, ask Sam Fords about it), Mosaic enhancement, built-in A/B-testing, continue to refine the UI, more rich tiles, on the long term use plone.server.
We use React because it is better for small parts of the page. If you create a whole app, Angular2 would be better I think. We can fight.
Alexander Loechel - Plone Security in Context
Talk by Alexander Loechel at the Plone Conference 2016 in Boston.
In Europe there is the CMS Garden project: combined marketing for open source CMSes. We are partners and learn from each other.
Is Plone secure? It depends. Core is pretty secure. But security of an installation is dependent upon maintenance: if you don't apply hotfixes, it is not secure.
You can look at number of hacked sites, but security is a process, not a state. You may get a zero day export today. Are you ready for it? Are there bugfix or hotfix release processes? How do you discover those?
OWASP has a top ten report on common vulnerabilities in web sites. Plone is handling them. [Edit: alternative link, as the main OWASP list is in a PDF, is from vpnmentor. Thanks to Paola Cherlan.]
Study from BSI 2013: the vulnerabilities in Plone are in the core, mostly not the add-ons, which is different in other systems. So Plone actually protects the add-ons: you don't usually make a site insecure with an add-on. New BSI study this year, not yet published, raw number may seem not so good for Plone, but there was only one really important issue, they were looking at the fresh Plone 5.0, and most problems have meanwhile been fixed.
For most of the other CMSes you need a lot of add-ons to come to a comparable functionality as Plone, and that may be less secure: their add-ons have more problems. On my university I see hacks for wordpress and Typo3 sites every week, for Plone: none.
Plone has a different focus. It is good for intranets, and is not only a CMS, but a portal engine. Security is built in, with RestrictedPython, AccessControl. There is no SQL database, which means you avoid a whole category of problems. We have generators for add-ons, giving a secure base for adding features, so you don't make beginner's faults.
Plone's market share is not so large, so large botnets will mostly ignore us. That does not mean we are more secure, but it does help in practice. But we are used by several high value targets, like the FBI, which will normally get attacked first. Zope/Plone users are usually more aware of security.
Permissions and workflow are a real strength in Zope and Plone. An institute like BSI will give Plone at most a medium security level. Not high security, because admins can see all information. If you really would want this, you could actually build it with workflow.
In PHP, data and code are mixed, also for add-ons. In Plone, code is on the filesystem, and you cannot change it.
Sanitised input. Warning: don't use the structure keyword to display unfiltered user input. We do automatic csrf protection.
Plone does not enforce active bans of ip addresses, and security studies may complain about it missing out of the box, but you can simply use fail2ban in front of it. Use tools like that. And use good caching to avoid your site going down under an attack. There are ways outside of Plone, or any other CMS, that you can use.
The Joomla security team does a good job of communication, we could learn from that.
But other security teams often belong to one company. Often only bug fix releases, not security hotfixes. Bug fix releases may contain all kinds of small or large feature updates. Sometimes no security information is available, especially for add-ons, which is where most of the issues may be.
Never use a system 'as is'. Think about extra security you can apply in front of it. Spend fifteen minutes a day per system to maintain it.
If you have a strong security need, check out the Zope Replication Service to have a read-only front-end.
Audience: shameless promotion for Radio Free Asia. It is using Plone, and it is a constant target of attacks, and we have a clean record, no successful hacks.
Eric Steele - Integrating SQL Data into your Plone Site
Talk by Eric Steele at the Plone Conference 2016 in Boston.
It's nice to do a tech talk again, instead of always doing a keynote about the future of Plone. Sorry, I can't show you details, because I got laid off and there is a non disclosure agreement.
SQLAlchemy allows you to talk to SQL databases in a Pythonic way, getting Python objects as results.
Martijn Faassen created Traject, combining routing and traversal. Izhar Firdaus wrote collective.trajectory to do this in Plone. We use this to traverse to SQLAlchemy objects, by registering functions. We hooked the results up to plone.app.contentlisting as well, so they show up correctly in listings, like the standard tabular view.
A trajectory example project: https://github.com/esteele/example.trajectory It has add and edit forms which talk to the SQL database.
We had really a lot of tabular data, so it made sense to do this in SQL, not in a normal Plone portal type.
Brandon Rhodes - Python Web Technologies
Brandon Rhodes gives the third keynote talk at the Plone Conference 2016 in Boston.
When I met the Plone world around 2008, I saw there were lots of small companies and individual consultants. It inspired me to quit my job and university. Then the economy collapsed. Coincidence?
Movie The Social Network, 2010. When I heard about this movie, I thought it was going to be horrible. A movie about Facebook, really? But it was really good writing. One line struck me especially: "We don't even know what it is yet." Often we will get a new technology, and in retrospect have no idea what it was yet. We invent something, but we don't even know what it is yet.
The web in Python, end of the nineties. It was not much, it did not look like clean code today. The web was young. The blink tag, anyone? These two not yet mature technologies met, and mingled.
What does Python offer? What sets it apart?
- Reflection. Your program can look at itself, like in a mirror. Things like: list all functions of a class, ask the type, get an attribute by name. Not a lot of languages did that.
- Object oriented. Not just object based, allowing you to split functionality over several objects. But sub classing, multiple inheritance. You get invited to write new methods for existing classes. (And it eats a whole level of indentation, but that is another story.)
- Dynamic. You can wind up with code and objects that don't even exist in your raw source code. You can generate code on the fly, insert variables dynamically/magically. Such things might not make your code more readable though: code that does not even exist on the file system?
- Simple. Einstein: "Things should be as simple as possible, but no simpler." Dan McKinley, book Choose Boring Technology: "Every company gets about three innovation tokens." You can do about three new things without falling over. Are you going to use MongoDB? You have just used up one of your innovation tokens. Only choose new exiting technology in strategic places. For the rest: just use MySQL, Postgres, Python. Boring, but stable.
Some Python web technologies: Django, Flask, Bottle, Pyramid, Morepath. Flask and Django are an order of magnitude more popular. I created this list, found the same list somewhere else. But some techs fitting in the middle were forgotten, like web.py, web2py, CherryPy.
Flask and Django:
- Views are plain functions
- explicit registration, not reflection
- your app can start small and simple
Today, Django stands as Python's default first framework. Flask actually has more github 'stars' for people who follow it. Yet there are far more Django conferences or meetings world wide. Django gives you a good start. Years later, you may get more opinionated, with good reasons, and choose your own ORM and a different, smaller framework. As a beginner, Django just helps you a lot.
And Django has a forms library. Wait, that may involve classes. You may need inheritance and introspection. But good programmers don't avoid complexity when it helps them. Then again, a form library is a bit like backing up a truck with multiple trailers. You have knobs to control how they are going, but can you do that without failure? And you are probably missing all kinds of knobs that the forms library has not thought about. But a forms library does automatic validation, refills forms for retry, does cross site scripting, which a new web developer probably does not even think about. So it is good for them to use it.
Flask has become the go-to second framework, for when you know what you are doing.
But will we keep writing Python for the web? We have competition. Javascript is in every browser. With Node.js you even have it on the server.
Javascript is a much cleaner story from the outside. Python has 2 versus 3, C Python versus PyPy. For Javascript on the server, everyone uses Node.js, which is the PyPy of Javascript. And for ES6 there is a cross compiler to ES5.
But, you say: Python has reflection, generators, iterators, classes, modules, and Javascript has not. Wrong. ES6 adds this. You turn your back, and suddenly a language has involved. Javascript has Python features! Vi has syntax highlighting! "Everything that rises must converge." Languages grow similar.
But what about the fact that Javascript is just dumb?
40 - '30' -> 10 40 + '30' -> '4030'
In Python you get a helpful traceback when you try this, instead of running into undebuggable errors later on. Ah, but in TypeScript you get the same. They fight back successfully against the broken type system. And you can cross compile it to ES5.
So Javascript is a contender, is gaining features, and becoming safer.
There has always been another language that was more popular than Python. There is nothing new. What are Python's advantages then?
It is becoming the world's default language. Science is moving to Python. Data is moving to Python. At weather conferences there is now a Python track. Simple syntax is perfect for the occasional programmer, who has other stuff to do, like science. You do your science, and one every three weeks you write a small program. Django Girls organizes lots of weekends around the world where women use Python to write their first website. The new programmer needs Python, because it is simple.
The web and Python met when both were immature. In the years, both learned the patterns to make the web possible and sometimes even simple to program for. And even after all this years, we don't even know what it is yet.
Questions?
First language for a child? Scratch is good for young children. Python is a good language to get used to. In the US it has become the default in universities for courses.
Where at the language syntax level has Python done anything really new? When you open a file, once PyPy got popular, it got clear that files would remain open. We had to have a way to run cleanup code, even when exceptions rose. So we did try/finally, like in the Java world. It solves the problem... and it is ugly because it is Java. So we decided we needed to do one better. I believe we innovated with the with statement. It was not intuitive for me at first, but it is very helpful. It is marvellous for any kind of recursive context management, changing directories being sure that you will return to the previous directory.
In Javascript you have classes now, finally, but the Python community is far more used to it, right, which helps? Yes. Node and Javascript are still in a period of change, features have not been around for a decade. Writing in NodeJS you have spent one of your innovation tokens, writing Python you still have that one.
Find me on Twitter: @brandon_rhodes.
