Weblog
Timo Stollenwerk: The Beauty and the Beast. Modern Javascript Depelopment with AngularJS and Plone
Talk by Timo Stollenwerk at the Plone Conference 2014 in Bristol.
What is the current state of javascript in Plone, version 4.3?
Javascript was always a needed skill for Plone developers. Progressive enhancement: the site works okay without Javascript and works nicer with it. We introduced overlays and other sugar on top of Plone.
My experience the last years is that I have a small project where I use Plone and do a small bit on top. And larger projects, for which I do not use anything from the standard Plone front-end at all. If the project is large enough, this is sensible.
Lazy loading of parts of the page. You do not want to load a thousand comments at once. That can still be done with jQuery and a bit extra on top of Plone.
Problems:
- Page load is too slow.
- With jQuery you end up with ever more spaghetti code, adding bindings to buttons everywhere.
For a project we made a list of requirements:
- full feature solution
- no best of breed solution where you need to combine three or more frameworks and know them all
- good documentation and books
- What I like about Zope is: it teaches me stuff. PHP did not teach me anything. It just did not work. I wrote crappy applications or I was using a crappy framework. Or both. Then Zope and Plone came along. I would learn new stuff, knowing I was learning really useful stuff. So I wanted a good framework, well thought through.
- A framework that was here to stay for a while.
I won't do a comparison, but we chose Angular. Why?
- Full feature frontend framework.
- Developed in 2009 by Google.
- Google is using it for more than a thousand apps internally. Very popular online and in conferences.
Angular does not try to reinvent the wheel.
- Model View Controller. You have a model in Javascript and bind it to your angular template.
- Two way binding: changing the model changes the input, but also if you change an input in a browser, the model is updated.
- Keep the complex code in your Javascript, not in your template.
- Create directives that contain code for your use case, for example for easy bootstrap integration.
- Dependency injection. You write an Angular script, with a function and the function call has getUsersService in it. Then it gets this service (from a file that is already loaded) and you can use it within that function.
BTW: it is cool to edit a css or js file and have the browser reloaded automatically, with grunt or gulp. Really fast development.
Might be nice to keep the good parts of Plone, like permission management, and do the front end in Angular (or a similar framework), where Plone mostly serves up some json.
See https://github.com/tisto/plone.app.angularjs
My personal opinion is that we need a RESTful json api.
Reasons not to use Javascript:
- You might still have users without javascript.
- Search engine optimization (Google cannot crawl data that is loaded by Angular).
- May not be nice for mobile users or for old browsers.
- You need a lot of testing, because a single error may break your complete javascript.
Let's see what we can come up with as Plone community, with Angular or ReactJS or something else. I certainly recommend that you check out Angular.
Watch the video and slides of this talk.
Watch the screencasts shown in the talk: BrowserSync, Two Way Binding, and plone.app.angularjs
Matt Russell: Optimising local role security
Talk by Matt Russell at the Plone Conference 2014 in Bristol.
This talk is about experimental security indexing.
What is the problem we are trying to solve? Performing operations on the sharing tab can take very long, for large sites. There are complex hierarchies around the problem, the allowedRolesAndUsers index in the portal_catalog. Depending on the number of objects that are affected, a request can timeout or worse.
Local roles can be assigned to any user, stored in __acl_local_roles__ on the context, inherited by default.
For the Plone Intranet Project, we started working on this: Matt Hamilton, Ben Cole, and me.
Various approaches were attempted.
- Using PAS groups to represent groups of permissions or roles. You then assign users to these groups. Can become a lot of groups, that you preferably do not want to see.
- Implementing a show tree as a ZCatalog index: https://github.com/netsight/experimental.localrolesindex Abandoned, because it was too complex at a wrong level.
Current implementation: https://github.com/netsight/experimental.securityindexing
Patches the reindexObjectSecurity method for each content type. A utility stores a shadow tree, mirroring the live site. Stores a hash of the local roles. Make sure the security is only reindexed if the local roles change. Some subscribers to make sure the tree is kept up to date.
- Keep shadow tree up to date.
- Compute hash of local roles.
- Reindex the main object.
- Compute hash of local roles again.
- If the local roles are the same, do nothing more with the children.
- If applicable, do the same for the children, grandchildren, etc.
We did benchmarks, showing that this solution is a lot faster, both for Archetypes and Dexterity.
Next phase is to test it extensively on some real world scenarios and sites, and make sure it meets the workspace requirements for Plone Intranet.
Don't go into the ZMI and add local roles there, or do other stuff that Plone does not know about. Don't clear the allowedRolesAndUsers index.
The buildout extends the Plone 4.3 core development buildout and somehow the speed gain is much less; I don't know yet why that is the case.
If you have a project where you have this problem, try it out, for now on a development instance. You could also try to install the [benchmark] extra, generating benchmark-at and benchmark-dx scripts to create sample data and time it.
Watch the video of this talk.
Ramon Navarro Bosch and Rob Gietema: Resource Registries
Talk by Ramon Navarro Bosch and Rob Gietema at the Plone Conference 2014 in Bristol. Original title of talk: front-end development with Plone.
Current state
js registry, css registry, tools. Register your files in cssregistry.xml and jsregistry.xml. There are limitations. We are missing out:
- js and css have evolved. No one wanted to touch javascript a few years ago and now it is quite fun.
- dependency management was added in the javascript world
- now there is precompiling with less/sass
- versioning
- Mockup to do this in Plone.
Other tools
- RequireJS: you include require.js, add all your dependencies in javascript, asynchronous loading of files and modules.
- grunt: task runner, run tests, concat, minimize, jshint, etc. Kind of like a Makefile, but more suited for Javascript.
- npm: nodeJS server based package management tool, dependencies, versions.
- Bower: same thing on client side.
Less/sass: precompilers for css, variables, nesting, macros.
Design of the solution
- Diazo on top of static html
- bundle, resource, registry
Bundles are the main resources, loaded in dependency order. They can have js and less files that are compiled. You can enable/disable a bundle. There is a special legacy bundle with the old stuff as you are used to in current Plone 4.
Resource: defines the js, any needed export/init code, css, url. All defined in the Plone configuration registry.
Define Less variables, in the config registry record plone.lessvariables. For example define your own favorite plone-link-color.
Pattern options, for example for the pickadate widget.
Development versus production
- Compile on browser on each reload using RequireJS/Less.
- Develop static html with compiled bundles for testing
- Grunt using browser config.js/mixins.less.
Diazo
You can now say which bundles are enabled.
Views
- config.js has requireJS config.
- less-variables.js: has the standard less variables.
- plone.less: import files, patterns
- plone.js: requires the files.
This all is fully customizable through the web.
Legacy Plone: cssregistry.xml and jsregistry.xml still work. You probably want to convert it though. There are some incompatibilities: there is no ordering in the new stuff, only dependencies. So you may need to tweak a bit.
generate_gruntfile.py extracts all the information from the resource registries and creates a Gruntconfig.js file so you can test it on a static site.
A new Plone 5 site uses production settings for the theme. For development you obviously want to put it in development mode in the Site Setup. You can edit files through the web, have it compile and see the changes.
Migration
- Define resources as pattern and less file.
- Always generate production bundle.
- Move jsregistry.xml and cssregistry.xml to registry.xml.
- On/live jquery 1.11 modifications.
- Never loose hope.
See a plone.app.mosaic branch for the changes I did for migrating that package. It took me less than one hour to migrate.
More information: http://www.nathanvangheem.com/news/plone-5-resource-registries
We are going to sprint on this.
Watch the video of this talk.
Víctor Fernández de Alba and Albert Casado: Plone 5 theming unleashed
Talk by Víctor Fernández de Alba and Albert Casado at the Plone Conference 2014 in Bristol.
Plone 5 has a toolbar. On the left by default, but you can move it to the top if you want.
New markup, so less Diazo. Updated html of header, navigation, portlets, etcetera.
Remember that when styling you can use the test_renderer page to see all the standard elements of Plone, to check that all is fine.
We all like bootstrap, but use Plone classes instead, like context for buttons, which you can extend.
A designer should learn basic rules of Diazo, reuse the fine existing Plone markup, understand Plone structure, have in mind Plone default classnames, prototype a lot in in static html.
How to transfer a prototype into a Diazo theme for Plone
- Prototype based on the default Plone markup as far as possible.
- Know the limits of prototyping and know when to stop and use the real site.
- Use Diazo to move blocks around, do easy modifications to the default markup. Do not forget to make space for the toolbar just after the tag.
- Only if strictly needed, modify existing templates to match the prototype ones, using z3c.jbot.
- Use iterative improvement. Demo it to the customer frequently.
- Add custom styling for standard user generated markup in TinyMCE.
main_template is now a browser view, instead of a skins resource: Products.CMFPlone/browser/main_template.py.
Resource registries have been completely updated for modern front-end developing. Plone bundle, Barceloneta theme bundle, Legacy bundle. In your Diazo theme in manifest.cfg you can enable or disable bundles.
Development mode: compilation on browser of less/js resources. Compilation on console via grunt watch.
Barceloneta theme
- Pretty well ordered files: plonetheme > barceloneta > static > less.
- Deductible LESS structure and file naming.
- You can reuse the Barceloneta LESS files in your own theme by importing it with the sitePath variable.
- We did not use much Diazo: the rules file is very simple.
Toolbar is not completely isolated, we ran into problems using an iframe for this, it is still the plan for the future, it is now about halfway.
Maybe we can add a checkbox for developing in LESS and not Javascript and the other way around, so your browser does not need to compile everything all the time.
Twitter: @albertcasado and @sneridagh.
Daniel Jowett: Journeys with Transmogrifier & friends
Or: how not to get stuck in the Plone dark ages. Talk by Daniel Jowett at the Plone Conference 2014 in Bristol.
I have been doing software since 1997. Background in C, Java, SQL, etc. Using Plone since 2008. Came to Plone conference in Bristol in 2010 and everyone seemed to be cleverer than me. Four years later, I know that they are cleverer than me. But I have done four or five projects with transmogrifier and jsonmigrator.
Transmogrifier comes from a Calvin and Hobbes comic. Look it up. A cardboard box becomes a transmogrifier: it can turn you into anything you like: dinosaur, giant slug, whatever.
There are a few variants.
- Plone transmogrifier uses two stages: export/import from/to Plone site via xml files or csv files.
- Funnelweb: crawls and parses static sites for import, which are then pushed into Plone.
- jsonmigrator: 1 stage process that crawls json views of an old site.
- Blueprints also exist that can for example read from SQL databases.
We will focus on jsonmigrator.
When to use jsonmigrator
- Good for migrating from old versions of Plone where you cannot install transmogrifier for exporting the contents.
- Even from old Zope/CMF sites.
- Particularly when you have no buildout.
- When changing from archetypes to dexterity.
- To clean cruft from an error-prone old site.
- When scared to upgrade because you don't know what might bite you later. You don't know what you don't know...
Note: this is not the whole story: you still need to update your Plone version, add-ons, custom code, theme.
When NOT to use transmogrifier
- From Plone 4.x.
- Probably not from 3.3.6 where you already have buildout: just use the standard Plone upgrade path.
Technology stack
- plone.app.transmogrifier + collective.transmogrifier as base
- collective.jsonify
- collective.jsonmigrator
- extra pipelines:
- transmogrify.dexterity
- quintagroup.transmogrifier (I had some issues with it, but it has useful stuff, so you may need it.)
Setting up jsonmigrator
- Duplicate your old Plone in your staging environment. Or use your public live site, but that is not recommended: it may have security issues, or you may run into corner cases where your site breaks. So do it on a copy please.
- Install collective.jsonify there. If you don't have buildout, you
may need to do it like this:
- Download the egg to your Plone directory
- Unzip it
- Add it to your Python with setup.py install, which will also pull in simplejson. Use a virtualenv if possible.
- Add some external methods in an Extensions directory, where you import the main jsonify functions. Add and use the methods in the ZMI of your old Plone Site. For example the get_item function, which you can then add to the end of a url to get the json version of content item. For a file you get the data included in base64 encoding.
- Add collective.jsonmigrator to the eggs of your new Plone Site.
- Go to the @@jsonmigrator view. Select a pipeline, update any settings, and run it.
In our case we had setup a really long pipeline to migrate from a Zope 2.6 site to Plone 4.
You can define your own pipeline sections, usually based on collective.transmogrifier.sections.manipulator to tweak data in the dictionary of the current item or .inserter to add data. Start with the collective.jsonmigrator.remotesource section for getting the data.
You may have to get your hands dirty when things don't exactly work the first time. Adding some print statements can already help to get the problem clear.
Caveats
- It does not export/import users, though see collective.blueprint.usersandgroups.
- Doesn't do portlets.
- collective.jsonify is not security safe to use at a live site, as mentioned.
Credits
- Transmogrifier: Martijn Pieters, Jarn.
- Jsonmigrator: Rok Garbas
- RCS: letting me loose on this
- Calvin and Hobbes: Bill Watterson, building the first migrator.
Warning from Steve: I have run into problems with character sets. Paul: there are encoding and decoding blueprints that help, though it takes some getting.
Maarten: You are getting json data from an old site that is still running. Can't you save the json data on disk and use that? Erico: yes, you can, that is what I usually do. Maarten: I am using a blueprint for that, also because sometimes the export gave an error and sometimes the import.
Watch the video of this talk.