Plone
This is here to serve as contents for the atom/rss feed for Plone, also read by planet.plone.org.
PLOG: new plone.org work
Víctor Fernández de Alba talks about the work on the new plone.org site that is in the works.
It is on new.plone.org, but you need to login immediately. Your plone.org account does not work yet.
Does it still make sense to do it in Plone 4 instead of Plone 5? Eric: let's get it out there soon. So Plone 4.
I have been looking at lot at the theme. I am now bored by it. Do you feel the same? Should we create a more modern theme? Still lots of columns. Moving banner / carousel. Seems still good.
Paul: It should say: the community is great. Community news, sprints, big button for the news letter.
Display some technical metrics, like number of committers, pulling info from github, twitter, etc. Refresh them nightly.
Philosophy behind it? Community site, for contributors and users. Profile page for users. Reach out to existing and new community members.
Look at mozilla.org. Tell people how to contribute.
There needs to be a spot for say the Plone Foundation, where the boring bits like meeting notes get stored. So kind of a traditional site, not brochureware.
On top a bar, same as for the national Plone sites.
Dropdown menu: maybe not.
Get across that people are constantly working on it. Some activity stream?
What are main issues that still need to be done?
- Profile pages, badges, from collective.roster, integrate it. collective.workspace (for teams) can help too. Badges for being on a team.
- Design those badges.
- Big photos planned for profile page. Aren't developers too shy for that? I expect bad photo quality. People will submit photos of kangaroos. With small photo you get a lot of empty space. We may need a redesign of this page.
- Theme related issues.
- Finish contributor and community member page.
- Get it out. When it is still a future project, people will be less inclined to participate in remaining tasks. Finish critical stuff, worry about the rest later.
- PLIPs: we will do that on github.
- LDAP. Steve is our resident LDAP guru.
There is a ploneorg.core issue tracker: https://github.com/plone/ploneorg.core
Some content will be lost. This was decided earlier. Of course the product pages from PloneSoftwareCenter. We need to keep a copy of other pages somewhere, like the old World Plone Day pages. See if someone complains. We did not want to automate migrating the old mess.
All kinds of forms, for example for ordering marketing material, could be added. Currently, just mail the Plone board, they handle it. So not needed for now.
Profile is intended to be a canonical, non-judgemental list of things you have done for Plone. Might need input from Plone embassadors, as not everyone will for example answer questions on stackoverflow in English, but they may do cool things locally.
Team: Víctor, Paul, Gil, Christina, Fulvio.
PLOG Wednesday noon: backend
Our current backend stack. Plone Open Garden.
Timo Stollenwerk leads discussion about status of our current backend stack.
Gather a full report, what happens to the components we are using, put it together. Start discussion about what we can do about it.
[Note that this is not necessarily the vision of the Plone community. This is a gathering of thoughts of individuals present at the meeting. Not everyone will agree.]
- Python 2 and 3. Python 3 is pretty far away with our current stack. Several packages work on Python 3, even some of the 'scary' parts. We would need to solve other problems first.
- Zope 2 currently has no maintainer. Oh, Godefroid is release manager. People are still contributing, but not everything gets released. There are automated builds for testing. If we need something from Zope 2, we can talk to people, but we need to develop it ourselves. We use the latest Zope 2 release.
- Zope toolkit. On github. Version wise we are behind, which is not actually that bad. It is quite stable. Version updates seem to have mostly been for some reorganisation, and also getting rid of zope.app stuff. Various people still has commit rights, people do stuff when they need it. Tres Seaver and Marius Gedminas change stuff, avoiding bitrot and keeping tests running. No big plan. The Zope Foundation organisation is not really active.
- ZODB 3. We could move to 4. A Python 2 ZODB cannot be migrated to Python 3. But the package supports both.
- ZCatalog. David Glick and Hanno Schlichting worked on it. It needs Zope 2, so no one outside is using it. Technology is way beyond its limits, holding catalog data in zodb is over. ZCatalog is what makes ZODB unmarketable, becoming a bottle neck.
- CMF. Still on subversion on zope.org, which makes our continuous integration a bit unstable. Basically only one committer, who prefers not to switch to github. There are some ways to automatically sync subversion to github. We can talk to Tres Seaver as well. In CMF everything depends on everything, hard to separate. We are subclassing much of CMF. Essential is at least the portal_types tool and workflow. We are not actually using the latest release I think, which might remove the CMFDefault dependency, not sure. Custom code out there will reference core CMF stuff that is not really in the API, like underscore methods. We can at least try to rip out stuff and see what happens. Flatten the inheritance structure. Create an explorative team that takes a few hours to try things out and come up with an estimate.
- Archetypes and dexterity content types. Still supporting Archetypes, kind of, mostly some UI issues. Testing against both Archetypes and dexterity content types is tricky, brittle. If people use it and depend on it, it will be maintained. Suggest deprecating it in Plone 5.1, rip it out in 6? If people put effort in it, it is fine. If it is moved out of the core, it is dying. There is migration code for data. Plone community is smart, but we cannot handle all parts forever, we might need to drop stuff that is hardly getting used. If we want to rip it out in Plone 6, we should announce that officially. We need a roadmap for that. And find somebody who cares about Archetypes enough to keep an eye on it, that it remains working nicely. Please step up if you care, maybe we could still keep it in Plone then. Making it more smooth to move content types from Archetypes to dexterity, also in the actual Python code instead of just the data, would be very helpful. Make a script that can take an Archetypes schema and creates a dexterity schema from it, as a start. We are good at creating ideas, but bad at communicating what people would need to do, that they should move out of Archetypes. Publish it on plone.org, etcetera. Make a decision and communicate.
- Zope 4. Many options for the next step that we could take. It matters a lot where you want to go. ZODB and object model make Plone unique, flexible, very good for content management. So on long term: keep ZODB, do not move to relational database (well, RelStorage is fine). Storing json in postgres could be an option too. Writing object oriented, polymorphic code is sane with ZODB, including hierarchical data. Security on object level is awesome, not for only View permission, but other permissions, which other frameworks don't do. RestrictedPython that checks whether you have the right permissions is very nice. Some don't agree. Role based access control is great. Building products on top of Plone is very powerful. A lot of that is based on features in CMF, think of workflows, etc. But it would be good to get rid of all the layers on layers on layers. There is so much Plone. You currently need to be an expert developer to cope with this. Plone has been around for a long, long time, which is actually good. We have lots of knowledge, we need some new answers on which parts could and should really be removed, without irreparably throwing stuff away. Pyramid has the same concepts, making it very familiar, clean, without over designing things, with expressive APIs. It is not about whether we should use Pyramid, but about figuring out what the essence is that we should keep. And what small steps can be done now to make changes in the future easier. For Pyramid the right solution was to rip everything out and build the essence. This may not be the best approach for Plone. We need to stop building layers and layers and layers. We need a stronger core with the essence. The DemoStorage is really very helpful for testing, much harder to do right in relational databases. A cleanup is not very rewarding to those working on it, you need to know where you want to go, where you want to end up. Rip out stuff and move to the shiny stuff on the front-end. Think about how to cut away significant parts of the complexity. If Plone is reduced to a json store, should we want that, we can rip out much. If you know where you want to be in five years, it avoids the temptation to do intermediary stuff that you will rip out a little while later. Complex systems evolved from simple systems that already work. We cannot start from scratch. Plone has always been an innovator. Keep doing that. But also be smart to switch to the best of breed if another community has come up with something better.
- Move the discussion back to Zope 4. Several years ago Laurence did work, some lost during a move to github. It is a significant amount of work. Does it really get us where we want to be? Do we need to rebuild everything we currently have in Zope 4, or do we need to build something else, sidestepping the current code? If we have a RESTful API, getting a shiny front-end becomes easier. Makes mobile development much easier too. All web development frameworks face this problem. A full JSON api will not be there overnight. It will be hybrid. We need to keep working on our Javascript story. Pyramid has template renderers and JSON renderers, making for a clearer separation of data and presentation. We can just serve the JSON and have some more or less separate front-end show it. Plone is a CMS, not really a framework, so it differs a lot from Pyramid. Pyramid is more like Zope. In Pyramid you build your application from the inside out, with Pyramid just as a tool.
So likely directions from here:
- Reduce CMF to one layer within Plone.
- RESTful JSON api. This has the possibility of making a lot of the code irrelevant, so making it easier to remove. Makes it easier to use a different layer below for publishing.
There is no roadmap in the Javascript world, it can move everywhere, so we cannot know exactly what we would need to support or what we can count on there.
Do not forget the Python API in plone.api. Add-ons and higher level Plone layers should use this API. Still hard to get rid of the underlying stuff.
What you want in a JSON API is not a one on one rebuild of the Python API. More content centric: a JSON representation of the content.
More important to focus on where we want to be than on how we solve it. A vision is needed as a start. And more important to have a vision than to have the correct vision. Otherwise you go in circles. Do not plan on too many things that might possibly go wrong. Be bold. Do not loose vision based on what your customers want, which customers you may loose.
PLOG Wednesday morning talks
Talks, not exactly lightning, at Plone Open Garden Sorrento.
Paul: We want to have a rough roadmap at the end of the week, with something actionable. Now we start with talks.
Alex Ghica: Bucharest conference
The next big Plone conference will be in October this year, in Bucharest, capital of Romania. Five star hotel in the city center. High tower. By flight it is connected to lots of cities. Currently still cheap, about 40 euros from Amsterdam for a return flight. Sponsors are welcome. In about a month the conference tickets will be available, early bird about 360 euro.
Gil Forcada: WPOD
WPOD is: World Plone Office Day. Last Friday of the month, do not work on customer projects, but work on Plone. We should have entry-level tasks ready. Talk about it, so that people know this is happening. In each city, meet at a company's office, have a good, productive, enjoyable time with each other. Be available as mentor, team leader. Open a sprint channel on irc.
First one: April 24th.
Philip: I pledge that my company will do this every month, at least half a day, maybe a day. I challenge other companies to do the same. And be welcoming and helpful for newbies.
Guido Stevens: Plone Intranet
We have a Plone Intranet Consortium. Would that approach be doable for Plone core? It takes a long preparation, the first steps were started six years ago. It takes willingness to invest time and money. It is a business opportunity, asking investment and needing payment after a while by happy clients.
Demo of Mercury development version of Plone Intranet. Activity stream. Important sticky notes that remain visible until you click them. @mention people. Uploading for example a word document will convert it to PDF. Work spaces. Raptor as rich text editor. Wrapper around the sharing tab: three sliders to set your work space policy. Using experimental.securityindexing for permission indexing speedup. We want to do the Venus release before the summer. Workflow state transitions as milestones, use it for case management, fully flexible. Faceted SOLR search. Image bank. Render previews in the search. Personal page with personal stream. Find people by expertise. Follow people.
So that is the product. What is interesting is the process. Nine companies participate currently. Thousand euro per month, one developer day per week. Working from six or seven countries, sprinting remotely. Difficult to get traction, or on-boarding new developers. You need a few months to become productive. If you want to sell these solutions, the investment pays back. It is not just an add-on that you add to your Plone Site with a couple of others. First approach is to install it, change logo and other theming, do LDAP integration, upgrade, support it, make money that way. Second approach is to do your own design, and really think that through for a specific client. You can develop apps for it too.
Running on Plone 5 alpha 3. Dexterity content types. No z3c.form. Patternslib with the inject pattern. We keep accessibility in mind, it also works without Javascript. (Remember that accessibility is not the same as no Javascript. But some countries, at least the Netherlands, require non-javascript for certain sites.) Storing objects in btrees for performance. Lots of development goes into front-end code. Mobile-first design: working nicely on phones and tablets.
We develop it in a consortium, but the code is GPL and is donated to the Plone Foundation. Participating as a business is a risk, a possibility, and a lot of fun and opportunity to learn. If you want to participate, then make your decision. It should be a sound business decision.
It is one repository on github, but composed of multiple Python packages.
See the documentation: http://docs.ploneintranet.org
Ramon, Aleix: Machine learning for Plone content
Machine learning for text. Automatically learn from data, then predict for new content.
First step: from text to vectors. Second step: from vector to model.
Clustering: we automatically cluster documents together, so you get a list of related items, without needing to select them manually. It will allow us to create an automatic sitemap based on the clusters or topics.
Classification: text and tags. Predicting which tags would be appropriate for a newly added document.
We would love to have a Plone Machine Learning API.
- getLearningText(content)
- getLearningTags(content)
- getLearningRelation(content)
Computing the models: using scikit-learn. If you recompute at each new document, it will be slow.
Multi resolution time series, for continuously changing data, like number of visitors. Numbers instead of text. Lossy storage: less data, only the interesting info. Related work: RRDtool.
Ideas: generate taxonomy of tags.
PLOG Tuesday evening
Laurence Rowe on cache invalidation, at Plone Open Garden Sorrento.
Laurence Rowe: Cache invalidation
Sharing experience from a non-Plone project. Cache invalidation is always hard. Pages may contain snippets from other pages, from other content items. It is hard to invalidate all pages that may contain such a snippet when updating this item.
ZODB caches the objects. Zope stores the transactions. A transaction has a list of objects that are changed. Other versions of the same object can then be removed from the cache.
I wondered if I could do this for caching pages in Varnish or in Elasticsearch. I tried this in the experimental.depends package. The first simple way kind-of worked. But new items that would need to end up on the page and were never there before, did not get inserted.
I looked at the Darcs version control system, which does similar things.
A folder contents page that shows the first five items, needs to take all items into account for possible inclusion.
I have an indexing process, asynchronous, that updates the info for anything that needs to be invalidated. It leads to a materialized view of the data, allowing faceted search across a deep join.
I think this principle could be extended to Plone. It would allow indexing to a page cache like varnish, or a search cache like Elasticsearch.
Varnish helps making often visited pages fast, it does not help with the long tail of pages that are hardly ever visited.
PLOG Tuesday afternoon
Current state of various teams. Plone Open Garden Sorrento.
Kim Nguyen presents results from the Strategic Summit Survey
35 results back. Mostly: developer, solution provider, site-admin, deployer. Mostly North America and Europe, a couple from Asia, Africa, South America, Caribbean or world wide. Most have ten or more years of experience with Plone. Majority is not a Plone Foundation member. Some of you should be.
Now to some answers by those people.
What should be on the technical agenda? Nothing: this should be an open discussion of the future of the product. API. Plone 5 release. Moving away from Zope 2 and CMF. Making Plone more customizable through the web, easier to get started. Block editing, mosaic, tiles. Improving learning curve for developers, making it easier to come on board. Plone Intranet, Patternslib, Mockup, Python 3 (one), Continuous Integration. Refocus Plone as secure workflow tool, not necessarily a CMS. Replace catalog with SOLR. Better custom themes. Sidebar UI, some of the choices around this. Better mobile theming out of the box, use for mobile applications.
What should be on the non technical agenda? Marketing, visibility, promotions, formalise it, maybe professionalise? New plone.com and plone.org, get your success stories or your provider on it. Better communication from sprints, team leaders to the community and to marketing or external audiences. Get more money for the Plone Foundation. Who is the target user, client, customer? Concerns about upgrade from Plone 4. Be more welcoming to developers and other contributors. Monthly global sprints, similar to the earlier Plone TuneUps, re-energize the community. Releasing Plone 5. Better sustained documentation and consolidation, more help needed there. Consolidated bug tracking, merging into one place. Better Plone training, getting new people up to speed. Improve collaboration with Python, Pyramid, Django, Javascript communities. Decision process for PLIPs and their inclusion.
What should be the outcome of the summit? Roadmap, structured plan. Vision, better marketing. Plone 5 ship date (six months?). New community energy and synergy.
Paul Roeland and Eric Steele: summarizing complaints
- TinyMCE showing its age.
- People being spoiled by Google Docs.
- Where to do content editing.
- Scheduling content, how that works, expiring.
- Old and new style collections do different things. Difficult programmatically.
- Documentation: dexterity without Grok is not documented well, if at all.
- Sorting portlets. There is an add-on though.
- Deploying to different sites. Some add-on as well, so: notifying people what add-ons are available.
- ZCatalog takes ages, indexing, reindexing, unindexing.
- Adapters are hard to grasp as concept for new developers.
- Changing a label in a dexterity behavior for just one content type is not possible.
- Difficult to find Plone devs to hire, to train up new devs.
- Nobody wants to touch Zope and CMF parts.
- Collective add-ons, hard to get overview.
- How to get extra content on the page. Several ways.
- Plone complexity makes estimating a project complex.
- Editing complex pages. Deco, cover, mosaic.
- Concerns about migration process to Plone 5.
- Challenging to keep up with what is happening in community.
- Where are we at and where are we going.
- Clients like concrete release dates.
- What to do with the underlying stuff. More boring, but somebody needs to do that, clean it up.
- Not good as community at making decisions, which means nothing gets thrown away.
- Look at it from a business standpoint: how much would it cost to support Archetypes, how much to drop it, etcetera. Cost of current inefficiency is high, you can be bug hunting for hours. Cost of throwing lots of stuff away, is also high. Starting from scratch is very costly as well.
- Feature changes ending up in bug fix releases.
- LDAP integration is bad.
- Not updating outdated documentation.
Eric Steele: Plone 5
There is a buildout for the first beta. Mockup has had issues with frequent rewrites. Control panels have now been modernized, using z3c.form and the configuration registry. plone.app.contenttypes is in there. Migrations for standard content types is working, for custom content types it is there but needs more love and/or money; you still have to write your own content types, views, templates of course. Dexterity is default content type story. New Barceloneta theme. Improved folder contents tab.
We have completely rewriting the way we use Javascript in Plone. Using more modern techniques, like RequireJS. A new Resource Registries panel in the Plone UI to edit this. You can edit less variables through the web, kind of like the old base_properties. We are now able to use more modern widgets, without needing to maintain outdated ones ourselves.
When will it be out? The more you test it and report bugs, the earlier we can ship it.
Eric and Paul: Community and communication
We have semi-regular team leader meetings. We are putting notes up on http://community.plone.org. There are lots of duties to take on, certainly also for non-core developers. You are very welcome.
Documentation is in reorganisation. Restructuring documentation that was already there. The documentation team is not the one to write a doc about say creating a dexterity content type without Grok. Someone else will need to write it, and we can then help getting it more understandable, in better English, etcetera. You do not need to be good at writing, you just need to write.
A lot of the teams are currently one person. We need to do a better job of getting more people in. Some people are in multiple teams, overcommitted, like Timo and Ramon.
Víctor Fernández de Alba is working on new plone.org, mostly on his own. Migrating some of the content using transmogrifier. Using Plone 4, with plone.app.contenttypes. Documents, events, news, images, files are migrated, PloneSoftwareCenter will not be used anymore, so that content will not be migrated. We wanted to have the PLIP process on plone.org, so we no longer need dev.plone.org. Maybe profile page for Plone contributors, listing which teams they belong to, other information. Maybe we want to rework the theme, change the landing page. Should be discussed, maybe during these days. Package is plone.org.core. You can see on github what is left and what is done.
Paul: dev.plone.org is now spam-free again, mainly thanks to Chrissy Wainwright. Down to about 600 tickets to export and import to github. There is a way of moving tickets from one github repo to another, and a way of creating a ticket in an initial spot when you do not know where it should be reported. Needs a champion, someone to do it and/or slave drive others into doing it. Are you good at scripting, with at least minimal people skills, please step forward. Even better: a group of people.
Eric: https://community.plone.org is being used. We need to make the decision to flip the switch to make this the main forum. Yes, we will switch it this Friday. Not the developer list though.
Paul: http://paragon.plone.org is a site for searching the very best Plone add-ons. There is a jury. Work still needs to be done. Also: get your setup.py classifiers correct so they mention Plone and the Plone version, and they will show up in a list here as well.
Kim Nguyen: http://plone.com is live, soft-released. We used an issue tracker and systematically used it, filed and assigned tickets. We had weekly meetings to keep on track and to keep us honest. It is so important, it re-energizes people. Please step up in a project, help a little bit. [Christian Theune: Too many people working on their own: create a bigger team again, so people do not get stuck in their own little corner. It helped us a lot.] We want more success stories. We want to hear from you if you think the message needs more refinement. To get your provider up on plone.org, contact us and become a sponsor. It is a small amount, we want it to be affordable also if you are in an area of the world where you do not make a lot of money. The goal is not to get a lot of money from this, just to get an up to date list. Translations: we will discuss it on Thursday. If you want to be a part of plone.com as your country, contact us. Also fine to have your own site, like the Dutch http://plone.nl
Paul: documentation is there, but it is mostly for Plone 4. Plone 5 is getting closer, so this needs updating. Looking into getting robot tests for screen shots. That is actually easy to add in your code when you have something that is worthy of a screen shot. We will be contacting you about that,
Christina McNeill: one-person communication team. Plone Newsletter has been there for a year. About 160 subscribers. 80 percent is actually reading it, which is really high according to mailchimp. Marketing is not really my area of expertise, so we can use someone there. [Philip: please add a nicely visible button on the plone.org homepage to subscribe.]
Philip Bauer: David Glick and Tres Seaver have recently been working on getting Zope parts to run on Python 3. Is there a master plan? Eric: no.
Asko Soukka: Mosaic. We are working on it. It may be doing too much. May be hard for people to decide whether to use something in Mosaic versus Diazo. Lots of things in tiles are broken for lots of reasons, needing work. Many concepts. Store site-layouts as content types. Needs specific versions of plone.app.widgets and plone.app.events. There is a PLIP configuration to run it on Plone 5 in the core development buildout. Ramon: it works fine, but I am not trying to finish it for production. Philip: Mosaic should be a priority for us, let's not wait until our javascript story is completely finished. Crowd: do not try to do this on Plone 4 anymore, only on Plone 5. Asko: hide all tiles that do not work. Eric: everyone help out on sprints, donate time, get a customer to build this for.