plone / plone.app.upgrade Goto Github PK

View Code? Open in Web Editor NEW

8.0 154.0 20.0 4.56 MB

Database upgrade steps for the Plone CMS

Python 100.00%

plone.app.upgrade's Introduction

Overview

This package contains the upgrade machinery to upgrade a Plone site to a newer version.

Version compatibility

To update to Plone 4.x please use plone.app.upgrade versions up to 1.3.x.
To update to Plone 5.x, use plone.app.upgrade version 2.x.
To update to Plone 6.x, use plone.app.upgrade version 3.x.

plone.app.upgrade's People

Contributors

Stargazers

Watchers

Forkers

toutpt lentinj kiorky makinacorpus starzel khink eaudeweb gforcada gbastien ichim-david jcerjak tkimnguyen martronic-sa b0n jazkarta pabo3000 topiaruss tuticapi rber474

plone.app.upgrade's Issues

problem with plone.base.interfaces.syndication.ISiteSyndicationSettings when upgrading from 5.x to 6

While upgrading some Plone sites (5.2.6 and 5.2.7), I encountered an issue with the ISiteSyndicationSettings interface on all of them.

The problem seems to arise from the fact that the interface is still located inside Products.CMFPlone but points to plone.base.interfaces.syndication.

Therefore, attempting to access the Syndication control panel results in the error message:

"KeyError: 'Interface plone.base.interfaces.syndication.ISiteSyndicationSettings defines a field allowed, for which there is no record.'"

I also tried installing a vanilla Plone 5.2.7 and upgrading it to Plone 6.0.3; I still get the same error.
I'm using plone.app.upgrade 3.0.3

plone.api dependency has slipped in [5.2]

In move_dotted_to_named_behaviors I see from plone import api. This is not allowed in core code.
I noticed this when looking at something else.

WrongType error on checkout_workflow_policy, type unicode

When I do an upgrade from 4.3.16 to 5.0.8 I get the following error:

2017-10-24 09:14:07 ERROR plone.app.upgrade Upgrade aborted. Error:
Traceback (most recent call last):
  File "/usr/local/plone-5.0/buildout-cache/eggs/Products.CMFPlone-5.0.8-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 268, in upgrade
    step['step'].doStep(setup)
  File "/usr/local/plone-5.0/buildout-cache/eggs/Products.GenericSetup-1.8.8-py2.7.egg/Products/GenericSetup/upgrade.py", line 166, in doStep
    self.handler(tool)
  File "/usr/local/plone-5.0/buildout-cache/eggs/plone.app.upgrade-2.0.8-py2.7.egg/plone/app/upgrade/v50/betas.py", line 573, in to50rc3
    settings.checkout_workflow_policy = str(value)
  File "/usr/local/plone-5.0/buildout-cache/eggs/plone.registry-1.0.5-py2.7.egg/plone/registry/recordsproxy.py", line 49, in __setattr__
    self.__registry__[full_name] = value
  File "/usr/local/plone-5.0/buildout-cache/eggs/plone.registry-1.0.5-py2.7.egg/plone/registry/registry.py", line 47, in __setitem__
    self.records[name].value = value
  File "/usr/local/plone-5.0/buildout-cache/eggs/plone.registry-1.0.5-py2.7.egg/plone/registry/record.py", line 82, in _set_value
    field.validate(value)
  File "/usr/local/plone-5.0/buildout-cache/eggs/zope.schema-4.4.2-py2.7.egg/zope/schema/_bootstrapfields.py", line 183, in validate
    self._validate(value)
  File "/usr/local/plone-5.0/buildout-cache/eggs/zope.schema-4.4.2-py2.7.egg/zope/schema/_bootstrapfields.py", line 310, in _validate
    super(MinMaxLen, self)._validate(value)
  File "/usr/local/plone-5.0/buildout-cache/eggs/zope.schema-4.4.2-py2.7.egg/zope/schema/_bootstrapfields.py", line 210, in _validate
    raise WrongType(value, self._type, self.__name__)
WrongType: ('checkout_workflow_policy', <type 'unicode'>, 'value')

Do not fail if we do not have archetypes

When we do not have archetypes, the upgrade profile 4beta3-4beta4 with the error:

ValueError: undefined property 'content_meta_type'

plone/Products.CMFPlone#2555

Relation upgrade code in 5.2 beta fails

When I migrate a site from Plone 5.1.5 to 5.2.0, I get an error in the remove_interface_indexes_from_relations_catalog upgrade step:

2019-11-25 15:46:47,096 INFO    [plone.app.upgrade:29][waitress] Added BooleanIndex for field exclude_from_nav.
2019-11-25 15:46:47,096 INFO    [plone.app.upgrade:31][waitress] Indexing new indexes exclude_from_nav.
2019-11-25 15:46:47,616 INFO    [plone.app.upgrade:395][waitress] Cleaned up the toolset registry.
2019-11-25 15:46:47,631 ERROR   [plone.app.upgrade:298][waitress] Upgrade aborted. Error:
Traceback (most recent call last):
  File "/Users/maurits/shared-eggs/cp27m/Products.CMFPlone-5.2.0-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 291, in upgrade
    step['step'].doStep(setup)
  File "/Users/maurits/shared-eggs/cp27m/Products.GenericSetup-2.0-py2.7.egg/Products/GenericSetup/upgrade.py", line 168, in doStep
    self.handler(tool)
  File "/Users/maurits/shared-eggs/cp27m/plone.app.upgrade-2.0.27-py2.7.egg/plone/app/upgrade/v52/betas.py", line 110, in to52beta1
    remove_interface_indexes_from_relations_catalog()
  File "/Users/maurits/shared-eggs/cp27m/plone.app.upgrade-2.0.27-py2.7.egg/plone/app/upgrade/v52/betas.py", line 72, in remove_interface_indexes_from_relations_catalog
    catalog.unindex(rel)
  File "/Users/maurits/shared-eggs/cp27m/zc.relation-1.1.post2-py2.7.egg/zc/relation/catalog.py", line 564, in unindex
    self.unindex_doc(self._relTools['dump'](rel, self, {}))
  File "/Users/maurits/shared-eggs/cp27m/z3c.relationfield-0.9.0-py2.7.egg/z3c/relationfield/index.py", line 42, in dump
    return intids.getId(obj)
  File "/Users/maurits/shared-eggs/cp27m/five.intid-1.2.3-py2.7.egg/five/intid/intid.py", line 41, in getId
    return z3IntIds.getId(self, ob)
  File "/Users/maurits/shared-eggs/cp27m/zope.intid-4.3.0-py2.7.egg/zope/intid/__init__.py", line 108, in getId
    raise IntIdMissingError(ob)
IntIdMissingError: <z3c.relationfield.relation.RelationValue object at 0x10cadf5f0 oid 0x67a7 in <Connection at 108b08290>>
2019-11-25 15:46:47,635 INFO    [plone.app.upgrade:307][waitress] End of upgrade path, main migration has finished.
2019-11-25 15:46:47,637 ERROR   [plone.app.upgrade:310][waitress] The upgrade path did NOT reach current version.
2019-11-25 15:46:47,637 ERROR   [plone.app.upgrade:311][waitress] Migration has failed

The upgrade catches a KeyError around catalog.index(rel) and accepts this because it means a broken relation is removed. But for me the KeyError happens two lines earlier, in catalog.unindex(rel).

In my case, this can happen for two reasons:

Some relations are so broken that both from_object and to_object are None. So it is a relation between nothing and nothing... This relation could be safely removed, it seems.
Relations themselves apparently also have their own intid, but not all of them.

While investigating this, I have created a script that I can run on Plone 5.1 to fix it. Then the upgrade to 5.2 goes fine.
I will extend the existing upgrade step with this code.

upgrade steps for HTMLFilter defaults (5.1, 5.2, 6.0)

dependent PRs:

plone/Products.CMFPlone#3102
plone/Products.CMFPlone#3103
plone/Products.CMFPlone#3104

Add upgrade step for Plone 4.3.19

Skin layers with sub directories get removed during upgrades

I just upgraded a site from Plone 4.3.6 to 4.3.11. In portal_skins on the properties tab I had captcha_core/static as layer in one of the skin selections, from quintagroup.captcha.core. After the upgrade it was removed.

I have not checked yet, but I expect that this happens in cleanUpSkinsTool which is called during several migrations.

Implement an upgrade step for Plone 6.1

some registry entries should be upgraded via upgrade step if a site is upgraded to Plone 6.1. the interfaces ITinyMCEPluginSchema and IFilterSchema contains new values.

Add upgrade step to migrate markdown tranform settings to markup control panel

Add upgrade step so existing settings are migrated.

Upgrade Plone 5.2.10 to 6.0.0b3 does not keep site root's blocks and blocks_layout

I don't know if this is the right place to discuss this or the place should be plone.volto

We have a Plone 5.2.10 site as a backend for a Volto 16 app.

I have migrated Plone to 6.0.0b3 and I have seen that although the Plone site has been migrated into a Dexterity object, and although the old root having the blocks and blocks_layout properties so that a Volto page can be configured in the root of the site, the migration step did not take this into account.

IMHO the migration code should have taken into account that the site was been used as a Volto site (the blocks and blocks_layout properties on the root of the site signal them), and if so enable the Blocks layout on the Plone Site and migrated the content of the said properties to the proper behavior-provided fields.

Opinions @pbauer @mauritsvanrees @jensens ?

Add empty profile for 5.2a2

Remove not really needed Configlets

Remove Member Password and Member Prefs Configlets from Controlpanel
see #3031

Add empty upgrade for Plone 5.2rc5

The members info in MemberDataTool needs to be rebuilt

In the upgrade to 5.2, the class for MemberData records has changed. The items stored in portal_memberdata._members cannot be loaded any more.
See plone/Products.CMFPlone#2722

Plone Syndication registry entries not migrated to Plone 6

I have a Plone Site that was created in 5.2 and was later migrated to 6. I have just realised that the syndication settings were no migrated to the correct settings in Plone 6.

In Plone 5 those settings were under Products.CMFPlone.interfaces.syndication.ISiteSyndicationSettings and in Plone 6 those settings are under plone.base.interfaces.syndication.ISiteSyndicationSettings

Error removing resource registry tools in upgrade

I tried an upgrade from Plone 5.0.10 to latest 5.2 coredev. The upgrade broke:

Traceback (most recent call last):
  File "/Users/maurits/community/plone-coredev/5.2/src/Products.CMFPlone/Products/CMFPlone/MigrationTool.py", line 291, in upgrade
    step['step'].doStep(setup)
  File "/Users/maurits/shared-eggs/cp27m/Products.GenericSetup-2.0b5-py2.7.egg/Products/GenericSetup/upgrade.py", line 168, in doStep
    self.handler(tool)
  File "/Users/maurits/community/plone-coredev/5.2/src/plone.app.upgrade/plone/app/upgrade/v52/betas.py", line 102, in to52beta1
    remove_legacy_resource_registries(context)
  File "/Users/maurits/community/plone-coredev/5.2/src/plone.app.upgrade/plone/app/upgrade/v52/betas.py", line 48, in remove_legacy_resource_registries
    portal.manage_delObjects(tools)
  File "/Users/maurits/community/plone-coredev/5.2/src/Products.CMFPlone/Products/CMFPlone/Portal.py", line 126, in manage_delObjects
    "Do not have permissions to remove this object")
Unauthorized: Do not have permissions to remove this object

Any idea?
cc @ksuess

5.2: remove the "View History" permission

See: zopefoundation/AccessControl@6cba6d8

move dotted behaviors to named behaviors

this upgrade step will switch the behaviors on the type fti's to named behaviors

see more here:
plone/plone.app.contenttypes#519

aq_base(obj) instead of obj.aq_base

For some reason at
https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/v50/final.py#L73
the obj may not have an acquisition-wrapper (due to prior problems obj is a list in our case, but this must not stop the migration process here). So attribute access fails and migration breaks.

Using aq_base(obj) instead of obj.aq_base is considered best practice and has to be used here as well.

Some bundles are missing after upgrade to Plone 6 with ES6

Compared to a fresh site, a migrated site is missing the jquery, plone-logged-in and plone-legacy bundles. On a new site, the last one is created by plone.app.event. It is maybe no longer needed.

Result is that lots of javascript is not working. See comments in this PR: #269

I assign Johannes, but feel free to pass it to someone else. Or someone else can volunteer.

Huge performance issue with to501

This part of the upgrade is basically waking up and reindexing all of the dexterity object of the catalog: https://github.com/plone/plone.app.upgrade/blob/master/plone/app/upgrade/v50/final.py#L53-L55

iface = "plone.dexterity.interfaces.IDexterityContent"
for brain in search(object_provides=iface):
    brain._unrestrictedGetObject().reindexObject(idxs=['getIcon'])

It is just not feasible on a huge site. If this is for a getIcon index, this should be done for the whole catalog using reindexIndex. But I'm guessing this is for the getIcon metadata (there is no getIcon index?), so I'm not sure what the right method should be (or if there is even one)?

Add the timezone property

Added in Products.CMFPlone by this PR plone/Products.CMFPlone#3625.

We do not have an upgrade path for collective.dexterityindexer

The package collective.dexterityindexer was merged into core, see PLIP 2780.

IMO an upgrade step that renames the behavior collective.dexteritytextindexer to plone.dexteritytextindexer is missing.

add Collection to the default_page_types setting

See plone/plone.app.content#182

Add upgrade step for Plone 5.1.6

Plone 5.1.6 needs an empty upgrade step

run upgrade step plonetheme.barceloneta

plone/plonetheme.barceloneta#308

cc @petschki

alphas.py cause unicode error

I tried to upgrade plone from 4.3.11 to 5.0.6.

beta.py uses safe_unicode().
http://docs.plone.org/manage/troubleshooting/unicode.html#safe-unicode

let use safe_unicode at alphas.py.

2017-02-07 04:25:42 INFO plone.app.upgrade Dry run selected.
2017-02-07 04:25:42 INFO plone.app.upgrade Starting the migration from version: 4314
2017-02-07 04:25:42 INFO plone.app.upgrade Ran upgrade step: Miscellaneous
2017-02-07 04:25:42 INFO Products.GenericSetup.tool Importing profile profile-plone.app.upgrade.v50:to50alpha3 with dependency strategy upgrade.
2017-02-07 04:25:42 INFO Products.GenericSetup.tool Applying main profile profile-plone.app.upgrade.v50:to50alpha3
2017-02-07 04:25:42 INFO GenericSetup.rolemap Role / permission map imported.
2017-02-07 04:25:42 ERROR GenericSetup Step cli-uninstall-configlet has an invalid import handler
/home/vagrant/Plone/buildout-cache/eggs/Products.CMFCore-2.2.10-py2.7.egg/Products/CMFCore/DirectoryView.py:493: UserWarning: DirectoryView plone_deprecated refers to a non-existing path 'Products.CMFPlone:skins/plone_deprecated'
(self.id, reg_key), UserWarning)
/home/vagrant/Plone/buildout-cache/eggs/Products.CMFCore-2.2.10-py2.7.egg/Products/CMFCore/DirectoryView.py:493: UserWarning: DirectoryView classic_templates refers to a non-existing path 'plonetheme.classic:skins/classic_templates'
(self.id, reg_key), UserWarning)
/home/vagrant/Plone/buildout-cache/eggs/Products.CMFCore-2.2.10-py2.7.egg/Products/CMFCore/DirectoryView.py:493: UserWarning: DirectoryView classic_styles refers to a non-existing path 'plonetheme.classic:skins/classic_styles'
(self.id, reg_key), UserWarning)
/home/vagrant/Plone/buildout-cache/eggs/Products.CMFCore-2.2.10-py2.7.egg/Products/CMFCore/DirectoryView.py:493: UserWarning: DirectoryView classic_ecmascript refers to a non-existing path 'plonetheme.classic:skins/classic_ecmascript'
(self.id, reg_key), UserWarning)
2017-02-07 04:25:42 INFO GenericSetup.archetypetool Archetype tool imported.
2017-02-07 04:25:42 ERROR GenericSetup Step languagetool has an invalid import handler
2017-02-07 04:25:42 INFO plone.app.upgrade Ran upgrade step: Run to50alpha3 upgrade profile
2017-02-07 04:25:42 INFO plone.app.upgrade Ran upgrade step: Upgrade editing control panel settings
2017-02-07 04:25:42 INFO plone.app.upgrade Ran upgrade step: Upgrade maintenance control panel settings
2017-02-07 04:25:42 INFO plone.app.upgrade Ran upgrade step: Upgrade navigation control panel settings
2017-02-07 04:25:42 INFO plone.app.upgrade Ran upgrade step: Upgrade search control panel settings
2017-02-07 04:25:42 ERROR plone.app.upgrade Upgrade aborted. Error:
Traceback (most recent call last):
File "/home/vagrant/Plone/buildout-cache/eggs/Products.CMFPlone-5.0.6-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 268, in upgrade
step['step'].doStep(setup)
File "/home/vagrant/Plone/buildout-cache/eggs/Products.GenericSetup-1.8.3-py2.7.egg/Products/GenericSetup/upgrade.py", line 166, in doStep
self.handler(tool)
File "/home/vagrant/Plone/buildout-cache/eggs/plone.app.upgrade-1.3.27-py2.7.egg/plone/app/upgrade/v50/alphas.py", line 315, in upgrade_site_controlpanel_settings
settings.site_title = unicode(portal.title)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 0: ordinal not in range(128)

"v52.betas.to52beta1" puts the Python import machinery in an inconsistent state

Python 2.7, plone.app.upgrade==2.0.34, Products.ResourceRegistries installed:
After the call of to52beta1 import Products.ResourceRegistries fails with cannot import name "config" - even though config.py exists in Products.ResourceRegistries.

The reason: to52beta1 hacks the Python import machinery and brings it into an inconsistent state regarding Products.ResourceRegistries. The import machinery raises an ImportError for from m import n if n happens to be a module in sys.modules and is not an attribute of m. to52beta1 deletes Products.ResourceRegistries but lets Products.ResourceRegistries.config in sys.modules -- causing the inconsistency and the error above.

I recommend to remove all modules related to Products.ResourceRegistries from sys.modules in v52beta1 to properly clean up after the import machinery hack.

In recent versions (e.g. plone.app.upgrade==2.0.34), to52beta1 has got at its end an install_product('plone.staticresources'). This fails for us with the ImportError above (due to the import machinery inconsistency). Apparently, this ImportError is hidden by upper layers of the upgrade machinery (this likely should not happen).

The problems possibly only occur if Products.ResourceRegistries is installed.

Upgrade from 4.3.10 to 4.3.20 fails

I was trying to upgrade a 4.3.10 from a customer to 4.3.19 (And later tried 4.3.20 and I was getting the same), where it would fail with this:

2020-09-22 19:16:44 ERROR plone.app.upgrade Upgrade aborted. Error:
Traceback (most recent call last):
  File "/opt/plone/eggs/Products.CMFPlone-4.3.20-py2.7.egg/Products/CMFPlone/MigrationTool.py", line 285, in upgrade
    step['step'].doStep(setup)
  File "/opt/plone/eggs/Products.GenericSetup-1.8.11-py2.7.egg/Products/GenericSetup/upgrade.py", line 166, in doStep
    self.handler(tool)
  File "/vagrant/src/plone.app.upgrade/plone/app/upgrade/v43/final.py", line 358, in fix_double_smaxage
    if registry.records[maxage].field.recordName != def_smaxage:
AttributeError: 'Int' object has no attribute 'recordName'

Now, I haven't had time to check if this is happening with a vanilla Plone 4.3.10

Upgrade 4.3.7 to 5.0.2

I installed a new fresh empty(!) instance Plone 4.3.7 and 5.0.2.
Easy replace folders blobstorage and filestorage. 4.3.7 -> 5.0.2

Upgrade using @@plone-upgrade to 5013.
The profile "Products.CMFPlone:plone" is currently upgraded to version 5013.
The filesystem version for the "Products.CMFPlone:plone" profile is currently 5013.

As a result there are several problems:

The theme does not display correctly, without styles:
error while rendering plone.resourceregistries.scripts error while rendering plone.resourceregistries.styles
The theme in the control panel do not open:

Traceback (innermost last): 
    • Module ZPublisher.Publish, line 138, in publish 
    • Module ZPublisher.mapply, line 77, in mapply 
    • Module ZPublisher.Publish, line 48, in call_object 
    • Module plone.app.theming.browser.controlpanel, line 46, in __call__ 
    • Module plone.app.theming.browser.controlpanel, line 98, in update 
    • Module plone.app.theming.browser.controlpanel, line 56, in _setup 
    • Module plone.app.theming.utils, line 396, in getZODBThemes 
    • Module plone.resource.manifest, line 220, in getZODBResources 
    • Module zope.component._api, line 169, in getUtility 
ComponentLookupError: (<InterfaceClass plone.resource.interfaces.IResourceDirectory>, 'persistent')

Resource Registries in the control panel do not open:

Traceback (innermost last): 
    • Module ZPublisher.Publish, line 138, in publish 
    • Module ZPublisher.mapply, line 77, in mapply 
    • Module ZPublisher.Publish, line 48, in call_object 
    • Module Products.CMFPlone.controlpanel.browser.resourceregistry, line 188, in __call__ 
    • Module Products.Five.browser.pagetemplatefile, line 125, in __call__ 
    • Module Products.Five.browser.pagetemplatefile, line 59, in __call__ 
    • Module zope.pagetemplate.pagetemplate, line 132, in pt_render 
    • Module five.pt.engine, line 93, in __call__ 
    • Module z3c.pt.pagetemplate, line 163, in render 
    • Module chameleon.zpt.template, line 258, in render 
    • Module chameleon.template, line 191, in render 
    • Module chameleon.template, line 171, in render 
    • Module 877c6a78204c73ccce43a39535a31369.py, line 540, in render 
    • Module 124ec85f3c5133f38f1b43a56e33d462.py, line 282, in render_master 
    • Module 4c5035dc1232db945c3c31004fa1cd73.py, line 1227, in render_master 
    • Module 124ec85f3c5133f38f1b43a56e33d462.py, line 264, in __fill_content 
    • Module 4c5035dc1232db945c3c31004fa1cd73.py, line 458, in render_content 
    • Module 124ec85f3c5133f38f1b43a56e33d462.py, line 253, in __fill_main 
    • Module 877c6a78204c73ccce43a39535a31369.py, line 226, in __fill_prefs_configlet_main 
    • Module five.pt.expressions, line 161, in __call__ 
    • Module Products.CMFPlone.controlpanel.browser.resourceregistry, line 435, in config 
    • Module Products.CMFPlone.controlpanel.browser.resourceregistry, line 379, in get_overrides 
    • Module Products.CMFPlone.controlpanel.browser.resourceregistry, line 79, in __init__ 
    • Module zope.component._api, line 169, in getUtility 
ComponentLookupError: (<InterfaceClass plone.resource.interfaces.IResourceDirectory>, 'persistent') - Expression: "view/config" - Filename: ... oducts/CMFPlone/controlpanel/browser/resourceregistry.pt - Location: (line 32: col 82) - Source: ... ="data-pat-resourceregistry view/config"> ^^^^^^^^^^^ - Arguments: repeat: {...} (0) template: <ViewPageTemplateFile - at 0xb373f34cL> views: <ViewMapper - at 0xb141c6acL> modules: <instance - at 0xb639ad2cL> args: <tuple - at 0xb746602cL> here: <ImplicitAcquisitionWrapper Plone at 0xaf7e57d4L> user: <ImplicitAcquisitionWrapper - at 0xb0405aa4L> nothing: <NoneType - at 0x83ac190> container: <ImplicitAcquisitionWrapper Plone at 0xaf7e57d4L> request: <instance - at 0xb0429eecL> wrapped_repeat: <SafeMapping - at 0xb0e747fcL> traverse_subpath: <list - at 0xb141cdacL> default: <object - at 0xb748c6d0L> loop: {...} (0) context: <ImplicitAcquisitionWrapper Plone at 0xaf7e57d4L> view: <SimpleViewClass from /usr/local/Plone502-2/buildout-cache/eggs/Products.CMFPlone-5.0.2-py2.7.egg/Products/CMFPlone/controlpanel/browser/resourceregistry.pt resourceregistry-controlpanel at 0xb141c0acL> translate: <function translate at 0xb02ab7d4L> root: <ImplicitAcquisitionWrapper Zope at 0xb04a3cacL> options: {...} (0) target_language: <NoneType - at 0x83ac190>

Add upgrade step for Plone 5.2 final

Yay towncrier.

Action icons are not setup when upgrading from Plone 5.2 to Plone 6 classic

I just migrated a site and noticed that some actions icons are not setup:

The problem is fixed (for example) by setting the proper icon_expr to string:toolbar-action/sharing here:

http://localhost:8080/Plone/portal_actions/object/local_roles/manage_propertiesForm

Plone 6: remove upgrades from 5.1 and earlier

You cannot migrate from Plone 5.1 or earlier to Plone 6.0 in one step. You must always first migrate to Plone 5.2 on Python 2.7 and then migrate your database to Python 3. Only then can you migrate further to Plone 6.

So: the plone.app.upgrade version that goes into Plone 6 does not need any upgrade code from Plone 5.1 or earlier. Right?
That means we can get rid of most of the code.

Question is when to do that. Currently the master branch is also used by Plone 5.1, and I still expect one or two releases of that Plone version. As long as that is the case, it seems preferable to keep the code on master. Changes for 5.1, 5.2 and 6.0 can then all still go there.

We should probably branch sometime during the 6.0 alpha phase.

FTIs using RichText or LeadImage need to be updated

Caused by plone/plone.app.contenttypes#480
Related bug ticket: plone/Products.CMFPlone#2725

Image caption support in Plone 5.1.x Add Upgrade step for Plone 5.1.x

Enable the image caption on TinyMCE. The plone-logged-in-compided.js file was re compiled.
The upgradeStep update filed last_compilation.

ref:
plone/plone.outputfilters#37
plone/mockup#977
plone/Products.CMFPlone#3098

update_social_media_fields fails on empty values?

Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.GenericSetup.tool, line 1053, in manage_doUpgrades
  Module Products.GenericSetup.upgrade, line 166, in doStep
  Module plone.app.upgrade.v51.betas, line 130, in update_social_media_fields
  Module plone.registry.recordsproxy, line 60, in __setattr__
  Module plone.registry.registry, line 51, in __setitem__
  Module plone.registry.record, line 82, in _set_value
  Module zope.schema._bootstrapfields, line 183, in validate
  Module zope.schema._bootstrapfields, line 310, in _validate
  Module zope.schema._bootstrapfields, line 210, in _validate
WrongType: ('', <type 'unicode'>, 'value')

TypeError: 'NoneType' object is not iterable in fix_unicode_properties

Migrating a site from Plone 5.2 to 6.0.0a3:

Traceback (most recent call last):
  File "/Users/maurits/shared-eggs/cp39/Products.CMFPlone-6.0.0a3-py3.9.egg/Products/CMFPlone/MigrationTool.py", line 302, in upgrade
    step['step'].doStep(setup)
  File "/Users/maurits/shared-eggs/cp39/Products.GenericSetup-2.1.5-py3.9.egg/Products/GenericSetup/upgrade.py", line 169, in doStep
    self.handler(tool)
  File "/Users/maurits/shared-eggs/cp39/plone.app.upgrade-3.0.0a3-py3.9.egg/plone/app/upgrade/v60/alphas.py", line 266, in fix_unicode_properties
    portal.ZopeFindAndApply(portal, search_sub=1, apply_func=fix_properties)
  File "/Users/maurits/shared-eggs/cp39/Zope-5.4-py3.9.egg/OFS/FindSupport.py", line 171, in ZopeFindAndApply
    self.ZopeFindAndApply(ob, obj_ids, obj_metatypes,
  File "/Users/maurits/shared-eggs/cp39/Zope-5.4-py3.9.egg/OFS/FindSupport.py", line 165, in ZopeFindAndApply
    apply_func(ob, (apply_path + '/' + p))
  File "/Users/maurits/shared-eggs/cp39/Zope-5.4-py3.9.egg/ZPublisher/utils.py", line 149, in fix_properties
    prop_map = obj.propertyMap()
  File "/Users/maurits/shared-eggs/cp39/Zope-5.4-py3.9.egg/OFS/PropertyManager.py", line 252, in propertyMap
    return tuple(dict.copy() for dict in self._propertyMap())
TypeError: 'NoneType' object is not iterable

The object where it fails, is portal_skins/CMFEditions. This skin layer no longer exists, and will be removed further on in the upgrade.

In fix_unicode_properties we do a try/except AttributeError around the call to obj.propertyMap(). There may be more things that can go wrong in corner cases like this, so a try/except for AttributeError/TypeError/ValueError/KeyError may be best.
Note that the fix_unicode_properties function is in Zope (I created it), but we have an own copy, which I added for the time that a Zope 5.4 release was not yet made. We should fix it in both places, and use our own version for the moment.

Upgrade to 6.0.0a1 fails

Here is the upgrade report:

Upgrade report

Dry run selected.
Starting the migration from version: 5210
Ran upgrade step: Migrate site logo from native string to bytes again
Made portal_setup logs only available for Manager and Owner.
Made portal_setup snapshots only available for Manager and Owner.
Ran upgrade step: Make portal_setup logs accessible only to Manager/Owner.
Ran upgrade step: Miscellaneous
Ran upgrade step: Miscellaneous
Types tool imported.
Upgrade aborted. Error:
Traceback (most recent call last):
  File "/home/user/optplone/buildout-cache/eggs/Products.CMFPlone-6.0.0a1-py3.8.egg/Products/CMFPlone/MigrationTool.py", line 302, in upgrade
    step['step'].doStep(setup)
  File "/home/user/optplone/buildout-cache/eggs/Products.GenericSetup-2.1.4-py3.8.egg/Products/GenericSetup/upgrade.py", line 168, in doStep
    self.handler(tool)
  File "/home/user/optplone/buildout-cache/eggs/plone.app.upgrade-2.0.41-py3.8.egg/plone/app/upgrade/v60/alphas.py", line 66, in change_plone_site_fti
    loadMigrationProfile(context, "profile-plone.app.upgrade.v60:to_dx_site_root")
  File "/home/user/optplone/buildout-cache/eggs/plone.app.upgrade-2.0.41-py3.8.egg/plone/app/upgrade/utils.py", line 203, in loadMigrationProfile
    context.runAllImportStepsFromProfile(profile, purge_old=False)
  File "/home/user/optplone/buildout-cache/eggs/Products.GenericSetup-2.1.4-py3.8.egg/Products/GenericSetup/tool.py", line 399, in runAllImportStepsFromProfile
    result = self._runImportStepsFromContext(
  File "/home/user/optplone/buildout-cache/eggs/Products.GenericSetup-2.1.4-py3.8.egg/Products/GenericSetup/tool.py", line 1487, in _runImportStepsFromContext
    message = self._doRunImportStep(step, context)
  File "/home/user/optplone/buildout-cache/eggs/Products.GenericSetup-2.1.4-py3.8.egg/Products/GenericSetup/tool.py", line 1299, in _doRunImportStep
    return handler(context)
  File "/home/user/optplone/buildout-cache/eggs/Products.CMFCore-2.5.4-py3.8.egg/Products/CMFCore/exportimport/typeinfo.py", line 222, in importTypesTool
    importObjects(tool, '', context)
  File "/home/user/optplone/buildout-cache/eggs/Products.GenericSetup-2.1.4-py3.8.egg/Products/GenericSetup/utils.py", line 960, in importObjects
    importObjects(sub, path + '/', context)
  File "/home/user/optplone/buildout-cache/eggs/Products.GenericSetup-2.1.4-py3.8.egg/Products/GenericSetup/utils.py", line 956, in importObjects
    importer.body = body
  File "/home/user/optplone/buildout-cache/eggs/Products.GenericSetup-2.1.4-py3.8.egg/Products/GenericSetup/utils.py", line 557, in _importBody
    self._importNode(dom.documentElement)
  File "/home/user/optplone/buildout-cache/eggs/Products.CMFCore-2.5.4-py3.8.egg/Products/CMFCore/exportimport/typeinfo.py", line 61, in _importNode
    self._initProperties(node)
  File "/home/user/optplone/buildout-cache/eggs/Products.GenericSetup-2.1.4-py3.8.egg/Products/GenericSetup/utils.py", line 865, in _initProperties
    obj._updateProperty(prop_id, prop_value)
  File "/home/user/optplone/buildout-cache/eggs/plone.dexterity-2.10.4-py3.8.egg/plone/dexterity/fti.py", line 300, in _updateProperty
    modified(self, DexterityFTIModificationDescription(id, oldValue))
  File "/home/user/optplone/buildout-cache/eggs/zope.lifecycleevent-4.3-py3.8.egg/zope/lifecycleevent/__init__.py", line 78, in modified
    notify(ObjectModifiedEvent(object, *descriptions))
  File "/home/user/optplone/buildout-cache/eggs/zope.event-4.5.0-py3.8.egg/zope/event/__init__.py", line 32, in notify
    subscriber(event)
  File "/home/user/optplone/buildout-cache/eggs/zope.component-4.6.2-py3.8.egg/zope/component/event.py", line 27, in dispatch
    component_subscribers(event, None)
  File "/home/user/optplone/buildout-cache/eggs/zope.component-4.6.2-py3.8.egg/zope/component/_api.py", line 134, in subscribers
    return sitemanager.subscribers(objects, interface)
  File "/home/user/optplone/buildout-cache/eggs/zope.interface-5.4.0-py3.8-linux-x86_64.egg/zope/interface/registry.py", line 448, in subscribers
    return self.adapters.subscribers(objects, provided)
  File "/home/user/optplone/buildout-cache/eggs/zope.interface-5.4.0-py3.8-linux-x86_64.egg/zope/interface/adapter.py", line 899, in subscribers
    subscription(*objects)
  File "/home/user/optplone/buildout-cache/eggs/zope.component-4.6.2-py3.8.egg/zope/component/event.py", line 36, in objectEventNotify
    component_subscribers((event.object, event), None)
  File "/home/user/optplone/buildout-cache/eggs/zope.component-4.6.2-py3.8.egg/zope/component/_api.py", line 134, in subscribers
    return sitemanager.subscribers(objects, interface)
  File "/home/user/optplone/buildout-cache/eggs/zope.interface-5.4.0-py3.8-linux-x86_64.egg/zope/interface/registry.py", line 448, in subscribers
    return self.adapters.subscribers(objects, provided)
  File "/home/user/optplone/buildout-cache/eggs/zope.interface-5.4.0-py3.8-linux-x86_64.egg/zope/interface/adapter.py", line 899, in subscribers
    subscription(*objects)
  File "/home/user/optplone/buildout-cache/eggs/plone.dexterity-2.10.4-py3.8.egg/plone/dexterity/fti.py", line 531, in ftiModified
    unregister_factory(old_factory, site_manager)
  File "/home/user/optplone/buildout-cache/eggs/plone.dexterity-2.10.4-py3.8.egg/plone/dexterity/fti.py", line 450, in unregister_factory
    utilities = list(site_manager.registeredUtilities())
  File "/home/user/optplone/buildout-cache/eggs/five.localsitemanager-3.2.2-py3.8.egg/five/localsitemanager/registry.py", line 239, in registeredUtilities
    reg.component = _wrap(reg.component, self)
  File "/home/user/optplone/buildout-cache/eggs/five.localsitemanager-3.2.2-py3.8.egg/five/localsitemanager/registry.py", line 142, in _wrap
    if registry.__bases__ and IAcquirer.providedBy(comp):
  File "/home/user/optplone/buildout-cache/eggs/ZODB-5.6.0-py3.8.egg/ZODB/Connection.py", line 795, in setstate
    self._reader.setGhostState(obj, p)
  File "/home/user/optplone/buildout-cache/eggs/ZODB-5.6.0-py3.8.egg/ZODB/serialize.py", line 633, in setGhostState
    state = self.getState(pickle)
  File "/home/user/optplone/buildout-cache/eggs/ZODB-5.6.0-py3.8.egg/ZODB/serialize.py", line 626, in getState
    return unpickler.load()
  File "/home/user/optplone/buildout-cache/eggs/zope.interface-5.4.0-py3.8-linux-x86_64.egg/zope/interface/declarations.py", line 891, in Provides
    spec = ProvidesClass(*interfaces)
  File "/home/user/optplone/buildout-cache/eggs/zope.interface-5.4.0-py3.8-linux-x86_64.egg/zope/interface/declarations.py", line 822, in __init__
    Declaration.__init__(self, *self._add_interfaces_to_cls(interfaces, cls))
  File "/home/user/optplone/buildout-cache/eggs/zope.interface-5.4.0-py3.8-linux-x86_64.egg/zope/interface/declarations.py", line 87, in __init__
    Specification.__init__(self, _normalizeargs(bases))
  File "/home/user/optplone/buildout-cache/eggs/zope.interface-5.4.0-py3.8-linux-x86_64.egg/zope/interface/declarations.py", line 1307, in _normalizeargs
    _normalizeargs(v, output)
  File "/home/user/optplone/buildout-cache/eggs/zope.interface-5.4.0-py3.8-linux-x86_64.egg/zope/interface/declarations.py", line 1306, in _normalizeargs
    for v in sequence:
TypeError: 'type' object is not iterable
End of upgrade path, main migration has finished.
The upgrade path did NOT reach current version.
Migration has failed
Dry run selected, transaction aborted

Add missing upgrade step for CMFEditions

This fixes: plone/Products.CMFEditions#64 and plone/Products.CMFPlone#2701

Migration from 4.3.15 to 5.1.2 end up in message, Database need newer Plone version

When I do the upgrade, it seems to finsh almost all steps, but get's an strange error at the end.

------
2018-07-01T23:32:57 INFO Products.GenericSetup.tool Importing profile profile-plone.app.upgrade.v51:to511 with dependency strategy ignore.
------
2018-07-01T23:32:57 INFO Products.GenericSetup.tool Applying main profile profile-plone.app.upgrade.v51:to511
------
2018-07-01T23:32:57 INFO GenericSetup.rolemap Role / permission map imported.
------
2018-07-01T23:32:57 ERROR GenericSetup Step collective.z3cform.datetimewidget has an invalid import handler
------
2018-07-01T23:32:57 INFO GenericSetup.archetypetool Archetype tool imported.
------
2018-07-01T23:32:57 ERROR GenericSetup Step languagetool has an invalid import handler
------
2018-07-01T23:32:57 INFO plone.app.upgrade Ran upgrade step: Run to511 upgrade profile.
------
2018-07-01T23:32:57 INFO Products.GenericSetup.tool Importing profile profile-plone.app.upgrade.v51:to512 with dependency strategy ignore.
------
2018-07-01T23:32:57 INFO Products.GenericSetup.tool Applying main profile profile-plone.app.upgrade.v51:to512
------
2018-07-01T23:32:57 INFO GenericSetup.rolemap Role / permission map imported.
------
2018-07-01T23:32:57 ERROR GenericSetup Step collective.z3cform.datetimewidget has an invalid import handler
------
2018-07-01T23:32:57 INFO GenericSetup.archetypetool Archetype tool imported.
------
2018-07-01T23:32:57 ERROR GenericSetup Step languagetool has an invalid import handler
------
2018-07-01T23:32:57 INFO plone.app.upgrade Ran upgrade step: Run to512 upgrade profile.
------
2018-07-01T23:32:57 INFO plone.app.upgrade Action object/ical_import_enable was not found
------
2018-07-01T23:32:57 INFO plone.app.upgrade Action object/ical_import_disable was not found
------
2018-07-01T23:32:57 INFO plone.app.upgrade Ran upgrade step: Update an action i18n domain
------
2018-07-01T23:32:57 INFO plone.app.upgrade Ran upgrade step: Miscellaneous
------
2018-07-01T23:32:57 INFO plone.app.upgrade End of upgrade path, main migration has finished.
------
2018-07-01T23:32:57 ERROR plone.app.upgrade The upgrade path did NOT reach current version.
------
2018-07-01T23:32:57 ERROR plone.app.upgrade Migration has failed