Multiple ZEO data corruptions due to concurrency bugs about zeo HOT 13 CLOSED

Tests for Bug2 : zopefoundation/ZODB#368.

from zeo.

Fix for Bug2 in ZEO4: https://lab.nexedi.com/kirr/ZEO/commit/b71e76ae.

from zeo.

from zeo.

@d-maurer, thanks for feedback. Invalidation sending happens in coroutine context in the IO thread - yes, but if the storage is multithreaded - the acceptor code in another thread can mutate the data structure, that IO thread is using, simultaeneously. And that leads to some clients being skipped on sending the invalidation message.

Here I'm talking about ZEO_MTACCEPTOR=1 mode. Yes, this mode recently became deprecated, but there was no conscious understanding that multithreading mode removal is needed for correctness. If this is indeed the case, I believe we should explicitly document single-threaded assumption in the places where it is important.

from zeo.

from zeo.

I agree with Dieter. I don't see any value in a wild goose chase uncovering issues with a non-standard implementation that hasn't seen much use and zero active support for many years.

from zeo.

from zeo.

@d-maurer, @dataflake, thanks for feedback, kind words, and I apologize for the delay with replying on my side.

@dataflake, please do not get me wrong. In the first place I'm not trying to raise up incorrectness issues in ancient configurations, that noone is actually using. What we had in #207 (comment) was sign of data corruption due to a concurrency bug, and it was unclear where the bug sits and whether it affects only old code we stopped (or are stopping) to care about, or modern configuration as well. A data corruption is potentially significant problem in my view. It's like when an aircraft or spaceship crashes, or shows significant issues, all other aircrafts of the same model are landed and stop to fly until the issue is understood. Isn't it?

Said that, with current understanding, I suggest the following steps:

1. remove support for ZEO4 from ZEO5 client without waiting for ZEO6. The data corruption described by Bug1 is significant in my view and deserves the fix without waiting, nor without forcing users to upgrade to new major version. If we keep support for ZEO4 server, the fix is non-trivial. But in situation, when, I believe, nobody is using ZEO5.client - ZEO4.server configuration today, the fix could be to reject connection to ZEO4 server at all. If upon such rejection we log appropriate message, explaining that ZEO4 server support is dropped because of the data corruption reasons, I believe it should be clear for people, even if someone is using such configuration, to stop using it. And we can also add to the logged message, that if, for some reason, someone actually depends on such configuration today, that the person should speak up and describe those reasons on a ZEO issue. After all in 2015/2016 it was practical to add ZEO4 compatibility to newly-introduced and unstable ZEO5 client. But in 2022 the situation is different - ZEO5 is believed to be relatively stable, and ZEO4 stopped to receive updates several years ago. That's why I believe that dropping ZEO4 server support from ZEO5 should be ok.

2. Regarding MTACCEPTOR in ZEO5 server, we already scheduled removal of this mode in ZEO 5.3.0 with the following entry in the changelog:

   Remove tests for the asyncio/mtacceptor module, it appears unused and presents a maintenance burden. The module will be removed in ZEO version 6.

   (https://pypi.org/project/ZEO/)

   and now, given that this mode also exposes data corruption issues I would
   also remove it with corresponding changelog entry without waiting for ZEO6 to
   be released.

What do you think?

If we agree on 1 and/or 2, I think I could implement those steps.

Kirill

from zeo.

The general policy throughout Zope Foundation projects for feature removals is this: Feature removal is normally preceded by clear deprecation warnings in the current release branch and the final removal requires a new major release. That way nasty surprises for users are kept to a minimum.

I can agree with removing MTACCEPTOR earlier - I think it's truly unused. Scheduling it for ZEO 6 was just an expression of our feature removal policy.

As you know my main concern (or main use case) for ZODB and ZEO is Zope deployments. So I checked the ZEO version required by Zope 4 and Zope 5 and noticed that ZEO 5 has been required from the earliest releases of Zope 4. That means there's a super low probability of Zope 4/5 deployments running on ZEO 4. I also noticed that ZEO 4 has last been released as version 4.3.1 6 years ago, it's basically dead. So I can agree with removal of ZEO 4 support as well.

In summary, both 1 and 2 are fine by me. I see very little risk breaking anyone's Zope deployment. And if you believe the data corruption risk is significant that's a good reason.

from zeo.

@dataflake, thanks for feedback, and it is good to see we generally agree here. I've created #211 to proceed with 2. Could you please have a look?

from zeo.

I've created #212 to start work on 1.

from zeo.

Pull request to remove support for interoperability with ZEO4 server: #213.

from zeo.

Fixed by #211 and #213.

from zeo.