Reduce common case resource overhead of update operation read-repairs by introducing metadata-only read phase about vespa HOT 4 CLOSED

Sounds good to me.

from vespa.

Seems like we should be able to support this without any wire format changes by sending down Gets with a field-set of [none] and only returning the last modified timestamp. Just have to verify that the timestamp is always set if present, regardless of field-set.

from vespa.

A subset of this proposal is implemented as part of #11319. It does not implement the metadata-only additional phase, but does trigger fast path updates if documents are in sync across replicas after the existing read phase.

from vespa.

We've very recently identified a race condition regression that was introduced with #11319 that may trigger inconsistent document versions to be created in the following scenario:

• The update is sent with create: true
• The replicas of the target bucket are already out of sync when the update arrives on the distributor
• A concurrent mutation to the target bucket modifies the size of its replica set (i.e. adds a replica) in the time between sending the Get operations and the subsequent Update operations

A log warning of the following form should be emitted if this particular edge case is triggered:

Update operation for 'id:foo::bar' in bucket Bucket(BucketSpace(0x0000000000000001), BucketId(0x4000000000f00baa)) updated documents with different timestamps. This should not happen and may indicate undetected replica divergence. Found ts=0 on node 5, ts=1576000000000001 on node 6

A grep for updated documents with different timestamps in the vespa logs will show if this has happened. If it has, the safest option is to re-feed the given document ID, as it may have replica divergence that Vespa cannot detect and automatically fix.

This regression shall have been fixed with #11561, but note that this fix is not yet pushed as a public release. In the meantime, if this problem is observed, it's possible to disable the code path that triggers the bug by adding a config override to services.xml for the affected content cluster:

<config name="vespa.config.content.core.stor-distributormanager">
<restart_with_fast_update_path_if_all_get_timestamps_are_consistent>false</restart_with_fast_update_path_if_all_get_timestamps_are_consistent>
</config>

This is a live change and does not require a process restart.

from vespa.