Comments (8)
nvanbenschoten
commented on June 29, 2024
We see the test fail with the error should have 3 item(s), but has 13:
Here are the descriptors that it found:
r69:/Tenant/2{-a} [(n1,s1):1, next=2, gen=4, sticky=1718626166.934881645,0]
r77:/Tenant/2{a-b} [(n1,s1):1, next=2, gen=5, sticky=9223372036.854775807,2147483647]
r80:/Tenant/2{b-/Table/11} [(n1,s1):1, next=2, gen=5, sticky=9223372036.854775807,2147483647]
r71:/Tenant/2/Table/1{1-2} [(n1,s1):1, next=2, gen=4]
r72:/Tenant/2/Table/{12-25} [(n1,s1):1, next=2, gen=5]
r73:/Tenant/2/Table/2{5-6} [(n1,s1):1, next=2, gen=6]
r74:/Tenant/2/Table/2{6-7} [(n1,s1):1, next=2, gen=7]
r75:/Tenant/2/Table/2{7-8} [(n1,s1):1, next=2, gen=8]
r76:/Tenant/2/Table/{28-45} [(n1,s1):1, next=2, gen=9]
r78:/Tenant/2/Table/4{5-6} [(n1,s1):1, next=2, gen=10]
r79:/Tenant/2/Table/4{6-7} [(n1,s1):1, next=2, gen=11]
r81:/Tenant/2/Table/4{7-8} [(n1,s1):1, next=2, gen=12]
r82:/Tenant/{2/Table/48-3} [(n1,s1):1, next=2, gen=12]from cockroach.
nvanbenschoten
commented on June 29, 2024
As is often the case with these kinds of failures, the test is broken, but it's surviving most of the time by winning a race with async range splits. The following diff breaks it:
diff --git a/pkg/kv/kvclient/kvtenant/tenant_scan_range_descriptors_test.go b/pkg/kv/kvclient/kvtenant/tenant_scan_range_descriptors_test.go
index 766c3aed2a2..d89fb533fa1 100644
--- a/pkg/kv/kvclient/kvtenant/tenant_scan_range_descriptors_test.go
+++ b/pkg/kv/kvclient/kvtenant/tenant_scan_range_descriptors_test.go
@@ -13,6 +13,7 @@ package kvtenant_test
import (
"context"
"testing"
+ "time"
"github.com/cockroachdb/cockroach/pkg/base"
"github.com/cockroachdb/cockroach/pkg/keys"
@@ -68,6 +69,8 @@ func TestScanRangeDescriptors(t *testing.T) {
tc.SplitRangeOrFatal(t, ten2Codec.TenantEndKey()) // Last range
}
+ time.Sleep(10 * time.Second)
+
iter, err := iteratorFactory.NewIterator(ctx, ten2Codec.TenantSpan())
require.NoError(t, err)from cockroach.
nvanbenschoten
commented on June 29, 2024
This range splits on Table 11, 25, and 27 are due to #119947.
from cockroach.
nvanbenschoten
commented on June 29, 2024
Will be fixed by #125809.
from cockroach.
nvanbenschoten
commented on June 29, 2024
EDIT: my previous messages were all looking at the artifacts from #125780, not from this issue.
from cockroach.
nvanbenschoten
commented on June 29, 2024
We see from the stack trace that an AdminMerge is running a Subsume request and is not blocked.
From the logs, it looks like the merge is spinning, which lines up with the stack trace:
I240617 12:21:18.608163 43 kv/kvserver/replica_command.go:830 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 21270 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
I240617 12:21:18.975656 43 kv/kvserver/replica_command.go:830 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 21271 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
I240617 12:21:19.350851 43 kv/kvserver/replica_command.go:830 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 21272 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
I240617 12:21:19.744568 43 kv/kvserver/replica_command.go:830 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 21273 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
I240617 12:21:20.123045 43 kv/kvserver/replica_command.go:830 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 21274 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
I240617 12:21:20.460222 43 kv/kvserver/replica_command.go:830 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 21275 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
I240617 12:21:20.857108 43 kv/kvserver/replica_command.go:830 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 21276 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
I240617 12:21:21.182655 43 kv/kvserver/replica_command.go:830 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 21277 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
I240617 12:21:21.540745 43 kv/kvserver/replica_command.go:830 ⋮ [T1,Vsystem,n1,s1,r69/1:‹/System?aaa-testing{-\x00\x00}›,*kvpb.AdminMergeRequest] 21278 initiating a merge of r70:‹/System{\x00aaa-testing\x00\x00-/NodeLiveness}› [(n1,s1):1, next=2, gen=2, sticky=9223372036.854775807,2147483647] into this range (manual)
from cockroach.
nvanbenschoten
commented on June 29, 2024
This has been a long-standing rare test flake (see #110340) which is possibly due to the manual clock that the test is using. I can't reproduce, but I've landed #125854 on master so that we'll have more information next time that it flakes. For now, I'll move to a P-3.
from cockroach.
arulajmani
commented on June 29, 2024
Closing this out as a duplicate of #126249. This one's on an RC branch anyway, so there's not much benefit in keeping it open anyway.
from cockroach.
Related Issues (20)
- roachtest: kv0/enc=false/nodes=3/size=4kb failed
- operator classes are only allowed for the last column of an inverted index HOT 2
- sql/stats: disable partial stats using extremes for enum and bool type columns HOT 1
- kv/kvserver/intentresolver: TestIntentResolutionUnavailableRange failed
- opt: poor index recommendations for queries filtering non-indexed columns
- pkg/server/authserver/authserver_test: TestLogoutClearsCookies failed
- util/log: TestMessageReceived failed
- pkg/sql/logictest/tests/cockroach-go-testserver-23.2/cockroach-go-testserver-23_2_test: TestLogic_upgrade_system_table_changes failed
- storage: categorized write metrics are inflated HOT 1
- Sentry: replica_application_state_machine.go:367: log.Fatal: unhandled field in ReplicatedEvalResult: [] (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/kv/kvserver... HOT 1
- pkg/compose/compose_test: TestComposeCompare failed
- roachtest: admission-control/row-level-ttl/expired-rows=false failed
- roachtest: tpcc/w=100/nodes=3/chaos=true failed HOT 1
- roachtest: tpcc/large-schema-benchmark/tables=1000 failed
- roachtest: tpcc/large-schema-benchmark/tables=500 failed
- roachtest: tpcc-nowait/isolation-level=mixed/nodes=3/w=1 failed
- roachtest: tpcc-nowait/isolation-level=read-committed/nodes=3/w=1 failed
- roachtest: c2c/import/7tb/kv0 failed
- roachtest: tpcc-nowait/isolation-level=snapshot/nodes=3/w=1 failed
- roachtest: tpcc/large-schema-benchmark/tables=10000 failed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.