Comments (14)
I bisected this to 25edb96, cc @Dedej-Bergin @rafiss @fqazi thoughts on how that change can affect this test? (I used ./dev test ./pkg/sql/logictest/tests/local -f=TestLogic_select_for_update --stress --cpus=48 --count=1000
on the gceworker for the bisect.)
from cockroach.
This reproduces quickly with ./dev test pkg/sql/logictest/tests/local -f TestLogic_select_for_update --stress
. It looks like something in the last ~50 commits on master is causing this. I'm bisecting now.
from cockroach.
I'm having trouble bisecting because I think I was overly optimistic on how quickly the failure occurs under stress. Sometimes it will fail in the first few seconds, but other times I'll make it through all 1000 iterations without a failure, on a commit that I believe contains the change.
from cockroach.
pkg/sql/logictest/tests/local/local_test.TestLogic_select_for_update failed on master @ 992d4573fb8b71e28717b4d14c4c7ebccb38c412:
=== RUN TestLogic_select_for_update
test_log_scope.go:170: test logs captured to: outputs.zip/logTestLogic_select_for_update2654335675
test_log_scope.go:81: use -show-logs to present logs inline
[07:36:53] setting distsql_workmem='38107B';
logic.go:2968:
/var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update:575: SELECT k, u FROM t3 WHERE u = 2 FOR UPDATE SKIP LOCKED
expected:
2 2
4 2
but found (query options: "rowsort" -> ignore the following ordering of rows) :
[07:36:53] --- done: /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update with config local: 141 tests, 1 failures
logic.go:4135:
/var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update:581: error while processing
logic.go:4135: /var/lib/engflow/worker/work/2/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update:581: too many errors encountered, skipping the rest of the input
panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestLogic_select_for_update2654335675
--- FAIL: TestLogic_select_for_update (0.91s)
Parameters:
attempt=1
run=23
shard=15
This test on roachdash | Improve this report!
from cockroach.
pkg/sql/logictest/tests/local/local_test.TestLogic_select_for_update failed on master @ e6b993509e6ecdc0a0a88a8363cdd8a206ecf7f3:
=== RUN TestLogic_select_for_update
test_log_scope.go:170: test logs captured to: outputs.zip/logTestLogic_select_for_update3919272814
test_log_scope.go:81: use -show-logs to present logs inline
[19:45:39] setting distsql_workmem='36530B';
logic.go:2968:
/var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update:575: SELECT k, u FROM t3 WHERE u = 2 FOR UPDATE SKIP LOCKED
expected:
2 2
4 2
but found (query options: "rowsort" -> ignore the following ordering of rows) :
[19:45:39] --- done: /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update with config local: 141 tests, 1 failures
logic.go:4135:
/var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update:581: error while processing
logic.go:4135: /var/lib/engflow/worker/work/1/exec/bazel-out/k8-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update:581: too many errors encountered, skipping the rest of the input
panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestLogic_select_for_update3919272814
--- FAIL: TestLogic_select_for_update (0.77s)
Parameters:
attempt=1
run=1
shard=15
This test on roachdash | Improve this report!
from cockroach.
pkg/sql/logictest/tests/local/local_test.TestLogic_select_for_update failed with artifacts on master @ c511a3284cd821c40145f151537dd6b971dce1eb:
=== RUN TestLogic_select_for_update
test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/e30685d5351bb37f7931bdadd013af1b/logTestLogic_select_for_update1937186316
test_log_scope.go:81: use -show-logs to present logs inline
[10:25:50] setting distsql_workmem='75581B';
logic.go:2968:
/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/15423/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update:575: SELECT k, u FROM t3 WHERE u = 2 FOR UPDATE SKIP LOCKED
expected:
2 2
4 2
but found (query options: "rowsort" -> ignore the following ordering of rows) :
[10:25:51] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/15423/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update with config local: 141 tests, 1 failures
logic.go:4135:
/home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/15423/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update:581: error while processing
logic.go:4135: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/15423/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/sql/logictest/tests/local/local_test_/local_test.runfiles/com_github_cockroachdb_cockroach/pkg/sql/logictest/testdata/logic_test/select_for_update:581: too many errors encountered, skipping the rest of the input
panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e30685d5351bb37f7931bdadd013af1b/logTestLogic_select_for_update1937186316
--- FAIL: TestLogic_select_for_update (1.51s)
This test on roachdash | Improve this report!
from cockroach.
Can we skip the test for now?
from cockroach.
I wonder if this is related to #121917?
from cockroach.
I was able to repro with this smaller testdata file:
statement ok
CREATE TABLE t (k INT PRIMARY KEY, v int, FAMILY (k, v))
statement ok
GRANT admin TO testuser
# The SKIP LOCKED wait policy skip rows when a conflicting lock is encountered.
statement ok
INSERT INTO t VALUES (1,1), (2, 2), (3, 3), (4, 4)
statement ok
CREATE TABLE t3 (
k INT PRIMARY KEY,
v INT,
u INT,
INDEX (u),
FAMILY (k, v, u)
);
INSERT INTO t3 VALUES (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 2);
statement ok
BEGIN; UPDATE t SET v = 3 WHERE k = 2; UPDATE t3 SET v = 3 WHERE k = 2
user testuser
statement ok
BEGIN
query II rowsort
SELECT * FROM t FOR UPDATE SKIP LOCKED
----
1 1
3 3
4 4
statement ok
UPDATE t SET v = 4 WHERE k = 3
query II rowsort
SELECT * FROM t FOR UPDATE SKIP LOCKED
----
1 1
3 4
4 4
# All columns are available from the secondary index on u, so no index join is
# needed. The secondary index can produce the row where k=2 since the row is
# only locked in the primary index.
query II rowsort
SELECT k, u FROM t3 WHERE u = 2 FOR UPDATE SKIP LOCKED
----
2 2
4 2
That's pretty surprising to me -- this example doesn't use any GRANT <privilege> ...
statements, which were the ones that 25edb96 modified.
from cockroach.
I was able to reproduce this on 9203985 as well, which is the parent commit of 25edb96. I'm starting another bisect to find the first failing commit.
from cockroach.
@rafiss is this entirely serializable? If it's read committed, it might be worth waiting until #121917 is fixed.
from cockroach.
This fails all the way back to d93923b from over a year ago (I still haven't found the first failing commit), so it does seem likely to be #121917, and unrelated to the change we made in 25edb96. I will add this back to the SQL Queries board.
from cockroach.
Ack, thanks Rafi. I'm guessing that 25edb96 somehow made the flake more likely.
from cockroach.
My repro with the example above in pkg/sql/logictest/testdata/logic_test/_tmp
running with ./dev test ./pkg/sql/logictest/tests/local -f=TestLogic_tmp --stress --count=1000
fails just as quickly before or after 25edb96.
from cockroach.
Related Issues (20)
- kv/kvserver/liveness: TestNodeLivenessAppearsAtStart failed HOT 1
- `to_regrole` produces 0 unexpectedly HOT 2
- pkg/sql/opt/exec/execbuilder/tests/5node/5node_test: TestExecBuild_distsql_misc failed HOT 3
- roachtest: unoptimized-query-oracle/disable-rules=all/rand-tables failed HOT 1
- streamingccl: reduce replication lag from bulk ops from O(bulkOpRuntime*numNodes) to O(2*bulkOpRuntime) via replanning HOT 1
- roachtest: follower-reads/mixed-version/survival=region/locality=global/reads=strong failed HOT 1
- pkg/ccl/serverccl/statusccl/statusccl_test: TestTenantCannotSeeNonTenantStats failed HOT 1
- pkg/sql/logictest/tests/cockroach-go-testserver-23.2/cockroach-go-testserver-23_2_test: TestLogic_mixed_version_sequence_per_node_cache failed HOT 1
- cli: Zip tests failed HOT 1
- sqlstats: statements run in internal executors with outer txns have a transaction fingerprint id 0 of and no corresponding entry in txn stats HOT 1
- schema: benchmark workload with multiple schema changes HOT 1
- pkg/sql/logictest/tests/cockroach-go-testserver-23.2/cockroach-go-testserver-23_2_test: TestLogic_mixed_version_sequence_per_node_cache failed HOT 1
- pkg/sql/ttl/ttljob/ttljob_test: TestRowLevelTTLJobRandomEntries failed HOT 1
- pkg/ccl/testccl/workload/schemachange/schemachange_test: TestWorkload failed HOT 1
- : failed HOT 1
- roachtest: failover/chaos/read-write/lease=expiration failed
- kv/kvnemesis: TestKVNemesisMultiNode failed HOT 1
- util/admission: TestWorkQueueBasic failed HOT 2
- roachtest: sysbench/oltp_read_write/nodes=3/cpu=32/conc=256 failed
- : failed HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.