Comments (8)
I'm going to see if I can reproduce this with #122056.
from cockroach.
This is reproducing 1/10,000 failures. Interestingly, it's not hitting a validation failure even with lock promotion validation in there. I'll switch debugging tacts here.
from cockroach.
It's readily reproducible under deadlock. Debugging using #122009.
from cockroach.
It's readily reproducible under deadlock.
This turned out to be a red herring. Turns out, this test is extremely susceptible to overload. If I don't constrain the number of parallel runs (using --cpu
), we see failures on release-23.2
as well.
I haven't been able to piece the entire story together with the logging I've added. Most failures take the form where all requests are queued behind a single lock holder. For example:
Lock Table
num=10
lock: "a"
holder: txn: 00000000-0000-0000-0000-000000000194 epoch: 0, iso: Serializable, ts: 0.000000002,0, info: unrepl [(str: Exclusive seq: 0)]
queued locking requests:
active: true req: 388, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fd
lock: "b"
queued locking requests:
active: false req: 389, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fe
active: true req: 402, strength: Intent, txn: 00000000-0000-0000-0000-000000000107
active: true req: 403, strength: Intent, txn: 00000000-0000-0000-0000-000000000108
active: true req: 407, strength: Intent, txn: 00000000-0000-0000-0000-00000000010b
active: true req: 418, strength: Intent, txn: 00000000-0000-0000-0000-000000000112
active: true req: 436, strength: Intent, txn: 00000000-0000-0000-0000-00000000011b
active: true req: 441, strength: Intent, txn: 00000000-0000-0000-0000-000000000115
active: true req: 456, strength: Intent, txn: 00000000-0000-0000-0000-000000000122
active: true req: 469, strength: Intent, txn: 00000000-0000-0000-0000-000000000130
active: true req: 476, strength: Intent, txn: 00000000-0000-0000-0000-000000000125
active: true req: 499, strength: Intent, txn: 00000000-0000-0000-0000-000000000142
active: true req: 506, strength: Intent, txn: 00000000-0000-0000-0000-000000000146
active: true req: 514, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000014b
active: true req: 520, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000014e
active: true req: 554, strength: Intent, txn: 00000000-0000-0000-0000-000000000167
active: true req: 558, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000169
active: true req: 560, strength: Intent, txn: 00000000-0000-0000-0000-000000000168
active: true req: 561, strength: Intent, txn: 00000000-0000-0000-0000-000000000168
active: true req: 563, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000016b
active: true req: 620, strength: Intent, txn: 00000000-0000-0000-0000-00000000018f
active: true req: 630, strength: Intent, txn: 00000000-0000-0000-0000-000000000190
lock: "c"
queued locking requests:
active: false req: 417, strength: Intent, txn: 00000000-0000-0000-0000-000000000111
active: true req: 428, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000119
active: true req: 431, strength: Shared, txn: 00000000-0000-0000-0000-00000000011a
active: true req: 449, strength: Intent, txn: 00000000-0000-0000-0000-000000000123
active: true req: 477, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000133
active: true req: 486, strength: Intent, txn: 00000000-0000-0000-0000-00000000012c
active: true req: 511, strength: Intent, txn: 00000000-0000-0000-0000-000000000141
active: true req: 513, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000014a
active: true req: 556, strength: Intent, txn: 00000000-0000-0000-0000-000000000167
active: true req: 562, strength: Intent, txn: 00000000-0000-0000-0000-000000000167
active: true req: 565, strength: Intent, txn: 00000000-0000-0000-0000-00000000016c
active: true req: 570, strength: Intent, txn: 00000000-0000-0000-0000-000000000171
active: true req: 575, strength: Intent, txn: 00000000-0000-0000-0000-000000000174
active: true req: 588, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000017d
active: true req: 607, strength: Intent, txn: 00000000-0000-0000-0000-00000000017f
active: true req: 619, strength: Intent, txn: 00000000-0000-0000-0000-000000000183
active: true req: 626, strength: Intent, txn: 00000000-0000-0000-0000-000000000184
lock: "d"
queued locking requests:
active: false req: 389, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fe
active: true req: 417, strength: Intent, txn: 00000000-0000-0000-0000-000000000111
active: false req: 429, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106
active: false req: 431, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000011a
active: true req: 432, strength: Intent, txn: 00000000-0000-0000-0000-000000000106
active: true req: 443, strength: Shared, txn: 00000000-0000-0000-0000-00000000011f
active: true req: 452, strength: Intent, txn: 00000000-0000-0000-0000-000000000125
active: true req: 471, strength: Shared, txn: 00000000-0000-0000-0000-00000000012c
active: true req: 478, strength: Intent, txn: 00000000-0000-0000-0000-000000000134
active: true req: 484, strength: Intent, txn: 00000000-0000-0000-0000-000000000139
active: true req: 497, strength: Intent, txn: 00000000-0000-0000-0000-00000000013a
active: true req: 517, strength: Intent, txn: 00000000-0000-0000-0000-00000000014c
active: true req: 521, strength: Intent, txn: 00000000-0000-0000-0000-000000000149
active: true req: 523, strength: Intent, txn: 00000000-0000-0000-0000-000000000150
active: true req: 537, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000015a
active: true req: 541, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000015d
active: true req: 568, strength: Intent, txn: 00000000-0000-0000-0000-00000000016f
active: true req: 573, strength: Shared, txn: 00000000-0000-0000-0000-000000000168
active: true req: 582, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000178
active: true req: 618, strength: Intent, txn: 00000000-0000-0000-0000-00000000018e
active: true req: 633, strength: Intent, txn: 00000000-0000-0000-0000-000000000197
lock: "e"
queued locking requests:
active: false req: 389, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fe
active: false req: 396, strength: Intent, txn: 00000000-0000-0000-0000-000000000103
active: true req: 401, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106
active: true req: 406, strength: Intent, txn: 00000000-0000-0000-0000-00000000010a
active: true req: 425, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000117
active: true req: 429, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106
active: true req: 435, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000011c
active: true req: 444, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000120
active: true req: 474, strength: Intent, txn: 00000000-0000-0000-0000-000000000131
active: true req: 501, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000141
active: true req: 505, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000145
active: true req: 539, strength: Intent, txn: 00000000-0000-0000-0000-00000000015b
active: true req: 549, strength: Intent, txn: 00000000-0000-0000-0000-000000000162
active: true req: 567, strength: Intent, txn: 00000000-0000-0000-0000-00000000016e
active: true req: 569, strength: Intent, txn: 00000000-0000-0000-0000-000000000170
active: true req: 589, strength: Intent, txn: 00000000-0000-0000-0000-00000000017e
active: true req: 616, strength: Intent, txn: 00000000-0000-0000-0000-00000000018d
active: true req: 623, strength: Intent, txn: 00000000-0000-0000-0000-000000000189
active: true req: 629, strength: Intent, txn: 00000000-0000-0000-0000-000000000193
lock: "f"
queued locking requests:
active: false req: 388, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fd
active: true req: 389, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fe
active: true req: 395, strength: Intent, txn: 00000000-0000-0000-0000-000000000101
active: true req: 396, strength: Shared, txn: 00000000-0000-0000-0000-000000000103
active: true req: 399, strength: Intent, txn: 00000000-0000-0000-0000-000000000104
active: true req: 410, strength: Intent, txn: 00000000-0000-0000-0000-00000000010c
active: true req: 411, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106
active: true req: 419, strength: Intent, txn: 00000000-0000-0000-0000-00000000010d
active: true req: 420, strength: Intent, txn: 00000000-0000-0000-0000-000000000112
active: true req: 433, strength: Intent, txn: 00000000-0000-0000-0000-000000000119
active: true req: 437, strength: Intent, txn: 00000000-0000-0000-0000-000000000115
active: true req: 439, strength: Intent, txn: 00000000-0000-0000-0000-00000000011d
active: true req: 447, strength: Intent, txn: 00000000-0000-0000-0000-000000000122
active: true req: 450, strength: Intent, txn: 00000000-0000-0000-0000-000000000124
active: true req: 455, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000126
active: true req: 458, strength: Intent, txn: 00000000-0000-0000-0000-000000000125
active: true req: 462, strength: Intent, txn: 00000000-0000-0000-0000-00000000012c
active: true req: 468, strength: Shared, txn: 00000000-0000-0000-0000-00000000012f
active: true req: 480, strength: Intent, txn: 00000000-0000-0000-0000-000000000135
active: true req: 489, strength: Intent, txn: 00000000-0000-0000-0000-000000000139
active: true req: 496, strength: Shared, txn: 00000000-0000-0000-0000-000000000140
active: true req: 507, strength: Intent, txn: 00000000-0000-0000-0000-000000000144
active: true req: 508, strength: Intent, txn: 00000000-0000-0000-0000-000000000147
active: true req: 510, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000149
active: true req: 518, strength: Shared, txn: 00000000-0000-0000-0000-00000000014c
active: true req: 522, strength: Intent, txn: 00000000-0000-0000-0000-00000000014f
active: true req: 527, strength: Intent, txn: 00000000-0000-0000-0000-000000000152
active: true req: 529, strength: Intent, txn: 00000000-0000-0000-0000-000000000153
active: true req: 530, strength: Shared, txn: 00000000-0000-0000-0000-000000000154
active: true req: 534, strength: Intent, txn: 00000000-0000-0000-0000-000000000157
active: true req: 536, strength: Intent, txn: 00000000-0000-0000-0000-000000000159
active: true req: 543, strength: Intent, txn: 00000000-0000-0000-0000-00000000015f
active: true req: 547, strength: Intent, txn: 00000000-0000-0000-0000-000000000160
active: true req: 550, strength: Intent, txn: 00000000-0000-0000-0000-000000000163
active: true req: 551, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000164
active: true req: 552, strength: Intent, txn: 00000000-0000-0000-0000-000000000165
active: true req: 572, strength: Intent, txn: 00000000-0000-0000-0000-000000000168
active: true req: 583, strength: Intent, txn: 00000000-0000-0000-0000-000000000179
active: true req: 597, strength: Shared, txn: 00000000-0000-0000-0000-000000000176
lock: "g"
queued locking requests:
active: false req: 389, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fe
active: true req: 393, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000100
active: true req: 421, strength: Intent, txn: 00000000-0000-0000-0000-000000000113
active: true req: 442, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000011e
active: true req: 445, strength: Shared, txn: 00000000-0000-0000-0000-000000000121
active: true req: 457, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000127
active: true req: 466, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000012d
active: true req: 467, strength: Intent, txn: 00000000-0000-0000-0000-00000000012e
active: true req: 473, strength: Shared, txn: 00000000-0000-0000-0000-00000000012c
active: true req: 481, strength: Shared, txn: 00000000-0000-0000-0000-000000000136
active: true req: 483, strength: Intent, txn: 00000000-0000-0000-0000-000000000137
active: true req: 503, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000144
active: true req: 512, strength: Intent, txn: 00000000-0000-0000-0000-000000000142
active: true req: 553, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000166
active: true req: 555, strength: Intent, txn: 00000000-0000-0000-0000-000000000168
active: true req: 578, strength: Intent, txn: 00000000-0000-0000-0000-000000000175
active: true req: 584, strength: Intent, txn: 00000000-0000-0000-0000-00000000017a
active: true req: 596, strength: Intent, txn: 00000000-0000-0000-0000-000000000183
active: true req: 599, strength: Intent, txn: 00000000-0000-0000-0000-000000000184
active: true req: 603, strength: Intent, txn: 00000000-0000-0000-0000-000000000176
active: true req: 612, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000018a
active: true req: 622, strength: Shared, txn: 00000000-0000-0000-0000-000000000190
active: true req: 632, strength: Intent, txn: 00000000-0000-0000-0000-000000000196
active: true req: 634, strength: Intent, txn: 00000000-0000-0000-0000-000000000198
lock: "h"
queued locking requests:
active: false req: 389, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fe
active: false req: 417, strength: Intent, txn: 00000000-0000-0000-0000-000000000111
active: false req: 429, strength: Intent, txn: 00000000-0000-0000-0000-000000000106
active: true req: 438, strength: Intent, txn: 00000000-0000-0000-0000-000000000115
active: true req: 448, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000121
active: true req: 460, strength: Intent, txn: 00000000-0000-0000-0000-000000000129
active: true req: 461, strength: Intent, txn: 00000000-0000-0000-0000-00000000012a
active: true req: 463, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000012b
active: true req: 487, strength: Intent, txn: 00000000-0000-0000-0000-00000000013a
active: true req: 524, strength: Shared, txn: 00000000-0000-0000-0000-00000000014c
active: true req: 526, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000152
active: true req: 574, strength: Intent, txn: 00000000-0000-0000-0000-000000000173
active: true req: 585, strength: Intent, txn: 00000000-0000-0000-0000-000000000172
active: true req: 587, strength: Intent, txn: 00000000-0000-0000-0000-00000000017c
active: true req: 594, strength: Intent, txn: 00000000-0000-0000-0000-00000000016a
active: true req: 604, strength: Intent, txn: 00000000-0000-0000-0000-00000000017f
active: true req: 627, strength: Intent, txn: 00000000-0000-0000-0000-00000000018b
active: true req: 631, strength: Intent, txn: 00000000-0000-0000-0000-000000000195
lock: "i"
queued locking requests:
active: false req: 388, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fd
active: true req: 392, strength: Intent, txn: 00000000-0000-0000-0000-0000000000fe
active: true req: 405, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000109
active: true req: 415, strength: Intent, txn: 00000000-0000-0000-0000-00000000010f
active: true req: 423, strength: Intent, txn: 00000000-0000-0000-0000-000000000115
active: true req: 426, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000118
active: true req: 446, strength: Intent, txn: 00000000-0000-0000-0000-000000000122
active: true req: 470, strength: Intent, txn: 00000000-0000-0000-0000-00000000012f
active: true req: 472, strength: Intent, txn: 00000000-0000-0000-0000-00000000012f
active: true req: 475, strength: Intent, txn: 00000000-0000-0000-0000-000000000132
active: true req: 485, strength: Intent, txn: 00000000-0000-0000-0000-000000000131
active: true req: 488, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000012c
active: true req: 490, strength: Intent, txn: 00000000-0000-0000-0000-00000000013b
active: true req: 494, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000013e
active: true req: 500, strength: Intent, txn: 00000000-0000-0000-0000-000000000120
active: true req: 502, strength: Intent, txn: 00000000-0000-0000-0000-000000000143
active: true req: 509, strength: Intent, txn: 00000000-0000-0000-0000-000000000148
active: true req: 519, strength: Intent, txn: 00000000-0000-0000-0000-00000000014d
active: true req: 533, strength: Intent, txn: 00000000-0000-0000-0000-000000000156
active: true req: 535, strength: Intent, txn: 00000000-0000-0000-0000-000000000158
active: true req: 540, strength: Intent, txn: 00000000-0000-0000-0000-00000000015c
active: true req: 542, strength: Intent, txn: 00000000-0000-0000-0000-00000000015e
active: true req: 544, strength: Intent, txn: 00000000-0000-0000-0000-000000000160
active: true req: 545, strength: Intent, txn: 00000000-0000-0000-0000-000000000161
active: true req: 546, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000161
active: true req: 548, strength: Intent, txn: 00000000-0000-0000-0000-00000000015f
active: true req: 559, strength: Intent, txn: 00000000-0000-0000-0000-00000000016a
active: true req: 566, strength: Intent, txn: 00000000-0000-0000-0000-00000000016d
active: true req: 586, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000017b
active: true req: 592, strength: Intent, txn: 00000000-0000-0000-0000-000000000180
active: true req: 593, strength: Shared, txn: 00000000-0000-0000-0000-000000000182
active: true req: 609, strength: Intent, txn: 00000000-0000-0000-0000-000000000187
active: true req: 611, strength: Intent, txn: 00000000-0000-0000-0000-000000000189
active: true req: 614, strength: Intent, txn: 00000000-0000-0000-0000-00000000018a
active: true req: 617, strength: Intent, txn: 00000000-0000-0000-0000-00000000018d
active: true req: 624, strength: Intent, txn: 00000000-0000-0000-0000-000000000191
active: true req: 625, strength: Intent, txn: 00000000-0000-0000-0000-000000000192
lock: "j"
queued locking requests:
active: false req: 389, strength: Intent, txn: 00000000-0000-0000-0000-0000000000fe
active: true req: 391, strength: Intent, txn: 00000000-0000-0000-0000-0000000000ff
active: true req: 413, strength: Intent, txn: 00000000-0000-0000-0000-00000000010d
active: true req: 414, strength: Intent, txn: 00000000-0000-0000-0000-00000000010e
active: true req: 416, strength: Intent, txn: 00000000-0000-0000-0000-000000000110
active: true req: 424, strength: Intent, txn: 00000000-0000-0000-0000-000000000116
active: true req: 493, strength: Shared, txn: 00000000-0000-0000-0000-00000000013c
active: true req: 495, strength: Intent, txn: 00000000-0000-0000-0000-00000000013f
active: true req: 498, strength: Shared, txn: 00000000-0000-0000-0000-000000000141
active: true req: 531, strength: Intent, txn: 00000000-0000-0000-0000-000000000155
active: true req: 557, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000160
active: true req: 571, strength: Intent, txn: 00000000-0000-0000-0000-000000000172
active: true req: 577, strength: Intent, txn: 00000000-0000-0000-0000-000000000174
active: true req: 581, strength: Intent, txn: 00000000-0000-0000-0000-000000000177
active: true req: 590, strength: Shared, txn: 00000000-0000-0000-0000-00000000017f
active: true req: 591, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000181
active: true req: 610, strength: Intent, txn: 00000000-0000-0000-0000-000000000188
active: true req: 615, strength: Intent, txn: 00000000-0000-0000-0000-00000000018b
We see that all requests are blocked behind req 388, which itself is waiting on the lock held by txn 000000000194
. In all my reproductions, the transaction in question that holds the lock has been finalized previously. Moreover, its locks have also been resolved, and resolved correctly, as judging by the lock table's state. How is this lock appearing in the lock table again?
One possibility would be if the test is slow and there's a "lagging" lock acquisition that lands after the locks have been resolved. But that doesn't seem plausible given we won't kick off intent resolution if there are in-flight requests still pending. See:
cockroach/pkg/kv/kvserver/concurrency/lock_table_test.go
Lines 1363 to 1368 in cead8be
The test also uses monotonic transaction IDs, so it's not possible for us to be creating a new transaction with the same ID.
Right now, I'm simply failing the test if I find a request is waiting for > 4 minutes. One thread to pull on would be to check introspect the waitingState
on the request. I also want to add some sanity checks around known-to-be-finalized transactions.
All this being said, it seems like we're ending up in a state where the lock is held by a transaction that has been abandoned. If we were to end up in such a situation in reality, we'd detect and handle it correctly when the lock holder is pushed. Given this, and how rare[*] this is (outside of an overloaded test), I'm inclined to downgrade this to a GA-blocker. I still want to understand this a bit better before we're fully confident here. (cc @nvanbenschoten)
[*] For context, I've given this over 150K runs on my GCE worker today, without running under --deadlock
, and I've only seen it fail once.
from cockroach.
kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ f117eea22dd7be380c7141cdf6cd7aba92dd9c70:
Fatal error:
panic: test timed out after 24m57s
running tests:
TestLockTableConcurrentRequests (24m57s)
TestLockTableConcurrentRequests/concurrency_8 (24m54s)
Stack:
goroutine 833581 [running]:
testing.(*M).startAlarm.func1()
GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error
=== RUN TestLockTableConcurrentRequests
test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests2351147515
test_log_scope.go:81: use -show-logs to present logs inline
=== RUN TestLockTableConcurrentRequests/concurrency_2
items: 1692, aborted: 0, concurrency violations: 758, lock table: num=0
=== RUN TestLockTableConcurrentRequests/concurrency_8
Parameters:
attempt=1
deadlock=true
run=3
shard=6
This test on roachdash | Improve this report!
from cockroach.
kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 67547d7724f8a52646e2e8ecb3ca48b923957d90:
Fatal error:
panic: test timed out after 24m57s
running tests:
TestLockTableConcurrentRequests (24m57s)
TestLockTableConcurrentRequests/concurrency_8 (24m55s)
Stack:
goroutine 1033150 [running]:
testing.(*M).startAlarm.func1()
GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error
=== RUN TestLockTableConcurrentRequests
test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests3274046890
test_log_scope.go:81: use -show-logs to present logs inline
=== RUN TestLockTableConcurrentRequests/concurrency_2
items: 1671, aborted: 0, concurrency violations: 661, lock table: num=0
=== RUN TestLockTableConcurrentRequests/concurrency_8
Parameters:
attempt=1
deadlock=true
run=2
shard=6
This test on roachdash | Improve this report!
from cockroach.
kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on release-24.1 @ 6e3e9e6012dc04dabc64333a87e77a80d9f9d46b:
Fatal error:
panic: test timed out after 24m57s
running tests:
TestLockTableConcurrentRequests (24m57s)
TestLockTableConcurrentRequests/concurrency_16 (24m39s)
Stack:
goroutine 2588996 [running]:
testing.(*M).startAlarm.func1()
GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error
=== RUN TestLockTableConcurrentRequests
test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests1983242438
test_log_scope.go:81: use -show-logs to present logs inline
=== RUN TestLockTableConcurrentRequests/concurrency_16
Parameters:
attempt=1
deadlock=true
run=2
shard=6
This test on roachdash | Improve this report!
from cockroach.
kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 8daa6933836c8ce1108fb22bf6663117fc71599b:
Fatal error:
panic: test timed out after 24m57s
running tests:
TestLockTableConcurrentRequests (24m57s)
TestLockTableConcurrentRequests/concurrency_16 (24m46s)
Stack:
goroutine 2996049 [running]:
testing.(*M).startAlarm.func1()
GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
GOROOT/src/time/sleep.go:177 +0x2d
Log preceding fatal error
=== RUN TestLockTableConcurrentRequests
test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests4126696952
test_log_scope.go:81: use -show-logs to present logs inline
=== RUN TestLockTableConcurrentRequests/concurrency_2
items: 1677, aborted: 0, concurrency violations: 861, lock table: num=0
=== RUN TestLockTableConcurrentRequests/concurrency_8
items: 1677, aborted: 0, concurrency violations: 857, lock table: num=0
=== RUN TestLockTableConcurrentRequests/concurrency_16
Parameters:
attempt=1
deadlock=true
run=1
shard=6
This test on roachdash | Improve this report!
from cockroach.
Related Issues (20)
- sql/pgwire/pgwirecancel: TestCancelQueryOtherNode failed HOT 1
- pkg/server/storage_api/storage_api_test: TestLivenessAPI failed HOT 1
- ccl/changefeedccl: TestChangefeedWithNoDistributionStrategyAndConstrainedLocality failed HOT 1
- ccl/changefeedccl: TestChangefeedWithSimpleDistributionStrategyAndConstrainedLocality failed HOT 1
- sql/physicalplan: TestFakeSpanResolver failed HOT 1
- pkg/sql/sem/eval/cast_test/cast_test_test: TestAssignmentCastsMatchPostgres failed HOT 1
- ccl/revertccl: TestRevertSpansFanout failed HOT 1
- pkg/ccl/workloadccl/workloadccl_test: TestImportFixtureNodeCount failed
- sql/stats: TestCacheAutoRefresh failed
- sql/importer: TestMultiNodeExportStmt failed HOT 1
- sql/importer: TestImportDefault failed HOT 1
- sql/importer: TestUniqueUUID failed HOT 1
- sql/importer: TestImportDefaultNextVal failed HOT 1
- sql/importer: TestImportIntoMysql failed HOT 1
- sql/importer: TestImportComputed failed HOT 1
- sql/importer: TestImportDelimited failed HOT 1
- ccl/multitenantccl/tenantcostclient: TestScheduledJobsConsumption failed HOT 20
- roachtest: [st_minimumclearanceline caused SIGSEGV] sqlsmith/setup=rand-tables/setting=no-mutations failed HOT 3
- ccl/backupccl: TestDataDriven_user_defined_types failed
- pkg/sql/logictest/tests/5node/5node_test: TestLogic_distsql_stats failed HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.