kv/kvserver/concurrency.TestLockTableConcurrentRequests <a href="https://tanzanite.clu

I'm going to see if I can reproduce this with <a class="issue-link js-issue-link" data

It's readily reproducible under deadlock. Debugging using <a class="issue-link js-issu

It's readily reproducible under deadlock. <p dir="auto"

kv/kvserver/concurrency: TestLockTableConcurrentRequests failed,about cockroachdb/cockroach

Comments (8)

arulajmani commented on August 26, 2024

I'm going to see if I can reproduce this with #122056.

from cockroach.

arulajmani commented on August 26, 2024

This is reproducing 1/10,000 failures. Interestingly, it's not hitting a validation failure even with lock promotion validation in there. I'll switch debugging tacts here.

from cockroach.

arulajmani commented on August 26, 2024

It's readily reproducible under deadlock. Debugging using #122009.

from cockroach.

arulajmani commented on August 26, 2024

It's readily reproducible under deadlock.

This turned out to be a red herring. Turns out, this test is extremely susceptible to overload. If I don't constrain the number of parallel runs (using --cpu), we see failures on release-23.2 as well.

I haven't been able to piece the entire story together with the logging I've added. Most failures take the form where all requests are queued behind a single lock holder. For example:

Lock Table

            num=10
             lock: "a"
              holder: txn: 00000000-0000-0000-0000-000000000194 epoch: 0, iso: Serializable, ts: 0.000000002,0, info: unrepl [(str: Exclusive seq: 0)]
               queued locking requests:
                active: true req: 388, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fd
             lock: "b"
               queued locking requests:
                active: false req: 389, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fe
                active: true req: 402, strength: Intent, txn: 00000000-0000-0000-0000-000000000107
                active: true req: 403, strength: Intent, txn: 00000000-0000-0000-0000-000000000108
                active: true req: 407, strength: Intent, txn: 00000000-0000-0000-0000-00000000010b
                active: true req: 418, strength: Intent, txn: 00000000-0000-0000-0000-000000000112
                active: true req: 436, strength: Intent, txn: 00000000-0000-0000-0000-00000000011b
                active: true req: 441, strength: Intent, txn: 00000000-0000-0000-0000-000000000115
                active: true req: 456, strength: Intent, txn: 00000000-0000-0000-0000-000000000122
                active: true req: 469, strength: Intent, txn: 00000000-0000-0000-0000-000000000130
                active: true req: 476, strength: Intent, txn: 00000000-0000-0000-0000-000000000125
                active: true req: 499, strength: Intent, txn: 00000000-0000-0000-0000-000000000142
                active: true req: 506, strength: Intent, txn: 00000000-0000-0000-0000-000000000146
                active: true req: 514, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000014b
                active: true req: 520, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000014e
                active: true req: 554, strength: Intent, txn: 00000000-0000-0000-0000-000000000167
                active: true req: 558, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000169
                active: true req: 560, strength: Intent, txn: 00000000-0000-0000-0000-000000000168
                active: true req: 561, strength: Intent, txn: 00000000-0000-0000-0000-000000000168
                active: true req: 563, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000016b
                active: true req: 620, strength: Intent, txn: 00000000-0000-0000-0000-00000000018f
                active: true req: 630, strength: Intent, txn: 00000000-0000-0000-0000-000000000190
             lock: "c"
               queued locking requests:
                active: false req: 417, strength: Intent, txn: 00000000-0000-0000-0000-000000000111
                active: true req: 428, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000119
                active: true req: 431, strength: Shared, txn: 00000000-0000-0000-0000-00000000011a
                active: true req: 449, strength: Intent, txn: 00000000-0000-0000-0000-000000000123
                active: true req: 477, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000133
                active: true req: 486, strength: Intent, txn: 00000000-0000-0000-0000-00000000012c
                active: true req: 511, strength: Intent, txn: 00000000-0000-0000-0000-000000000141
                active: true req: 513, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000014a
                active: true req: 556, strength: Intent, txn: 00000000-0000-0000-0000-000000000167
                active: true req: 562, strength: Intent, txn: 00000000-0000-0000-0000-000000000167
                active: true req: 565, strength: Intent, txn: 00000000-0000-0000-0000-00000000016c
                active: true req: 570, strength: Intent, txn: 00000000-0000-0000-0000-000000000171
                active: true req: 575, strength: Intent, txn: 00000000-0000-0000-0000-000000000174
                active: true req: 588, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000017d
                active: true req: 607, strength: Intent, txn: 00000000-0000-0000-0000-00000000017f
                active: true req: 619, strength: Intent, txn: 00000000-0000-0000-0000-000000000183
                active: true req: 626, strength: Intent, txn: 00000000-0000-0000-0000-000000000184
             lock: "d"
               queued locking requests:
                active: false req: 389, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fe
                active: true req: 417, strength: Intent, txn: 00000000-0000-0000-0000-000000000111
                active: false req: 429, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106
                active: false req: 431, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000011a
                active: true req: 432, strength: Intent, txn: 00000000-0000-0000-0000-000000000106
                active: true req: 443, strength: Shared, txn: 00000000-0000-0000-0000-00000000011f
                active: true req: 452, strength: Intent, txn: 00000000-0000-0000-0000-000000000125
                active: true req: 471, strength: Shared, txn: 00000000-0000-0000-0000-00000000012c
                active: true req: 478, strength: Intent, txn: 00000000-0000-0000-0000-000000000134
                active: true req: 484, strength: Intent, txn: 00000000-0000-0000-0000-000000000139
                active: true req: 497, strength: Intent, txn: 00000000-0000-0000-0000-00000000013a
                active: true req: 517, strength: Intent, txn: 00000000-0000-0000-0000-00000000014c
                active: true req: 521, strength: Intent, txn: 00000000-0000-0000-0000-000000000149
                active: true req: 523, strength: Intent, txn: 00000000-0000-0000-0000-000000000150
                active: true req: 537, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000015a
                active: true req: 541, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000015d
                active: true req: 568, strength: Intent, txn: 00000000-0000-0000-0000-00000000016f
                active: true req: 573, strength: Shared, txn: 00000000-0000-0000-0000-000000000168
                active: true req: 582, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000178
                active: true req: 618, strength: Intent, txn: 00000000-0000-0000-0000-00000000018e
                active: true req: 633, strength: Intent, txn: 00000000-0000-0000-0000-000000000197
             lock: "e"
               queued locking requests:
                active: false req: 389, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fe
                active: false req: 396, strength: Intent, txn: 00000000-0000-0000-0000-000000000103
                active: true req: 401, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106
                active: true req: 406, strength: Intent, txn: 00000000-0000-0000-0000-00000000010a
                active: true req: 425, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000117
                active: true req: 429, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106
                active: true req: 435, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000011c
                active: true req: 444, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000120
                active: true req: 474, strength: Intent, txn: 00000000-0000-0000-0000-000000000131
                active: true req: 501, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000141
                active: true req: 505, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000145
                active: true req: 539, strength: Intent, txn: 00000000-0000-0000-0000-00000000015b
                active: true req: 549, strength: Intent, txn: 00000000-0000-0000-0000-000000000162
                active: true req: 567, strength: Intent, txn: 00000000-0000-0000-0000-00000000016e
                active: true req: 569, strength: Intent, txn: 00000000-0000-0000-0000-000000000170
                active: true req: 589, strength: Intent, txn: 00000000-0000-0000-0000-00000000017e
                active: true req: 616, strength: Intent, txn: 00000000-0000-0000-0000-00000000018d
                active: true req: 623, strength: Intent, txn: 00000000-0000-0000-0000-000000000189
                active: true req: 629, strength: Intent, txn: 00000000-0000-0000-0000-000000000193
             lock: "f"
               queued locking requests:
                active: false req: 388, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fd
                active: true req: 389, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fe
                active: true req: 395, strength: Intent, txn: 00000000-0000-0000-0000-000000000101
                active: true req: 396, strength: Shared, txn: 00000000-0000-0000-0000-000000000103
                active: true req: 399, strength: Intent, txn: 00000000-0000-0000-0000-000000000104
                active: true req: 410, strength: Intent, txn: 00000000-0000-0000-0000-00000000010c
                active: true req: 411, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000106
                active: true req: 419, strength: Intent, txn: 00000000-0000-0000-0000-00000000010d
                active: true req: 420, strength: Intent, txn: 00000000-0000-0000-0000-000000000112
                active: true req: 433, strength: Intent, txn: 00000000-0000-0000-0000-000000000119
                active: true req: 437, strength: Intent, txn: 00000000-0000-0000-0000-000000000115
                active: true req: 439, strength: Intent, txn: 00000000-0000-0000-0000-00000000011d
                active: true req: 447, strength: Intent, txn: 00000000-0000-0000-0000-000000000122
                active: true req: 450, strength: Intent, txn: 00000000-0000-0000-0000-000000000124
                active: true req: 455, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000126
                active: true req: 458, strength: Intent, txn: 00000000-0000-0000-0000-000000000125
                active: true req: 462, strength: Intent, txn: 00000000-0000-0000-0000-00000000012c
                active: true req: 468, strength: Shared, txn: 00000000-0000-0000-0000-00000000012f
                active: true req: 480, strength: Intent, txn: 00000000-0000-0000-0000-000000000135
                active: true req: 489, strength: Intent, txn: 00000000-0000-0000-0000-000000000139
                active: true req: 496, strength: Shared, txn: 00000000-0000-0000-0000-000000000140
                active: true req: 507, strength: Intent, txn: 00000000-0000-0000-0000-000000000144
                active: true req: 508, strength: Intent, txn: 00000000-0000-0000-0000-000000000147
                active: true req: 510, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000149
                active: true req: 518, strength: Shared, txn: 00000000-0000-0000-0000-00000000014c
                active: true req: 522, strength: Intent, txn: 00000000-0000-0000-0000-00000000014f
                active: true req: 527, strength: Intent, txn: 00000000-0000-0000-0000-000000000152
                active: true req: 529, strength: Intent, txn: 00000000-0000-0000-0000-000000000153
                active: true req: 530, strength: Shared, txn: 00000000-0000-0000-0000-000000000154
                active: true req: 534, strength: Intent, txn: 00000000-0000-0000-0000-000000000157
                active: true req: 536, strength: Intent, txn: 00000000-0000-0000-0000-000000000159
                active: true req: 543, strength: Intent, txn: 00000000-0000-0000-0000-00000000015f
                active: true req: 547, strength: Intent, txn: 00000000-0000-0000-0000-000000000160
                active: true req: 550, strength: Intent, txn: 00000000-0000-0000-0000-000000000163
                active: true req: 551, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000164
                active: true req: 552, strength: Intent, txn: 00000000-0000-0000-0000-000000000165
                active: true req: 572, strength: Intent, txn: 00000000-0000-0000-0000-000000000168
                active: true req: 583, strength: Intent, txn: 00000000-0000-0000-0000-000000000179
                active: true req: 597, strength: Shared, txn: 00000000-0000-0000-0000-000000000176
             lock: "g"
               queued locking requests:
                active: false req: 389, strength: Exclusive, txn: 00000000-0000-0000-0000-0000000000fe
                active: true req: 393, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000100
                active: true req: 421, strength: Intent, txn: 00000000-0000-0000-0000-000000000113
                active: true req: 442, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000011e
                active: true req: 445, strength: Shared, txn: 00000000-0000-0000-0000-000000000121
                active: true req: 457, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000127
                active: true req: 466, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000012d
                active: true req: 467, strength: Intent, txn: 00000000-0000-0000-0000-00000000012e
                active: true req: 473, strength: Shared, txn: 00000000-0000-0000-0000-00000000012c
                active: true req: 481, strength: Shared, txn: 00000000-0000-0000-0000-000000000136
                active: true req: 483, strength: Intent, txn: 00000000-0000-0000-0000-000000000137
                active: true req: 503, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000144
                active: true req: 512, strength: Intent, txn: 00000000-0000-0000-0000-000000000142
                active: true req: 553, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000166
                active: true req: 555, strength: Intent, txn: 00000000-0000-0000-0000-000000000168
                active: true req: 578, strength: Intent, txn: 00000000-0000-0000-0000-000000000175
                active: true req: 584, strength: Intent, txn: 00000000-0000-0000-0000-00000000017a
                active: true req: 596, strength: Intent, txn: 00000000-0000-0000-0000-000000000183
                active: true req: 599, strength: Intent, txn: 00000000-0000-0000-0000-000000000184
                active: true req: 603, strength: Intent, txn: 00000000-0000-0000-0000-000000000176
                active: true req: 612, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000018a
                active: true req: 622, strength: Shared, txn: 00000000-0000-0000-0000-000000000190
                active: true req: 632, strength: Intent, txn: 00000000-0000-0000-0000-000000000196
                active: true req: 634, strength: Intent, txn: 00000000-0000-0000-0000-000000000198
             lock: "h"
               queued locking requests:
                active: false req: 389, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fe
                active: false req: 417, strength: Intent, txn: 00000000-0000-0000-0000-000000000111
                active: false req: 429, strength: Intent, txn: 00000000-0000-0000-0000-000000000106
                active: true req: 438, strength: Intent, txn: 00000000-0000-0000-0000-000000000115
                active: true req: 448, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000121
                active: true req: 460, strength: Intent, txn: 00000000-0000-0000-0000-000000000129
                active: true req: 461, strength: Intent, txn: 00000000-0000-0000-0000-00000000012a
                active: true req: 463, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000012b
                active: true req: 487, strength: Intent, txn: 00000000-0000-0000-0000-00000000013a
                active: true req: 524, strength: Shared, txn: 00000000-0000-0000-0000-00000000014c
                active: true req: 526, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000152
                active: true req: 574, strength: Intent, txn: 00000000-0000-0000-0000-000000000173
                active: true req: 585, strength: Intent, txn: 00000000-0000-0000-0000-000000000172
                active: true req: 587, strength: Intent, txn: 00000000-0000-0000-0000-00000000017c
                active: true req: 594, strength: Intent, txn: 00000000-0000-0000-0000-00000000016a
                active: true req: 604, strength: Intent, txn: 00000000-0000-0000-0000-00000000017f
                active: true req: 627, strength: Intent, txn: 00000000-0000-0000-0000-00000000018b
                active: true req: 631, strength: Intent, txn: 00000000-0000-0000-0000-000000000195
             lock: "i"
               queued locking requests:
                active: false req: 388, strength: Shared, txn: 00000000-0000-0000-0000-0000000000fd
                active: true req: 392, strength: Intent, txn: 00000000-0000-0000-0000-0000000000fe
                active: true req: 405, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000109
                active: true req: 415, strength: Intent, txn: 00000000-0000-0000-0000-00000000010f
                active: true req: 423, strength: Intent, txn: 00000000-0000-0000-0000-000000000115
                active: true req: 426, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000118
                active: true req: 446, strength: Intent, txn: 00000000-0000-0000-0000-000000000122
                active: true req: 470, strength: Intent, txn: 00000000-0000-0000-0000-00000000012f
                active: true req: 472, strength: Intent, txn: 00000000-0000-0000-0000-00000000012f
                active: true req: 475, strength: Intent, txn: 00000000-0000-0000-0000-000000000132
                active: true req: 485, strength: Intent, txn: 00000000-0000-0000-0000-000000000131
                active: true req: 488, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000012c
                active: true req: 490, strength: Intent, txn: 00000000-0000-0000-0000-00000000013b
                active: true req: 494, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000013e
                active: true req: 500, strength: Intent, txn: 00000000-0000-0000-0000-000000000120
                active: true req: 502, strength: Intent, txn: 00000000-0000-0000-0000-000000000143
                active: true req: 509, strength: Intent, txn: 00000000-0000-0000-0000-000000000148
                active: true req: 519, strength: Intent, txn: 00000000-0000-0000-0000-00000000014d
                active: true req: 533, strength: Intent, txn: 00000000-0000-0000-0000-000000000156
                active: true req: 535, strength: Intent, txn: 00000000-0000-0000-0000-000000000158
                active: true req: 540, strength: Intent, txn: 00000000-0000-0000-0000-00000000015c
                active: true req: 542, strength: Intent, txn: 00000000-0000-0000-0000-00000000015e
                active: true req: 544, strength: Intent, txn: 00000000-0000-0000-0000-000000000160
                active: true req: 545, strength: Intent, txn: 00000000-0000-0000-0000-000000000161
                active: true req: 546, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000161
                active: true req: 548, strength: Intent, txn: 00000000-0000-0000-0000-00000000015f
                active: true req: 559, strength: Intent, txn: 00000000-0000-0000-0000-00000000016a
                active: true req: 566, strength: Intent, txn: 00000000-0000-0000-0000-00000000016d
                active: true req: 586, strength: Exclusive, txn: 00000000-0000-0000-0000-00000000017b
                active: true req: 592, strength: Intent, txn: 00000000-0000-0000-0000-000000000180
                active: true req: 593, strength: Shared, txn: 00000000-0000-0000-0000-000000000182
                active: true req: 609, strength: Intent, txn: 00000000-0000-0000-0000-000000000187
                active: true req: 611, strength: Intent, txn: 00000000-0000-0000-0000-000000000189
                active: true req: 614, strength: Intent, txn: 00000000-0000-0000-0000-00000000018a
                active: true req: 617, strength: Intent, txn: 00000000-0000-0000-0000-00000000018d
                active: true req: 624, strength: Intent, txn: 00000000-0000-0000-0000-000000000191
                active: true req: 625, strength: Intent, txn: 00000000-0000-0000-0000-000000000192
             lock: "j"
               queued locking requests:
                active: false req: 389, strength: Intent, txn: 00000000-0000-0000-0000-0000000000fe
                active: true req: 391, strength: Intent, txn: 00000000-0000-0000-0000-0000000000ff
                active: true req: 413, strength: Intent, txn: 00000000-0000-0000-0000-00000000010d
                active: true req: 414, strength: Intent, txn: 00000000-0000-0000-0000-00000000010e
                active: true req: 416, strength: Intent, txn: 00000000-0000-0000-0000-000000000110
                active: true req: 424, strength: Intent, txn: 00000000-0000-0000-0000-000000000116
                active: true req: 493, strength: Shared, txn: 00000000-0000-0000-0000-00000000013c
                active: true req: 495, strength: Intent, txn: 00000000-0000-0000-0000-00000000013f
                active: true req: 498, strength: Shared, txn: 00000000-0000-0000-0000-000000000141
                active: true req: 531, strength: Intent, txn: 00000000-0000-0000-0000-000000000155
                active: true req: 557, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000160
                active: true req: 571, strength: Intent, txn: 00000000-0000-0000-0000-000000000172
                active: true req: 577, strength: Intent, txn: 00000000-0000-0000-0000-000000000174
                active: true req: 581, strength: Intent, txn: 00000000-0000-0000-0000-000000000177
                active: true req: 590, strength: Shared, txn: 00000000-0000-0000-0000-00000000017f
                active: true req: 591, strength: Exclusive, txn: 00000000-0000-0000-0000-000000000181
                active: true req: 610, strength: Intent, txn: 00000000-0000-0000-0000-000000000188
                active: true req: 615, strength: Intent, txn: 00000000-0000-0000-0000-00000000018b

We see that all requests are blocked behind req 388, which itself is waiting on the lock held by txn 000000000194. In all my reproductions, the transaction in question that holds the lock has been finalized previously. Moreover, its locks have also been resolved, and resolved correctly, as judging by the lock table's state. How is this lock appearing in the lock table again?

One possibility would be if the test is slow and there's a "lagging" lock acquisition that lands after the locks have been resolved. But that doesn't seem plausible given we won't kick off intent resolution if there are in-flight requests still pending. See:

cockroach/pkg/kv/kvserver/concurrency/lock_table_test.go

Lines 1363 to 1368 in cead8be

 func (e *workloadExecutor) tryFinishTxn( 

 ctx context.Context, group *errgroup.Group, txnID uuid.UUID, tstate *transactionState, 

 ) bool { 

 if !tstate.finish || len(tstate.ongoingRequests) > 0 { 

 return false 

 }

The test also uses monotonic transaction IDs, so it's not possible for us to be creating a new transaction with the same ID.

Right now, I'm simply failing the test if I find a request is waiting for > 4 minutes. One thread to pull on would be to check introspect the waitingState on the request. I also want to add some sanity checks around known-to-be-finalized transactions.

All this being said, it seems like we're ending up in a state where the lock is held by a transaction that has been abandoned. If we were to end up in such a situation in reality, we'd detect and handle it correctly when the lock holder is pushed. Given this, and how rare[*] this is (outside of an overloaded test), I'm inclined to downgrade this to a GA-blocker. I still want to understand this a bit better before we're fully confident here. (cc @nvanbenschoten)

[*] For context, I've given this over 150K runs on my GCE worker today, without running under --deadlock, and I've only seen it fail once.

from cockroach.

cockroach-teamcity commented on August 26, 2024

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ f117eea22dd7be380c7141cdf6cd7aba92dd9c70:

Fatal error:

panic: test timed out after 24m57s
running tests:
	TestLockTableConcurrentRequests (24m57s)
	TestLockTableConcurrentRequests/concurrency_8 (24m54s)

Stack:

goroutine 833581 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

=== RUN   TestLockTableConcurrentRequests
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests2351147515
    test_log_scope.go:81: use -show-logs to present logs inline
=== RUN   TestLockTableConcurrentRequests/concurrency_2
items: 1692, aborted: 0, concurrency violations: 758, lock table: num=0

=== RUN   TestLockTableConcurrentRequests/concurrency_8

Parameters:

attempt=1
deadlock=true
run=3
shard=6

Help

_{This test on roachdash | Improve this report!}

from cockroach.

cockroach-teamcity commented on August 26, 2024

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 67547d7724f8a52646e2e8ecb3ca48b923957d90:

Fatal error:

panic: test timed out after 24m57s
running tests:
	TestLockTableConcurrentRequests (24m57s)
	TestLockTableConcurrentRequests/concurrency_8 (24m55s)

Stack:

goroutine 1033150 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

=== RUN   TestLockTableConcurrentRequests
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests3274046890
    test_log_scope.go:81: use -show-logs to present logs inline
=== RUN   TestLockTableConcurrentRequests/concurrency_2
items: 1671, aborted: 0, concurrency violations: 661, lock table: num=0

=== RUN   TestLockTableConcurrentRequests/concurrency_8

Parameters:

attempt=1
deadlock=true
run=2
shard=6

Help

_{This test on roachdash | Improve this report!}

from cockroach.

cockroach-teamcity commented on August 26, 2024

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on release-24.1 @ 6e3e9e6012dc04dabc64333a87e77a80d9f9d46b:

Fatal error:

panic: test timed out after 24m57s
running tests:
	TestLockTableConcurrentRequests (24m57s)
	TestLockTableConcurrentRequests/concurrency_16 (24m39s)

Stack:

goroutine 2588996 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

=== RUN   TestLockTableConcurrentRequests
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests1983242438
    test_log_scope.go:81: use -show-logs to present logs inline
=== RUN   TestLockTableConcurrentRequests/concurrency_16

Parameters:

attempt=1
deadlock=true
run=2
shard=6

Help

_{This test on roachdash | Improve this report!}

from cockroach.

cockroach-teamcity commented on August 26, 2024

kv/kvserver/concurrency.TestLockTableConcurrentRequests failed on master @ 8daa6933836c8ce1108fb22bf6663117fc71599b:

Fatal error:

panic: test timed out after 24m57s
running tests:
	TestLockTableConcurrentRequests (24m57s)
	TestLockTableConcurrentRequests/concurrency_16 (24m46s)

Stack:

goroutine 2996049 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

Log preceding fatal error

=== RUN   TestLockTableConcurrentRequests
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLockTableConcurrentRequests4126696952
    test_log_scope.go:81: use -show-logs to present logs inline
=== RUN   TestLockTableConcurrentRequests/concurrency_2
items: 1677, aborted: 0, concurrency violations: 861, lock table: num=0

=== RUN   TestLockTableConcurrentRequests/concurrency_8
items: 1677, aborted: 0, concurrency violations: 857, lock table: num=0

=== RUN   TestLockTableConcurrentRequests/concurrency_16

Parameters:

attempt=1
deadlock=true
run=1
shard=6

Help

_{This test on roachdash | Improve this report!}

from cockroach.

kv/kvserver/concurrency: TestLockTableConcurrentRequests failed about cockroach HOT 8 OPEN

Comments (8)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	func (e *workloadExecutor) tryFinishTxn(
	ctx context.Context, group errgroup.Group, txnID uuid.UUID, tstate transactionState,
	) bool {
	if !tstate.finish \|\| len(tstate.ongoingRequests) > 0 {
	return false
	}