roachtest: vm_preemption failed about cockroach HOT 78 OPEN

test cdc/initial-scan failed: VMs preempted during the test run: teamcity-15465864-1717134770-27-n4cpu16-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_072619.245450587_n4_workload-run-tpcc-wa.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan/run_1/ssh/ssh_072619.278401475_n4_workload-run-tpcc-wa.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(cdc.go:388).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan/run_1/ssh/ssh_073629.617407897_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/initial-scan/run_1

test cdc/ledger failed: VMs preempted during the test run: teamcity-15465864-1717134770-31-n4cpu16-0001 (us-east1-d), teamcity-15465864-1717134770-31-n4cpu16-0002 (us-east1-d), teamcity-15465864-1717134770-31-n4cpu16-0004 (us-east1-d), teamcity-15465864-1717134770-31-n4cpu16-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_073946.860657657_n4_workload-run-ledger-.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/ledger/run_1/ssh/ssh_073946.895215750_n4_workload-run-ledger-.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(ts_util.go:59).mustGetMetrics: roachtestutil.addCookie: unable to extract sessionID: roachtestutil.GetSessionID: failed to authenticate: context canceled
(cdc.go:388).Close: error shutting down prometheus/grafana: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/ledger/run_1/ssh/ssh_074757.875215665_n4_sudo-rm-rf-tmppromet.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/ledger/run_1

test kv0/enc=false/nodes=3/size=4kb failed: VMs preempted during the test run: teamcity-15465864-1717134770-88-n4cpu8-0002 (us-east1-c), teamcity-15465864-1717134770-88-n4cpu8-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/size=4kb/run_1

test kv95/enc=false/nodes=4 failed: VMs preempted during the test run: teamcity-15465864-1717134770-91-n5cpu8-0003 (us-east1-c), teamcity-15465864-1717134770-91-n5cpu8-0005 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=4/run_1

test kv95/enc=false/nodes=3/mt-shared-process failed: VMs preempted during the test run: teamcity-15465864-1717134770-90-n4cpu8-0002 (us-east1-d), teamcity-15465864-1717134770-90-n4cpu8-0004 (us-east1-d), teamcity-15465864-1717134770-90-n4cpu8-0001 (us-east1-d), teamcity-15465864-1717134770-90-n4cpu8-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/mt-shared-process/run_1

test ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15479078-1717220948-77-n4cpu32-0004 (us-east1-d), teamcity-15479078-1717220948-77-n4cpu32-0003 (us-east1-d), teamcity-15479078-1717220948-77-n4cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global/run_1

test replicate/rebalance/3to5 failed: VMs preempted during the test run: teamcity-15479078-1717220948-91-n6cpu4-0001 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/replicate/rebalance/3to5/run_1

test kv95/enc=false/nodes=3/mt-shared-process failed: VMs preempted during the test run: teamcity-15484213-1717307386-80-n4cpu8-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/mt-shared-process/run_1

test cdc/initial-scan failed: VMs preempted during the test run: teamcity-15491158-1717393677-25-n4cpu16-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_070255.847220535_n4_workload-run-tpcc-wa.log: COMMAND_PROBLEM: exit status 1
(cdc.go:388).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/initial-scan/run_1

test import/tpcc/warehouses=1000/nodes=4 failed: VMs preempted during the test run: teamcity-15504819-1717480196-57-n4cpu4-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpcc/warehouses=1000/nodes=4/run_1

test kv95/enc=false/nodes=4/ssds=8 failed: VMs preempted during the test run: teamcity-15504819-1717480196-84-n5cpu8-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=4/ssds=8/run_1

test kv95/enc=true/nodes=3 failed: VMs preempted during the test run: teamcity-15504819-1717480196-85-n4cpu8-0001 (us-east1-d), teamcity-15504819-1717480196-85-n4cpu8-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=true/nodes=3/run_1

test ycsb/E/nodes=3/cpu=32/isolation-level=read-committed failed: VMs preempted during the test run: teamcity-15504819-1717480196-77-n4cpu32-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/isolation-level=read-committed/run_1

test tpccbench/nodes=3/cpu=4 failed: VMs preempted during the test run: teamcity-15504819-1717480196-101-n4cpu4-0003 (us-east1-d), teamcity-15504819-1717480196-101-n4cpu4-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/run_1/ssh/ssh_131904.465579944_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/run_1

test cdc/initial-scan failed: VMs preempted during the test run: teamcity-15519805-1717566618-22-n4cpu16-0002 (us-east1-c), teamcity-15519805-1717566618-22-n4cpu16-0004 (us-east1-c), teamcity-15519805-1717566618-22-n4cpu16-0001 (us-east1-c), teamcity-15519805-1717566618-22-n4cpu16-0003 (us-east1-c)

**Other Failures:**
(cluster.go:2284).Start: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/initial-scan/run_1/ssh/ssh_063945.485417975_n3_upload-start-script.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/initial-scan/run_1

test cdc/tpcc-1000/sink=kafka failed: VMs preempted during the test run: teamcity-15519805-1717566618-21-n4cpu16-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_070901.591639602_n4_workload-run-tpcc-wa.log: COMMAND_PROBLEM: exit status 1
(cdc.go:388).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/tpcc-1000/sink=kafka/run_1

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15519805-1717566618-32-n7cpu16-0006 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null/run_1

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15519805-1717566618-28-n6cpu16-0002 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15519805-1717566618-33-n7cpu16-0003 (us-east1-d), teamcity-15519805-1717566618-33-n7cpu16-0004 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/server=processor/protocol=mux/format=json/sink=null/run_1

test decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region failed: VMs preempted during the test run: teamcity-15519805-1717566618-49-n8cpu16-geo-0005 (us-central1-b)

**Other Failures:**
(decommissionbench.go:721).runDecommissionBench: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/decommissionBench/nodes=6/warehouses=1000/drain-first/while-upreplicating/target=2/multi-region/run_1

test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-15519805-1717566618-66-n8cpu4-0006 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1

test tpccbench/nodes=6/cpu=16/multi-az failed: VMs preempted during the test run: teamcity-15519805-1717566618-102-n7cpu16-geo-0006 (us-central1-c), teamcity-15519805-1717566618-102-n7cpu16-geo-0005 (us-central1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15519805-1717566618-103-n12cpu4hm-geo-0006 (us-west1-b), teamcity-15519805-1717566618-103-n12cpu4hm-geo-0007 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

test tpccbench/nodes=3/cpu=4 failed: VMs preempted during the test run: teamcity-15519805-1717566618-99-n4cpu4-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/run_1/ssh/ssh_131602.798717054_n2_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/run_1

test failover/non-system/crash failed: VMs preempted during the test run: teamcity-15536048-1717652953-49-n7cpu2-0001 (us-east1-d)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/crash/run_1

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15536048-1717652953-103-n12cpu4hm-geo-0005 (us-west1-b), teamcity-15536048-1717652953-103-n12cpu4hm-geo-0007 (us-west1-b), teamcity-15536048-1717652953-103-n12cpu4hm-geo-0006 (us-west1-b), teamcity-15536048-1717652953-103-n12cpu4hm-geo-0008 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

test schemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-15536048-1717652953-96-n5cpu16-0001 (us-east1-d), teamcity-15536048-1717652953-96-n5cpu16-0003 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/index/tpcc/w=800/run_1

test failover/non-system/deadlock failed: VMs preempted during the test run: teamcity-15551410-1717739344-57-n7cpu2-0001 (us-east1-c), teamcity-15551410-1717739344-57-n7cpu2-0003 (us-east1-c)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/failover/non-system/deadlock/run_1

test ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15551410-1717739344-90-n4cpu32-0002 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global/run_1

test tpccbench/nodes=6/cpu=16/multi-az failed: VMs preempted during the test run: teamcity-15551410-1717739344-112-n7cpu16-geo-0006 (us-central1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=6/cpu=16/multi-az/run_1

test tpccbench/nodes=9/cpu=4/chaos/partition failed: VMs preempted during the test run: teamcity-15574261-1717912233-108-n10cpu4-0002 (us-east1-d), teamcity-15574261-1717912233-108-n10cpu4-0009 (us-east1-d)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1/ssh/ssh_130026.353782613_n9_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1

test c2c/MultiRegion/SameRegions/kv0 failed: VMs preempted during the test run: teamcity-15583933-1717998540-20-n9cpu8-geo-0007 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/MultiRegion/SameRegions/kv0/run_1

test allocbench/nodes=7/cpu=8/kv/r=0/access=skew failed: VMs preempted during the test run: teamcity-15583933-1717998540-10-n8cpu8-0006 (us-east1-d)

**Other Failures:**
(cluster.go:2390).Wipe: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1/ssh/ssh_072927.429918524_n6_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=0/access=skew/run_1

test kv95limited-spans/enc=false/nodes=1/seq/no-load-splitting failed: VMs preempted during the test run: teamcity-15583933-1717998540-09-n2cpu8-0002 (us-east1-c), teamcity-15583933-1717998540-09-n2cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95limited-spans/enc=false/nodes=1/seq/no-load-splitting/run_1

test failover/chaos/read-only failed: VMs preempted during the test run: teamcity-15583933-1717998540-52-n10cpu2-0008 (us-east1-d)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-only/run_1

test failover/chaos/read-write failed: VMs preempted during the test run: teamcity-15583933-1717998540-53-n10cpu2-0002 (us-east1-d), teamcity-15583933-1717998540-53-n10cpu2-0001 (us-east1-d), teamcity-15583933-1717998540-53-n10cpu2-0008 (us-east1-d), teamcity-15583933-1717998540-53-n10cpu2-0010 (us-east1-d), teamcity-15583933-1717998540-53-n10cpu2-0006 (us-east1-d)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/chaos/read-write/run_1

test tpccbench/nodes=9/cpu=4/multi-region failed: VMs preempted during the test run: teamcity-15583933-1717998540-100-n12cpu4hm-geo-0006 (us-west1-b), teamcity-15583933-1717998540-100-n12cpu4hm-geo-0008 (us-west1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/multi-region/run_1

test ycsb/F/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15583933-1717998540-81-n4cpu32-0003 (us-east1-d), teamcity-15583933-1717998540-81-n4cpu32-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/F/nodes=3/cpu=32/mvcc-range-keys=global/run_1

test tpccbench/nodes=3/cpu=16/mt-shared-process failed: VMs preempted during the test run: teamcity-15601173-1718085039-30-n4cpu16-0003 (us-east1-b)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1/ssh/ssh_090552.318186743_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/mt-shared-process/run_1

test kv95/enc=false/nodes=1/cpu=32/mt-shared-process failed: VMs preempted during the test run: teamcity-15601173-1718085039-78-n2cpu32-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=1/cpu=32/mt-shared-process/run_1

test kv0/enc=true/nodes=1 failed: VMs preempted during the test run: teamcity-15601173-1718085039-76-n2cpu8-0002 (us-east1-c), teamcity-15601173-1718085039-76-n2cpu8-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=true/nodes=1/run_1

test kv95/enc=false/nodes=3/cpu=32/size=4kb failed: VMs preempted during the test run: teamcity-15601173-1718085039-81-n4cpu32-0003 (us-east1-c), teamcity-15601173-1718085039-81-n4cpu32-0004 (us-east1-c), teamcity-15601173-1718085039-81-n4cpu32-0002 (us-east1-c), teamcity-15601173-1718085039-81-n4cpu32-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/size=4kb/run_1

test kv95/enc=false/nodes=3/size=4kb failed: VMs preempted during the test run: teamcity-15601173-1718085039-82-n4cpu8-0002 (us-east1-c), teamcity-15601173-1718085039-82-n4cpu8-0001 (us-east1-c), teamcity-15601173-1718085039-82-n4cpu8-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/size=4kb/run_1

test kv95/enc=false/nodes=3/cpu=32 failed: VMs preempted during the test run: teamcity-15601173-1718085039-80-n4cpu32-0004 (us-east1-c), teamcity-15601173-1718085039-80-n4cpu32-0002 (us-east1-c), teamcity-15601173-1718085039-80-n4cpu32-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/run_1

test schemachange/index/tpcc/w=800 failed: VMs preempted during the test run: teamcity-15601173-1718085039-98-n5cpu16-0001 (us-east1-c), teamcity-15601173-1718085039-98-n5cpu16-0002 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/schemachange/index/tpcc/w=800/run_1

test failover/non-system/blackhole/lease=expiration failed: VMs preempted during the test run: teamcity-15618473-1718171417-53-n7cpu2-0006 (us-east1-d)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole/lease=expiration/run_1

test failover/non-system/blackhole-recv/lease=expiration failed: VMs preempted during the test run: teamcity-15637260-1718257788-48-n7cpu2-0006 (us-east1-c)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole-recv/lease=expiration/run_1

test failover/non-system/blackhole/lease=expiration failed: VMs preempted during the test run: teamcity-15637260-1718257788-49-n7cpu2-0004 (us-east1-c), teamcity-15637260-1718257788-49-n7cpu2-0003 (us-east1-c), teamcity-15637260-1718257788-49-n7cpu2-0002 (us-east1-c), teamcity-15637260-1718257788-49-n7cpu2-0007 (us-east1-c)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/blackhole/lease=expiration/run_1

test kv95/enc=false/nodes=3/cpu=32/mt-shared-process failed: VMs preempted during the test run: teamcity-15637260-1718257788-77-n4cpu32-0002 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv95/enc=false/nodes=3/cpu=32/mt-shared-process/run_1

test tpccbench/nodes=9/cpu=4/chaos/partition failed: VMs preempted during the test run: teamcity-15637260-1718257788-95-n10cpu4-0009 (us-east1-d)

**Other Failures:**
(cluster.go:2284).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): expected 1 ports, got 0
test artifacts and logs in: /artifacts/tpccbench/nodes=9/cpu=4/chaos/partition/run_1

test schemachange/during/tpcc failed: VMs preempted during the test run: teamcity-15637260-1718257788-89-n5cpu16-0003 (us-east1-c), teamcity-15637260-1718257788-89-n5cpu16-0001 (us-east1-c), teamcity-15637260-1718257788-89-n5cpu16-0002 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_112541.772663529_n1_cockroach-workload-f.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/schemachange/during/tpcc/run_1/ssh/ssh_112541.793448857_n1_cockroach-workload-f.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/schemachange/during/tpcc/run_1

test scrub/all-checks/tpcc/w=100 failed: VMs preempted during the test run: teamcity-15637260-1718257788-91-n5cpu4-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2785).Conn: connecting to node 1: empty ip: []
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/scrub/all-checks/tpcc/w=100/run_1

test cdc/cloud-sink-gcs/assume-role failed: VMs preempted during the test run: teamcity-15654082-1718344196-22-n4cpu16-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_065238.858471788_n4_workload-run-tpcc-wa.log: COMMAND_PROBLEM: exit status 1
(cdc.go:388).Close: error shutting down prometheus/grafana: context canceled
test artifacts and logs in: /artifacts/cdc/cloud-sink-gcs/assume-role/run_1

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15654082-1718344196-32-n7cpu16-0001 (us-east1-c), teamcity-15654082-1718344196-32-n7cpu16-0006 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null/run_1

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control failed: VMs preempted during the test run: teamcity-15654082-1718344196-35-n7cpu16-0007 (us-east1-d), teamcity-15654082-1718344196-35-n7cpu16-0002 (us-east1-d), teamcity-15654082-1718344196-35-n7cpu16-0005 (us-east1-d), teamcity-15654082-1718344196-35-n7cpu16-0001 (us-east1-d), teamcity-15654082-1718344196-35-n7cpu16-0006 (us-east1-d), teamcity-15654082-1718344196-35-n7cpu16-0003 (us-east1-d), teamcity-15654082-1718344196-35-n7cpu16-0004 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control/run_1

test tpccbench/nodes=3/cpu=16/lease=expiration failed: VMs preempted during the test run: teamcity-15654082-1718344196-31-n4cpu16-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1/ssh/ssh_082104.176882049_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/lease=expiration/run_1

test failover/non-system/disk-stall/lease=expiration failed: VMs preempted during the test run: teamcity-15654082-1718344196-54-n7cpu2-0006 (us-east1-c), teamcity-15654082-1718344196-54-n7cpu2-0004 (us-east1-c)

**Other Failures:**
(failover.go:1775).sleepFor: sleep failed: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
(cluster.go:2398).Run: context canceled
test artifacts and logs in: /artifacts/failover/non-system/disk-stall/lease=expiration/run_1

test scrub/all-checks/tpcc/w=100 failed: VMs preempted during the test run: teamcity-15654082-1718344196-103-n5cpu4-0004 (us-east1-c), teamcity-15654082-1718344196-103-n5cpu4-0005 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/scrub/all-checks/tpcc/w=100/run_1

test tpcdsvec failed: VMs preempted during the test run: teamcity-15654082-1718344196-108-n3cpu4-0001 (us-east1-c)

**Other Failures:**
(tpcdsvec.go:109).1: pgx conn: failed to connect to ``host=34.139.15.89 user=roachprod database=``: server error (ERROR: server is not accepting clients, try another node (SQLSTATE 57P01))
test artifacts and logs in: /artifacts/tpcdsvec/run_1

test import/tpch/nodes=8 failed: VMs preempted during the test run: teamcity-15654082-1718344196-68-n8cpu4-0003 (us-east1-b)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/import/tpch/nodes=8/run_1

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15671072-1718430571-38-n7cpu16-0001 (us-east1-d), teamcity-15671072-1718430571-38-n7cpu16-0004 (us-east1-d), teamcity-15671072-1718430571-38-n7cpu16-0005 (us-east1-d), teamcity-15671072-1718430571-38-n7cpu16-0003 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null/run_1

test cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15671072-1718430571-37-n6cpu16-0005 (us-east1-d), teamcity-15671072-1718430571-37-n6cpu16-0006 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_080235.856829942_n6_cockroach-workload-i.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1/ssh/ssh_080235.882415608_n6_cockroach-workload-i.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/cdc/scan/catchup/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control failed: VMs preempted during the test run: teamcity-15671072-1718430571-41-n7cpu16-0003 (us-east1-c), teamcity-15671072-1718430571-41-n7cpu16-0005 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_082257.170631176_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/control/run_1

test replicate/up/1to3 failed: VMs preempted during the test run: teamcity-15671072-1718430571-74-n4cpu4-0001 (us-east1-c), teamcity-15671072-1718430571-74-n4cpu4-0003 (us-east1-c), teamcity-15671072-1718430571-74-n4cpu4-0002 (us-east1-c), teamcity-15671072-1718430571-74-n4cpu4-0004 (us-east1-c)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/replicate/up/1to3/run_1

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null failed: VMs preempted during the test run: teamcity-15681993-1718516960-28-n7cpu16-0005 (us-east1-d)

**Other Failures:**
(cluster.go:2284).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): expected 1 ports, got 0
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=processor/protocol=mux/format=json/sink=null/run_1

test multitenant/tpch/separate_process/enable_direct_scans failed: VMs preempted during the test run: teamcity-15681993-1718516960-34-n1cpu4-0001 (us-east1-c)

**Other Failures:**
(multitenant_tpch.go:70).func2: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/multitenant/tpch/separate_process/enable_direct_scans/run_1/ssh/ssh_080024.394823181_n1_cockroach-workload-r.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/multitenant/tpch/separate_process/enable_direct_scans/run_1

test tpccbench/nodes=3/cpu=16 failed: VMs preempted during the test run: teamcity-15681993-1718516960-27-n4cpu16-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2284).Start: one or more parallel execution failure(s): error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/run_1/ssh/ssh_090908.334759821_n1_init-certs.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/run_1

test tpccbench/nodes=3/cpu=16/enc=true failed: VMs preempted during the test run: teamcity-15681993-1718516960-26-n4cpu16-0003 (us-east1-c), teamcity-15681993-1718516960-26-n4cpu16-0001 (us-east1-c), teamcity-15681993-1718516960-26-n4cpu16-0004 (us-east1-c)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=16/enc=true/run_1/ssh/ssh_111010.381554369_n3_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=16/enc=true/run_1

test replicate/rebalance/3to5 failed: VMs preempted during the test run: teamcity-15681993-1718516960-86-n6cpu4-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2284).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): expected 1 ports, got 0
test artifacts and logs in: /artifacts/replicate/rebalance/3to5/run_1

test tpcc-nowait/nodes=3/w=1 failed: VMs preempted during the test run: teamcity-15693220-1718603328-24-n4cpu16-0003 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc-nowait/nodes=3/w=1/run_1

test allocbench/nodes=7/cpu=8/kv/r=95/access=skew failed: VMs preempted during the test run: teamcity-15693220-1718603328-11-n8cpu8-0005 (us-east1-d), teamcity-15693220-1718603328-11-n8cpu8-0007 (us-east1-d)

**Other Failures:**
(assertions.go:363).Fail: 
	Error Trace:	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:396
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            				github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            				main/pkg/cmd/roachtest/test_runner.go:1213
	            				src/runtime/asm_amd64.s:1695
	Error:      	Received unexpected error:
	            	monitor failure: EOF [owner=test-eng]
	            	(1) attached stack trace
	            	  -- stack trace:
	            	  | main.(*monitorImpl).WaitE
	            	  | 	main/pkg/cmd/roachtest/monitor.go:142
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBenchSample
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:395
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runAllocationBench
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:344
	            	  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAllocationBenchSpec.func1
	            	  | 	github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/allocation_bench.go:269
	            	  | main.(*testRunner).runTest.func2
	            	  | 	main/pkg/cmd/roachtest/test_runner.go:1213
	            	  | runtime.goexit
	            	  | 	src/runtime/asm_amd64.s:1695
	            	Wraps: (2) monitor failure
	            	Wraps: (3) secondary error attachment
	            	  | context canceled
	            	  | (1) context canceled
	            	  | Error types: (1) *errors.errorString
	            	Wraps: (4) EOF [owner=test-eng]
	            	Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) registry.ErrorWithOwnership
	Test:       	allocbench/nodes=7/cpu=8/kv/r=95/access=skew
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/allocbench/nodes=7/cpu=8/kv/r=95/access=skew/run_1

test tpcc-nowait/isolation-level=read-committed/nodes=3/w=1 failed: VMs preempted during the test run: teamcity-15693220-1718603328-26-n4cpu16-0002 (us-east1-d), teamcity-15693220-1718603328-26-n4cpu16-0003 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/tpcc-nowait/isolation-level=read-committed/nodes=3/w=1/run_1

test kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15693220-1718603328-83-n4cpu32-0003 (us-east1-d), teamcity-15693220-1718603328-83-n4cpu32-0002 (us-east1-d), teamcity-15693220-1718603328-83-n4cpu32-0004 (us-east1-d)

**Other Failures:**
(cluster.go:2398).Run: full command output in run_103749.490099349_n4_workload-run-kv-tole.log: error persisted after 3 attempts: _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global/run_1/ssh/ssh_103749.531020558_n4_workload-run-kv-tole.log): TRANSIENT_ERROR(ssh_problem): exit status 255
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global/run_1

test replicate/up/1to3 failed: VMs preempted during the test run: teamcity-15693220-1718603328-95-n4cpu4-0003 (us-east1-d)

**Other Failures:**
(cluster.go:2284).Start: failed to find services to register: failed to find 1 open ports: TRANSIENT_ERROR(open_ports): expected 1 ports, got 0
test artifacts and logs in: /artifacts/replicate/up/1to3/run_1

test tpccbench/nodes=3/cpu=4/enc=true failed: VMs preempted during the test run: teamcity-15693220-1718603328-105-n4cpu4-0001 (us-east1-d)

**Other Failures:**
(cluster.go:2335).Stop: cluster.StopE: one or more parallel execution failure(s): _potential_ SSH flake (``ssh -vvv`` log retained in /artifacts/tpccbench/nodes=3/cpu=4/enc=true/run_1/ssh/ssh_114831.038199600_n1_kill.log): TRANSIENT_ERROR(ssh_problem): exit status 255
test artifacts and logs in: /artifacts/tpccbench/nodes=3/cpu=4/enc=true/run_1

test ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15693220-1718603328-114-n4cpu32-0001 (us-east1-b)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global/run_1

test ycsb/D/nodes=3/cpu=32/mvcc-range-keys=global failed: VMs preempted during the test run: teamcity-15693220-1718603328-113-n4cpu32-0001 (us-east1-c)

**Other Failures:**
(cluster.go:2398).Run: context canceled
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/D/nodes=3/cpu=32/mvcc-range-keys=global/run_1

test scrub/index-only/tpcc/w=100 failed: VMs preempted during the test run: teamcity-15693220-1718603328-99-n5cpu4-0004 (us-east1-d)

**Other Failures:**
(monitor.go:154).Wait: monitor failure: EOF [owner=test-eng]
test artifacts and logs in: /artifacts/scrub/index-only/tpcc/w=100/run_1