ccl/schemachangerccl: TestBackupMixedVersionElements_base_drop_table failed about cockroach HOT 1 CLOSED

I240318 21:33:38.681112 1261888 server/init.go:288 ⋮ [T1,n?] 1976  joined cluster ‹f89b2bcd-c344-4cf7-bdb7-4d4aab974323› through join rpc
I240318 21:33:38.681135 1261888 server/init.go:289 ⋮ [T1,n?] 1977  received node ID: 3
I240318 21:33:38.681140 1261888 server/init.go:290 ⋮ [T1,n?] 1978  received cluster version: 22.2
W240318 21:33:38.681377 1261888 2@gossip/gossip.go:1406 ⋮ [T1,n3] 1979  no incoming or outgoing connections
I240318 21:33:38.681412 1261888 gossip/gossip.go:368 ⋮ [T1,n3] 1980  NodeDescriptor set to ‹node_id:3 address:<network_field:"tcp" address_field:"127.0.0.1:35511" > attrs:<> locality:<tiers:<key:"region" value:"us-east3" > > ServerVersion:<major_val:23 minor_val:1 patch:0 internal:0 > build_tag:"v23.1.17-dev" started_at:1710797618681407891 cluster_name:"" sql_address:<network_field:"tcp" address_field:"127.0.0.1:42537" > http_address:<network_field:"tcp" address_field:"127.0.0.1:34781" > ›
I240318 21:33:38.681759 1264596 1@server/server.go:1680 ⋮ [T1,n3] 1981  connecting to gossip network to verify cluster ID ‹"f89b2bcd-c344-4cf7-bdb7-4d4aab974323"›
I240318 21:33:38.681852 1261888 kv/kvserver/rebalance_objective.go:274 ⋮ [T1,n3,rebalance-objective] 1982  version doesn't support cpu objective, reverting to qps balance objective
I240318 21:33:38.682241 1261888 kv/kvserver/kvstorage/init.go:248 ⋮ [T1,n3,s3] 1983  beginning range descriptor iteration
I240318 21:33:38.682274 1261888 kv/kvserver/kvstorage/init.go:309 ⋮ [T1,n3,s3] 1984  range descriptor iteration done: 0 keys, 0 range descriptors (by suffix: map[]); scan stats: stepped 0 times (0 internal); seeked 2 times (2 internal); block-bytes: (total 0 B, cached 0 B); points: (count 0, key-bytes 0 B, value-bytes 0 B, tombstoned: 0) ranges: (count 0), (contained-points 0, skipped-points 0) evaluated requests: 0 gets, 1 scans, 0 reverse scans
I240318 21:33:38.682653 1261888 server/node.go:533 ⋮ [T1,n3] 1985  initialized store s3
I240318 21:33:38.682678 1261888 kv/kvserver/stores.go:264 ⋮ [T1,n3] 1986  read 0 node addresses from persistent storage
I240318 21:33:38.682716 1261888 server/node.go:627 ⋮ [T1,n3] 1987  started with engine type ‹2›
I240318 21:33:38.682723 1261888 server/node.go:629 ⋮ [T1,n3] 1988  started with attributes []
I240318 21:33:38.682822 1261888 1@server/server.go:1845 ⋮ [T1,n3] 1989  starting https server at ‹127.0.0.1:34781› (use: ‹127.0.0.1:34781›)
I240318 21:33:38.682835 1261888 1@server/server.go:1850 ⋮ [T1,n3] 1990  starting postgres server at ‹127.0.0.1:42537› (use: ‹127.0.0.1:42537›)
I240318 21:33:38.682842 1261888 1@server/server.go:1853 ⋮ [T1,n3] 1991  starting grpc server at ‹127.0.0.1:35511›
I240318 21:33:38.682850 1261888 1@server/server.go:1854 ⋮ [T1,n3] 1992  advertising CockroachDB node at ‹127.0.0.1:35511›
I240318 21:33:38.729195 1264693 gossip/client.go:124 ⋮ [T1,n3] 1993  started gossip client to n0 (‹127.0.0.1:33827›)
I240318 21:33:38.729574 1264596 1@server/server.go:1683 ⋮ [T1,n3] 1994  node connected via gossip
I240318 21:33:38.729677 1262486 kv/kvserver/stores.go:283 ⋮ [T1,n3] 1995  wrote 1 node addresses to persistent storage
I240318 21:33:38.729716 1262486 kv/kvserver/stores.go:283 ⋮ [T1,n3] 1996  wrote 2 node addresses to persistent storage
I240318 21:33:38.729973 1261984 kv/kvserver/stores.go:283 ⋮ [T1,n1] 1997  wrote 2 node addresses to persistent storage
I240318 21:33:38.730324 1262235 kv/kvserver/stores.go:283 ⋮ [T1,n2] 1998  wrote 2 node addresses to persistent storage
I240318 21:33:38.773112 1265007 kv/kvclient/rangefeed/rangefeedcache/watcher.go:335 ⋮ [T1,n3] 1999  spanconfig-subscriber: established range feed cache
I240318 21:33:38.773260 1261888 1@util/log/event_log.go:32 ⋮ [T1,n3] 2000 ={"Timestamp":1710797618773257990,"EventType":"node_join","NodeID":3,"StartedAt":1710797618681407891,"LastUp":1710797618681407891}
I240318 21:33:38.773355 1265010 kv/kvclient/rangefeed/rangefeedcache/watcher.go:335 ⋮ [T1,n3] 2001  settings-watcher: established range feed cache
I240318 21:33:38.775963 1261888 sql/sqlliveness/slinstance/slinstance.go:434 ⋮ [T1,n3] 2002  starting SQL liveness instance
I240318 21:33:38.777800 1265031 sql/sqlliveness/slstorage/slstorage.go:540 ⋮ [T1,n3] 2003  inserted sqlliveness session 0101804dbf88bb7b6a45e2937a2d24f999928f
I240318 21:33:38.777827 1265031 sql/sqlliveness/slinstance/slinstance.go:258 ⋮ [T1,n3] 2004  created new SQL liveness session 0101804dbf88bb7b6a45e2937a2d24f999928f
I240318 21:33:38.777848 1261888 sql/sqlinstance/instancestorage/instancestorage.go:342 ⋮ [T1,n3] 2005  assigning instance id to rpc addr ‹127.0.0.1:35511› and sql addr ‹127.0.0.1:42537›
I240318 21:33:38.779535 1261888 server/server_sql.go:1530 ⋮ [T1,n3] 2006  bound sqlinstance: Instance{RegionPrefix: gA==, InstanceID: 3, SQLAddr: ‹127.0.0.1:42537›, RPCAddr: ‹127.0.0.1:35511›, SessionID: 0101804dbf88bb7b6a45e2937a2d24f999928f, Locality: ‹region=us-east3›, BinaryVersion: 23.1}
I240318 21:33:38.780412 1265050 sql/temporary_schema.go:486 ⋮ [T1,n3] 2007  skipping temporary object cleanup run as it is not the leaseholder
I240318 21:33:38.780428 1265050 sql/temporary_schema.go:487 ⋮ [T1,n3] 2008  completed temporary object cleanup job
I240318 21:33:38.780435 1265050 sql/temporary_schema.go:639 ⋮ [T1,n3] 2009  temporary object cleaner next scheduled to run at 2024-03-18 22:03:38.780394551 +0000 UTC m=+2288.919247810
I240318 21:33:38.780491 1265051 sql/sqlstats/persistedsqlstats/provider.go:170 ⋮ [T1,n3] 2010  starting sql-stats-worker with initial delay: 10m51.655054128s
I240318 21:33:38.781384 1261888 upgrade/upgrademanager/manager.go:170 ⋮ [T1,n3] 2011  running permanent upgrades up to version: 22.2
I240318 21:33:38.781886 1265141 kv/kvclient/rangefeed/rangefeedcache/watcher.go:335 ⋮ [T1,n3] 2012  system-config-cache: established range feed cache
I240318 21:33:38.790567 1261888 upgrade/upgrademanager/manager.go:238 ⋮ [T1,n3] 2013  the last permanent upgrade (v22.1-42) does not appear to have completed; attempting to run all upgrades
I240318 21:33:38.790908 1261888 upgrade/upgrademanager/manager.go:283 ⋮ [T1,n3] 2014  running permanent upgrade for version 0.0-2
I240318 21:33:38.794042 1261888 upgrade/upgrademanager/manager.go:283 ⋮ [T1,n3] 2015  running permanent upgrade for version 0.0-4
I240318 21:33:38.795859 1261888 upgrade/upgrademanager/manager.go:283 ⋮ [T1,n3] 2016  running permanent upgrade for version 0.0-6
I240318 21:33:38.798488 1261888 upgrade/upgrademanager/manager.go:283 ⋮ [T1,n3] 2017  running permanent upgrade for version 0.0-8
I240318 21:33:38.800680 1261888 upgrade/upgrademanager/manager.go:283 ⋮ [T1,n3] 2018  running permanent upgrade for version 0.0-10
I240318 21:33:38.802624 1261888 upgrade/upgrademanager/manager.go:283 ⋮ [T1,n3] 2019  running permanent upgrade for version 0.0-12
I240318 21:33:38.804583 1261888 upgrade/upgrademanager/manager.go:283 ⋮ [T1,n3] 2020  running permanent upgrade for version 22.1-42
I240318 21:33:38.806208 1261888 server/server_sql.go:1638 ⋮ [T1,n3] 2021  done ensuring all necessary startup migrations have run
I240318 21:33:38.807003 1265385 server/auto_upgrade.go:43 ⋮ [T1,n3] 2022  auto upgrade disabled by testing
I240318 21:33:38.807039 1265381 jobs/job_scheduler.go:407 ⋮ [T1,n3] 2023  waiting 2m0s before scheduled jobs daemon start
I240318 21:33:38.808398 1265217 kv/kvclient/rangefeed/rangefeedcache/watcher.go:335 ⋮ [T1,n3] 2024  tenant-settings-watcher: established range feed cache
I240318 21:33:38.810821 1265417 kv/kvclient/rangefeed/rangefeedcache/watcher.go:335 ⋮ [T1,n3] 2025  tenant-capability-watcher: established range feed cache
I240318 21:33:38.811388 1265417 multitenant/tenantcapabilities/tenantcapabilitieswatcher/watcher.go:149 ⋮ [T1,n3] 2026  received results of a full table scan for tenant capabilities
I240318 21:33:38.817846 1265383 sql/syntheticprivilegecache/cache.go:204 ⋮ [T1,n3] 2027  warmed privileges for virtual tables in 10.867772ms
I240318 21:33:38.826907 1261888 1@server/server_sql.go:1756 ⋮ [T1,n3] 2028  serving sql connections
I240318 21:33:38.833927 1265525 upgrade/upgrademanager/manager.go:397 ⋮ [T1,n1,client=127.0.0.1:37560,hostssl,user=root,migration-mgr] 2029  migrating cluster from 22.2 to 22.2-48 (stepping through [22.2-2 22.2-4 22.2-6 22.2-8 22.2-10 22.2-12 22.2-14 22.2-16 22.2-18 22.2-20 22.2-22 22.2-24 22.2-26 22.2-28 22.2-30 22.2-32 22.2-34 22.2-36 22.2-38 22.2-40 22.2-42 22.2-44 22.2-46 22.2-48])
I240318 21:33:38.833992 1265525 upgrade/upgrademanager/manager.go:657 ⋮ [T1,n1,client=127.0.0.1:37560,hostssl,user=root,migration-mgr] 2030  executing operation validate-cluster-version=22.2-48
W240318 21:33:38.834147 1265525 upgrade/upgrademanager/manager.go:370 ⋮ [T1,n1,client=127.0.0.1:37560,hostssl,user=root,migration-mgr] 2031  error encountered during version upgrade: n3 required, but unavailable