Comments (3)
kvoli
commented on June 29, 2024
1
The write seems problematic, but still benign.
Possibly showing up now due to ba13697, which added the batch information cc @lyang24
Details
==================
WARNING: DATA RACE
Read at 0x00c00e2feb48 by goroutine 6146:
github.com/cockroachdb/cockroach/pkg/kv/kvpb.(*BatchRequest).SafeFormat()
github.com/cockroachdb/cockroach/pkg/kv/kvpb/pkg/kv/kvpb/batch.go:889 +0x6c4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*latch).SafeFormat()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:123 +0x1e5
github.com/cockroachdb/redact/internal/rfmt.(*pp).handleMethods()
github.com/cockroachdb/redact/internal/rfmt/external/com_github_cockroachdb_redact/internal/rfmt/print.go:630 +0x1099
github.com/cockroachdb/redact/internal/rfmt.(*pp).printArg()
github.com/cockroachdb/redact/internal/rfmt/external/com_github_cockroachdb_redact/internal/rfmt/print.go:810 +0x1577
github.com/cockroachdb/redact/internal/rfmt.(*pp).doPrintf()
github.com/cockroachdb/redact/internal/rfmt/external/com_github_cockroachdb_redact/internal/rfmt/print.go:1144 +0x379
github.com/cockroachdb/redact/internal/rfmt.Fprintf()
github.com/cockroachdb/redact/internal/rfmt/external/com_github_cockroachdb_redact/internal/rfmt/print.go:212 +0x87
github.com/cockroachdb/redact/builder.(*StringBuilder).Printf()
github.com/cockroachdb/redact/builder/external/com_github_cockroachdb_redact/builder/builder.go:100 +0x70
github.com/cockroachdb/cockroach/pkg/util/log.makeUnstructuredEntry()
github.com/cockroachdb/cockroach/pkg/util/log/log_entry.go:245 +0x1d2
github.com/cockroachdb/cockroach/pkg/util/log.Eventf()
github.com/cockroachdb/cockroach/pkg/util/log/trace.go:97 +0x10b
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).waitForSignal()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:584 +0x2d7
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).iterAndWait()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:563 +0x27e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).wait()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:530 +0x75b
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch.(*Manager).Acquire()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch/manager.go:251 +0x204
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*latchManagerImpl).Acquire()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/latch_manager.go:29 +0x71
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:296 +0x708
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:244 +0x331
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:461 +0x516
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185 +0x5cf
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0xd11
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203 +0xde
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal()
github.com/cockroachdb/cockroach/pkg/server/node.go:1441 +0x930
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch()
github.com/cockroachdb/cockroach/pkg/server/node.go:1646 +0x4a4
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x74
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12()
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x72b
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0xf0
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x2b6
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor-fm()
<autogenerated>:1 +0xd6
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x6a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337 +0x185
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x150
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x75
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8()
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x5ed
github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x199
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x462
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907 +0xf2
github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).Batch()
<autogenerated>:1 +0x29
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:214 +0x282
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:192 +0x139
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:76 +0x2ec
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
<autogenerated>:1 +0x6b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2769 +0x3688
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2111 +0x9c4
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1636 +0x1030
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1252 +0xa39
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:82 +0x26b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:46 +0x1d9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:239 +0x272
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:167 +0x1d8
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:194 +0x3fa
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:326 +0x244
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:117 +0x3d8
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:246 +0x7bb
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:537 +0xa76
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
github.com/cockroachdb/cockroach/pkg/kv/db.go:1138 +0x181
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send()
github.com/cockroachdb/cockroach/pkg/kv/txn.go:1282 +0x37c
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send-fm()
<autogenerated>:1 +0x55
github.com/cockroachdb/cockroach/pkg/kv.sendAndFill()
github.com/cockroachdb/cockroach/pkg/kv/db.go:970 +0x27d
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Run()
github.com/cockroachdb/cockroach/pkg/kv/txn.go:800 +0xa7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.splitTxnAttempt()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:283 +0xb2a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).adminSplitWithDescriptor.func2()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:555 +0x26e
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec()
github.com/cockroachdb/cockroach/pkg/kv/txn.go:1049 +0xad
github.com/cockroachdb/cockroach/pkg/kv.runTxn()
github.com/cockroachdb/cockroach/pkg/kv/db.go:1103 +0x6b
github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl()
github.com/cockroachdb/cockroach/pkg/kv/db.go:1066 +0xec
github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn()
github.com/cockroachdb/cockroach/pkg/kv/db.go:1041 +0x2866
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).adminSplitWithDescriptor()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_command.go:554 +0x1477
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*splitQueue).processAttempt()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/split_queue.go:292 +0x1e9
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*splitQueue).processAttemptWithTracing()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/split_queue.go:254 +0x259
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*splitQueue).process()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/split_queue.go:229 +0x8b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica.func1()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:984 +0x116
github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout()
github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29 +0x12d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processReplica()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:977 +0x65d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processOneAsyncAndReleaseSem.func1()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:927 +0x15a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x262
Previous write at 0x00c00e2feb48 by goroutine 5842:
github.com/cockroachdb/cockroach/pkg/kv/kvserver.tryBumpBatchTimestamp()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_batch_updates.go:242 +0x63b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.canDoServersideRetry()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_evaluate.go:587 +0x3c6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWithinUncertaintyIntervalError()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:812 +0x90
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:576 +0x8d9
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:181 +0x564
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0xd11
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203 +0xde
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal()
github.com/cockroachdb/cockroach/pkg/server/node.go:1441 +0x930
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch()
github.com/cockroachdb/cockroach/pkg/server/node.go:1646 +0x4a4
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x74
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12()
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x72b
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0xf0
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x2b6
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor-fm()
<autogenerated>:1 +0xd6
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x6a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337 +0x185
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x150
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x88
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x75
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8()
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x5ed
github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x199
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x462
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch()
github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907 +0xf2
github.com/cockroachdb/cockroach/pkg/rpc.(*internalClientAdapter).Batch()
<autogenerated>:1 +0x29
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:214 +0x282
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:192 +0x139
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.raceTransport.SendNext()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport_race.go:76 +0x2ec
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*raceTransport).SendNext()
<autogenerated>:1 +0x6b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2769 +0x3688
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2111 +0x9c4
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1636 +0x1030
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1252 +0xa39
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnLockGatekeeper).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_lock_gatekeeper.go:82 +0x26b
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnMetricRecorder).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_metric_recorder.go:46 +0x1d9
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).sendLockedWithRefreshAttempts()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:239 +0x272
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:167 +0x1d8
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnCommitter).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_committer.go:149 +0x95e
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnPipeliner).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_pipeliner.go:326 +0x244
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSeqNumAllocator).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_seq_num_allocator.go:117 +0x3d8
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnHeartbeater).SendLocked()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_heartbeater.go:246 +0x7bb
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).Send()
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:537 +0xa76
github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender()
github.com/cockroachdb/cockroach/pkg/kv/db.go:1138 +0x181
github.com/cockroachdb/cockroach/pkg/kv.(*Txn).Send()
github.com/cockroachdb/cockroach/pkg/kv/txn.go:1282 +0x37c
github.com/cockroachdb/cockroach/pkg/sql/row.newTxnKVFetcher.makeTxnKVFetcherDefaultSendFunc.func2()
github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:279 +0x55
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).fetch()
github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:583 +0x9e8
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch()
github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:866 +0x1f68
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).nextBatch-fm()
<autogenerated>:1 +0xb8
github.com/cockroachdb/cockroach/pkg/sql/row.(*kvBatchFetcherHelper).NextBatch()
github.com/cockroachdb/cockroach/pkg/sql/row/kv_batch_fetcher.go:1015 +0xcc
github.com/cockroachdb/cockroach/pkg/sql/row.(*txnKVFetcher).NextBatch()
<autogenerated>:1 +0xb9
github.com/cockroachdb/cockroach/pkg/sql/colfetcher.(*ColBatchDirectScan).Next()
github.com/cockroachdb/cockroach/pkg/sql/colfetcher/colbatch_direct_scan.go:79 +0x13c
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93 +0x6b
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils.(*CancelChecker).Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils/cancel_checker.go:59 +0x44
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93 +0x6b
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:247 +0x101
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).nextAdapter()
github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:272 +0x3b
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).nextAdapter-fm()
<autogenerated>:1 +0x1f
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError()
github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152 +0x7c
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:278 +0x84
github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*filtererProcessor).Next()
github.com/cockroachdb/cockroach/pkg/sql/rowexec/filterer.go:81 +0x82
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/columnarizer.go:239 +0x2aa
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93 +0x6b
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase.(*simpleProjectOp).Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase/simple_project.go:124 +0x5a
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*invariantsChecker).Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/invariants_checker.go:93 +0x6b
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:247 +0x101
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).nextAdapter()
github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:272 +0x3b
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).nextAdapter-fm()
<autogenerated>:1 +0x1f
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError()
github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152 +0x7c
github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Materializer).Next()
github.com/cockroachdb/cockroach/pkg/sql/colexec/materializer.go:278 +0x84
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).next()
github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:128 +0x6b
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).nextAdapter()
github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:145 +0x3b
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).nextAdapter-fm()
<autogenerated>:1 +0x1f
github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError()
github.com/cockroachdb/cockroach/pkg/sql/colexecerror/error.go:152 +0x7c
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Next()
github.com/cockroachdb/cockroach/pkg/sql/colflow/flow_coordinator.go:150 +0x5e
github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run()
github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:198 +0x49
github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run()
github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:723 +0xcc
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*FlowCoordinator).Run()
<autogenerated>:1 +0x5d
github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run()
github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:579 +0x77c
github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run()
github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:305 +0x3a5
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:928 +0x1815
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1990 +0x32e
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1704 +0x1b0
github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll()
github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1707 +0x353
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2440 +0xb88
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1987 +0x238c
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1136 +0x8553
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:145 +0x184
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:3442 +0x539
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:144 +0xbdd
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:250 +0x744
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2401 +0x1179
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2403 +0xe3e
github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run()
github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2218 +0x3ea
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).runWithEx.func1()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:228 +0x104
Goroutine 6146 (running) created at:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x69c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processOneAsyncAndReleaseSem()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:922 +0x513
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2()
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/queue.go:881 +0x298
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x262
Goroutine 5842 (running) created at:
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).runWithEx()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:227 +0x305
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:1174 +0x1411
github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryIteratorEx()
github.com/cockroachdb/cockroach/pkg/sql/internal.go:817 +0x274
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).processClaimedJobs()
github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:170 +0x1c1
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func7()
github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1011 +0x145
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession()
github.com/cockroachdb/cockroach/pkg/jobs/registry.go:906 +0x327
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func13()
github.com/cockroachdb/cockroach/pkg/jobs/registry.go:922 +0x4c
github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10()
github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1116 +0x54a
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x262
from cockroach.
stevendanna
commented on June 29, 2024
This is a data race that appears to be happening inside an admin split. From the stack it doesn't look like it is actually in the ingest code.
from cockroach.
lyang24
commented on June 29, 2024
would a shallow copy of batch fix this? let me send a patch
from cockroach.
Related Issues (20)
- roachtest: open_ports failed HOT 3
- roachtest: ssh_problem failed HOT 3
- roachtest: cluster_creation failed HOT 1
- roachtest: open_ports failed HOT 3
- pkg/sql/schemachanger/schemachanger_test: TestRollback_drop_column_unique_index failed HOT 3
- roachtest: open_ports failed
- server: TestHotRangesResponse failed HOT 1
- changefeedccl: add roachtests exercising execution_locality in multi-region clusters HOT 1
- roachtest: schemachange/random-load failed [ERROR: unsupported comparison operator: refcursor = refcursor] HOT 5
- sql/stmtdiagnostics: TestDiagnosticsRequest failed HOT 1
- roachtest: schemachange/mixed-versions failed [debug doctor validation: existing descriptors that still need to be dropped] HOT 3
- ccl/backupccl: TestDataDriven_plpgsql_user_defined_functions failed HOT 1
- roachtest: asyncpg failed
- roachtest: backup-restore/small-ranges failed [fingerprinting slow] HOT 2
- roachtest: failover/partial/lease-leader failed HOT 1
- kv/kvserver/closedts/sidetransport: TestRPCConnUnblocksOnStopper failed HOT 5
- roachtest: unoptimized-query-oracle/disable-rules=all/rand-tables failed HOT 1
- Cluster of 3 nodes crushes HOT 8
- roachtest: unoptimized-query-oracle/disable-rules=all/rand-tables failed HOT 3
- roachtest: failover/chaos/read-write failed HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.