Comments (4)
goroutine 11113 [sync.Mutex.Lock, 1 minutes]:
sync.runtime_SemacquireMutex(0x747b95e?, 0x6?, 0x7f0b029297f8?)
GOROOT/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc001c92150)
GOROOT/src/sync/mutex.go:171 +0x15d
sync.(*Mutex).Lock(0x10?)
GOROOT/src/sync/mutex.go:90 +0x32
github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.Init.NewDNSProvider.func1(0xc003466dc0)
github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/dns.go:59 +0x49
github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*dnsProvider).CreateRecords(0xc0035f6048, {0x9078530, 0xc001e90b40}, {0xc0014e61c0, 0x2, 0xc0014e61c0?})
github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/dns.go:116 +0x73e
github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).RegisterServices.func1({0x7f0b00df4c50, 0xda89220})
github.com/cockroachdb/cockroach/pkg/roachprod/install/services.go:324 +0x354
github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForDNSProvider({0xc0049f5fc0, 0x3}, 0xc002dd8da8)
github.com/cockroachdb/cockroach/pkg/roachprod/vm/dns.go:116 +0x122
github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).RegisterServices(0xc0031bc360, {0x9078530, 0xc001e90b40}, {0xc0027146c0, 0x2, 0x7714e01?})
github.com/cockroachdb/cockroach/pkg/roachprod/install/services.go:308 +0x348
github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).maybeRegisterServices(0xc0031bc360, {0x9078530, 0xc001e90b40}, 0xc004f19bc0, {0x0, {0xc0020847c0, 0x1, 0x1}, 0x1, {0x7714e01, ...}, ...}, ...)
github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:292 +0x2a5
github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start(0xc0031bc360, {0x9078530, 0xc001e90b40}, 0xc004f19bc0, {0x0, {0xc0020847c0, 0x1, 0x1}, 0x1, {0x7714e01, ...}, ...})
github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:408 +0x376
github.com/cockroachdb/cockroach/pkg/roachprod.Start({0x9078530, 0xc001e90b40}, 0xc004f19bc0, {0xc0054da360?, 0xc002180008?}, {0x0, {0xc0020847c0, 0x1, 0x1}, 0x1, ...}, ...)
github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:751 +0xba
main.(*clusterImpl).StartE(_, {_, _}, _, {{0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...}, ...)
main/pkg/cmd/roachtest/cluster.go:2076 +0x46e
main.(*clusterImpl).Start(_, {_, _}, _, {{0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...}, ...)
main/pkg/cmd/roachtest/cluster.go:2236 +0xbe
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runCheckLocalityIPAddress({0x9078530, 0xc001e90b40}, {0x911f5a0, 0xc0018b9760}, {0x916bf30, 0xc002844488})
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/gossip.go:516 +0x283
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAcceptance.func1({0x9078530?, 0xc001e90b40?}, {0x911f5a0?, 0xc0018b9760?}, {0x916bf30?, 0xc002844488?})
github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/acceptance.go:152 +0x3a
main.(*testRunner).runTest.func2()
main/pkg/cmd/roachtest/test_runner.go:1208 +0xf2
created by main.(*testRunner).runTest in goroutine 74
main/pkg/cmd/roachtest/test_runner.go:1192 +0x927
Node startup failed. Based on the stacks in __stacks.log
, this looks like some kind of infra failure. I'll move this to test-eng, who may know more.
from cockroach.
cc @cockroachdb/test-eng
from cockroach.
@herkolategan This is the mutex in NewDNSProviderWithExec
,
return NewDNSProviderWithExec(func(cmd *exec.Cmd) ([]byte, error) {
// Limit to one gcloud command at a time. At this time we are unsure if it's
// safe to make concurrent calls to the `gcloud` CLI to mutate DNS records
// in the same zone. We don't mutate the same record in parallel, but we do
// mutate different records in the same zone. See: #122180 for more details.
gcloudMu.Lock()
defer gcloudMu.Unlock()
return cmd.CombinedOutput()
})
which contends with WipeForReuse
, and causes the test to time out in the process. Did we hear back from GCE support on whether the global lock is required?
goroutine 68 [semacquire, 4 minutes]:
sync.runtime_Semacquire(0x0?)
GOROOT/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc002a1ad80?)
GOROOT/src/sync/waitgroup.go:116 +0x48
golang.org/x/sync/errgroup.(*Group).Wait(0xc001efe440)
golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:56 +0x25
github.com/cockroachdb/cockroach/pkg/roachprod/vm.FanOutDNS({0xc005848808, 0x4, 0x26?}, 0xc001cb88a0)
github.com/cockroachdb/cockroach/pkg/roachprod/vm/dns.go:99 +0x33c
github.com/cockroachdb/cockroach/pkg/roachprod.DestroyDNS({0x9078568, 0xc001ac5770}, 0x0?, {0xc0024c0090?, 0x4?})
github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:2291 +0xaa
main.(*clusterImpl).DestroyDNS(...)
main/pkg/cmd/roachtest/cluster.go:2955
main.(*clusterImpl).WipeForReuse(0xc0014e3688, {_, _}, _, {{0x0, 0x0}, 0x4, 0x4, 0x0, 0x0, ...})
main/pkg/cmd/roachtest/cluster.go:2943 +0x46e
main.(*testRunner).runWorker(0xc002714360, {0x9078568?, 0xc001771140?}, {0xc001c939ac, _}, _, _, _, _, {0x1, ...}, ...)
main/pkg/cmd/roachtest/test_runner.go:631 +0x5d8
main.(*testRunner).Run.func1({0x9078568, 0xc001771140})
main/pkg/cmd/roachtest/test_runner.go:366 +0x252
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 1
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:476 +0x3fe
from cockroach.
@srosenberg Thanks for the extra info, Renato and I were looking at this earlier. I'll create a support ticket; I planned to only create one if there were issues around the mutex, and unfortunately it doesn't seem to scale.
from cockroach.
Related Issues (20)
- roachtest: unoptimized-query-oracle: OOM after a query with `st_memcollect` HOT 1
- roachtest: schemachange/random-load failed HOT 2
- Sentry: table_meta.go:443: partial index predicate does not exist in table metadata (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql... HOT 1
- Sentry: table_meta.go:443: partial index predicate does not exist in table metadata (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql... HOT 1
- kvclient: address TODO in TestProxyTracing
- pkg/sql/logictest/tests/local-legacy-schema-changer/local-legacy-schema-changer_test: TestLogic_select_for_update failed HOT 1
- ccl/backupccl: TestDataDriven_views failed HOT 1
- workload: tpcc workload fails after after period with decimal errors HOT 1
- sql: TestDropIndex failed
- roachtest: tpcc/mixed-headroom/n5cpu16 failed
- kv,obs: persist lease history beyond 5 changes in the rangelog contained in debug.zip
- Sentry: expr.go:491: lookup for ComparisonExpr ×'s CmpOp failed (1) assertion failure Wraps: (2) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.MemoizeCo...
- sql/pgwire: TestPipelineMetric failed HOT 4
- roachprod/promhelperclient: TestUpdatePrometheusTargets failed
- pkg/kv/kvserver/concurrency/concurrency_test_/concurrency_test_shard_6/16: pkg failed HOT 2
- raft: relax fsync requirements HOT 2
- roachtest: sysbench/oltp_read_write/nodes=3/cpu=32/conc=256 failed HOT 5
- pkg/sql/logictest/tests/local-vec-off/local-vec-off_test: TestLogic_select_for_update failed HOT 1
- pkg/sql/logictest/tests/fakedist-disk/fakedist-disk_test: TestLogic_select_for_update failed HOT 1
- roachtest: failover/chaos/read-only failed HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cockroach.