Comments (6)
cockroach-teamcity
commented on August 20, 2024
github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ f03e7efeebfd:
// 05:42:27.497 INFO: [JOB 2577] sstable deleted 010609
// 05:42:27.497 INFO: [JOB 2577] sstable deleted 010610
// 05:42:27.498 INFO: [JOB 2577] sstable deleted 010611
// 05:42:27.498 INFO: [JOB 2577] sstable deleted 010612
// 05:42:27.498 INFO: [JOB 2577] sstable deleted 010613
// 05:42:27.498 INFO: [JOB 2577] sstable deleted 010614
// 05:42:27.498 INFO: [JOB 2577] sstable deleted 010615
// 05:42:27.499 INFO: [JOB 2577] sstable deleted 010616
// 05:42:27.499 INFO: [JOB 2577] sstable deleted 010617
// 05:42:27.499 INFO: [JOB 2577] sstable deleted 010618
// 05:42:27.499 INFO: [JOB 2577] sstable deleted 010619
// 05:42:27.499 INFO: [JOB 2577] sstable deleted 010620
// 05:42:27.499 INFO: [JOB 2577] sstable deleted 010621
// 05:42:27.500 INFO: [JOB 2577] sstable deleted 010622
// 05:42:27.500 INFO: [JOB 2577] sstable deleted 010623
// 05:42:27.501 INFO: [JOB 2853] compacting(default) L4 [011780] (1.0KB) Score=3.32 + L5 [006783] (1.2KB) Score=0.55; OverlappingRatio: Single 1.17, Multi 0.00
// 05:42:27.501 INFO: [JOB 2577] sstable deleted 010624
// 05:42:27.502 INFO: [JOB 2577] sstable deleted 010625
// 05:42:27.511 INFO: [JOB 2852] compacted(default) L3 [011651] (1.0KB) Score=4.66 + L4 [] (0B) Score=3.31 -> L4 [011811] (1.0KB), in 0.0s (0.0s total), output rate 128KB/s
// 05:42:28.702 INFO: [JOB 2855] compacting(default) L3 [011652] (1.0KB) Score=4.62 + L4 [] (0B) Score=3.27; OverlappingRatio: Single 0.00, Multi 0.00
// 05:42:28.703 INFO: [JOB 2853] compacting: sstable created 011812
// 05:42:28.703 INFO: [JOB 2577] sstable deleted 010626
// 05:42:28.703 INFO: [JOB 2577] sstable deleted 010627
// 05:42:28.703 INFO: [JOB 2855] compacting: sstable created 011813
// 05:42:28.704 INFO: [JOB 2577] sstable deleted 010628
// 05:42:28.704 INFO: [JOB 2577] sstable deleted 010629
// 05:42:28.704 INFO: [JOB 2577] sstable deleted 010630
// 05:42:28.704 INFO: [JOB 2577] sstable deleted 010633
// 05:42:28.713 INFO: [JOB 2853] compacted(default) L4 [011780] (1.0KB) Score=3.32 + L5 [006783] (1.2KB) Score=0.55 -> L5 [011812] (1.0KB), in 1.2s (1.2s total), output rate 859B/s
// 05:42:29.944 INFO: [JOB 2577] sstable deleted 010644
// 05:42:29.954 INFO: [JOB 2855] compacted(default) L3 [011652] (1.0KB) Score=4.62 + L4 [] (0B) Score=3.27 -> L4 [011813] (1.0KB), in 0.0s (1.3s total), output rate 536KB/s
// 05:42:31.340 INFO: [JOB 2577] sstable deleted 010739
// 05:42:31.342 INFO: [JOB 2858] compacting(default) L4 [011811] (1.0KB) Score=3.33 + L5 [007273] (1.2KB) Score=0.55; OverlappingRatio: Single 1.13, Multi 0.00
// 05:42:31.343 INFO: [JOB 2577] sstable deleted 010805
// 05:42:31.343 INFO: [JOB 2577] sstable deleted 010806
// 05:42:31.343 INFO: [JOB 2577] sstable deleted 010807
// 05:42:31.344 INFO: [JOB 2577] sstable deleted 010808
// 05:42:31.344 INFO: [JOB 2577] sstable deleted 010809
// 05:42:31.344 INFO: [JOB 2577] sstable deleted 010810
// 05:42:31.344 INFO: [JOB 2577] sstable deleted 010811
// 05:42:31.345 INFO: [JOB 2577] sstable deleted 010812
// 05:42:31.345 INFO: [JOB 2858] compacting: sstable created 011814
// 05:42:31.345 INFO: [JOB 2577] sstable deleted 010813
// 05:42:31.346 INFO: [JOB 2859] compacting(default) L3 [011653] (1012B) Score=4.60 + L4 [] (0B) Score=3.27; OverlappingRatio: Single 0.00, Multi 0.00
// 05:42:31.347 INFO: [JOB 2577] sstable deleted 010814
// 05:42:31.347 INFO: [JOB 2577] sstable deleted 010815
// 05:42:31.358 INFO: [JOB 2858] compacted(default) L4 [011811] (1.0KB) Score=3.33 + L5 [007273] (1.2KB) Score=0.55 -> L5 [011814] (1.0KB), in 0.0s (0.0s total), output rate 255KB/s
To reduce: go test ./internal/metamorphic -tags invariants -run 'TestMeta$' --run-dir /artifacts/meta/240507-052709.6402565222102/random-006 --try-to-reduce -v
--- FAIL: TestMeta/execution/random-006 (908.26s)
Help
To reproduce, try:
go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMeta$' ./internal/metamorphic -seed 1715059629640823123 -ops "uniform:5000-10000"This test on roachdash | Improve this report!
from pebble.
itsbilal
commented on August 20, 2024
[04:09:59] : [Step 1/1] === NAME TestMeta/execution/random-028
[04:09:59] : [Step 1/1] meta.go:241:
[04:09:59] : [Step 1/1] ===== SEED =====
[04:09:59] : [Step 1/1] 1715054800530688428
[04:09:59] : [Step 1/1] ===== ERR =====
[04:09:59] : [Step 1/1] exit status 2
[04:09:59] : [Step 1/1] ===== OUT =====
[04:09:59] : [Step 1/1] panic: operation took longer than 1m0s: db1.IngestExternalFiles(external0, "ehjujdrmzqgccc" /* start */, "ehjujdrmzqgcccq" /* end */, "@184" /* syntheticSuffix */, "ehj" /* syntheticPrefix */, external0, "ugnujdrmzqgccc" /* start */, "ugnujdrmzqgcccx" /* end */, "" /* syntheticSuffix */, "ugn" /* syntheticPrefix */)
[04:09:59] : [Step 1/1]
[04:09:59] : [Step 1/1] goroutine 16904 [running]:
[04:09:59] : [Step 1/1] github.com/cockroachdb/pebble/metamorphic.(*Test).runOp.func1()
[04:09:59] : [Step 1/1] github.com/cockroachdb/pebble/metamorphic/external/com_github_cockroachdb_pebble/metamorphic/test.go:412 +0xb4
[04:09:59] : [Step 1/1] created by time.goFunc
[04:09:59] : [Step 1/1] GOROOT/src/time/sleep.go:177 +0x2d
[04:09:59] : [Step 1/1]
[04:09:59] : [Step 1/1] ===== OPTIONS =====
[04:09:59] : [Step 1/1] [Version]
[04:09:59] : [Step 1/1] pebble_version=0.1
[04:09:59] : [Step 1/1]
[04:09:59] : [Step 1/1] [Options]
[04:09:59] : [Step 1/1] bytes_per_sync=2097152
[04:09:59] : [Step 1/1] cache_size=32768
[04:09:59] : [Step 1/1] cleaner=archive
[04:09:59] : [Step 1/1] compaction_debt_concurrency=1073741824
[04:09:59] : [Step 1/1] comparer=pebble.internal.testkeys
[04:09:59] : [Step 1/1] disable_wal=true
[04:09:59] : [Step 1/1] disable_ingest_as_flushable=true
[04:09:59] : [Step 1/1] flush_delay_delete_range=70ms
[04:09:59] : [Step 1/1] flush_delay_range_key=755ms
[04:09:59] : [Step 1/1] flush_split_bytes=16
[04:09:59] : [Step 1/1] format_major_version=17
[04:09:59] : [Step 1/1] l0_compaction_concurrency=1
[04:09:59] : [Step 1/1] l0_compaction_file_threshold=4
[04:09:59] : [Step 1/1] l0_compaction_threshold=12
[04:09:59] : [Step 1/1] l0_stop_writes_threshold=40
[04:09:59] : [Step 1/1] lbase_max_bytes=1
[04:09:59] : [Step 1/1] max_concurrent_compactions=2
[04:09:59] : [Step 1/1] max_concurrent_downloads=1
[04:09:59] : [Step 1/1] max_manifest_file_size=16
[04:09:59] : [Step 1/1] max_open_files=1000
[04:09:59] : [Step 1/1] mem_table_size=16777216
[04:09:59] : [Step 1/1] mem_table_stop_writes_threshold=6
[04:09:59] : [Step 1/1] min_deletion_rate=16777216
[04:09:59] : [Step 1/1] merger=pebble.concatenate
[04:09:59] : [Step 1/1] multilevel_compaction_heuristic=none
[04:09:59] : [Step 1/1] read_compaction_rate=16000
[04:09:59] : [Step 1/1] read_sampling_multiplier=16
[04:09:59] : [Step 1/1] strict_wal_tail=true
[04:09:59] : [Step 1/1] table_cache_shards=8
[04:09:59] : [Step 1/1] validate_on_ingest=true
[04:09:59] : [Step 1/1] wal_dir=
[04:09:59] : [Step 1/1] wal_bytes_per_sync=0
[04:09:59] : [Step 1/1] max_writer_concurrency=0
[04:09:59] : [Step 1/1] force_writer_parallelism=false
[04:09:59] : [Step 1/1] secondary_cache_size_bytes=33554432
[04:09:59] : [Step 1/1] create_on_shared=2
[04:09:59] : [Step 1/1] disable_delete_only_compactions=true
[04:09:59] : [Step 1/1] disable_elision_only_compactions=true
[04:09:59] : [Step 1/1] disable_lazy_combined_iteration=true
[04:09:59] : [Step 1/1]
[04:09:59] : [Step 1/1] [Level "0"]
[04:09:59] : [Step 1/1] block_restart_interval=32
[04:09:59] : [Step 1/1] block_size=524288
[04:09:59] : [Step 1/1] block_size_threshold=69
[04:09:59] : [Step 1/1] compression=Snappy
[04:09:59] : [Step 1/1] filter_policy=rocksdb.BuiltinBloomFilter
[04:09:59] : [Step 1/1] filter_type=table
[04:09:59] : [Step 1/1] index_block_size=4
[04:09:59] : [Step 1/1] target_file_size=1
[04:09:59] : [Step 1/1]
[04:09:59] : [Step 1/1] [TestOptions]
[04:09:59] : [Step 1/1] ingest_using_apply=true
[04:09:59] : [Step 1/1] delete_sized=true
[04:09:59] : [Step 1/1] replace_single_delete=true
[04:09:59] : [Step 1/1] threads=8
[04:09:59] : [Step 1/1] disable_block_property_collector=true
[04:09:59] : [Step 1/1] enable_value_blocks=true
[04:09:59] : [Step 1/1] disable_value_blocks_for_ingest_sstables=true
[04:09:59] : [Step 1/1] async_apply_to_db=true
[04:09:59] : [Step 1/1] shared_storage_enabled=true
[04:09:59] : [Step 1/1] external_storage_enabled=true
[04:09:59] : [Step 1/1] secondary_cache_enabled=true
[04:09:59] : [Step 1/1] seed_efos=2309337034655293500
[04:09:59] : [Step 1/1] ingest_split=true
[04:09:59] : [Step 1/1] io_latency_mean=877.704µs
[04:09:59] : [Step 1/1] io_latency_probability=0.0098303769
[04:09:59] : [Step 1/1] io_latency_seed=1758674309626369732
[04:09:59] : [Step 1/1] use_shared_replicate=true
[04:09:59] : [Step 1/1]
[04:09:59] : [Step 1/1] ===== OPS =====
[04:09:59] : [Step 1/1] Init(0 /* dbs */, 45 /* batches */, 63 /* iters */, 45 /* snapshots */, 1 /* externalObjs */)
[04:09:59] : [Step 1/1] db1.Merge("wfvuccmac@1", "pnryreo")
[04:09:59] : [Step 1/1] db1.Get("wfvuccmac@1")
[04:09:59] : [Step 1/1] db1.Delete("wfvuccmac@1")
[04:09:59] : [Step 1/1] db1.Set("nrghd@1", "cpi")
[04:09:59] : [Step 1/1] db1.DeleteRange("nrghd@1", "wfvuccmac@1")
[04:09:59] : [Step 1/1] db1.Delete("wfvuccmac@1")
[04:09:59] : [Step 1/1] db1.Merge("nrghd@2", "owceqcpv")
[04:09:59] : [Step 1/1] db1.Merge("nrghd@2", "ozomk")
[04:09:59] : [Step 1/1] db1.Get("nrghd@2")
[04:09:59] : [Step 1/1] db1.Delete("wfvuccmac@1")
[04:09:59] : [Step 1/1] db1.Get("nrghd@2")
[04:09:59] : [Step 1/1] db1.Delete("nrghd@1")
[04:09:59] : [Step 1/1] db1.RangeKeySet("nrghd", "wfvuccmac", "@1", "ibxdqzh")
[04:09:59] : [Step 1/1] db1.DeleteRange("nrghd@2", "nrghd@1")
[04:09:59] : [Step 1/1] batch0 = db1.NewIndexedBatch()
[04:09:59] : [Step 1/1] db1.Merge("nrghd@1", "vmqvqjboxehkwacdn")
[04:09:59] : [Step 1/1] db1.Ingest(batch0)
[04:09:59] : [Step 1/1] db1.SingleDelete("nrghd@1", false /* maybeReplaceDelete */)
[04:09:59] : [Step 1/1] db1.SingleDelete("wfvuccmac@1", false /* maybeReplaceDelete */)
[04:09:59] : [Step 1/1] db1.Get("wfvuccmac@1")
from pebble.
jbowens
commented on August 20, 2024
Saw another failure on #3593.
2024-05-07T17:53:41.7149735Z --- FAIL: TestMeta (402.11s)
2024-05-07T17:53:41.7152974Z --- FAIL: TestMeta/execution (0.00s)
2024-05-07T17:53:41.7154582Z --- FAIL: TestMeta/execution/random-027 (377.95s)
2024-05-07T17:53:41.7159241Z meta.go:241:
2024-05-07T17:53:41.7160165Z ===== SEED =====
2024-05-07T17:53:41.7161634Z 1715104012873781044
2024-05-07T17:53:41.7162286Z ===== ERR =====
2024-05-07T17:53:41.7163534Z exit status 2
2024-05-07T17:53:41.7164407Z ===== OUT =====
2024-05-07T17:53:41.7165941Z panic: operation took longer than 1m0s: db1.Compact("kwdmkna@13", "qjggfqpf@10", false /* parallelize */)
2024-05-07T17:53:41.7167497Z
2024-05-07T17:53:41.7168444Z goroutine 35068 [running]:
2024-05-07T17:53:41.7170024Z github.com/cockroachdb/pebble/metamorphic.(*Test).runOp.func1()
2024-05-07T17:53:41.7171812Z /home/runner/work/pebble/pebble/metamorphic/test.go:412 +0xb4
2024-05-07T17:53:41.7173460Z created by time.goFunc
2024-05-07T17:53:41.7174905Z /opt/hostedtoolcache/go/1.22.2/x64/src/time/sleep.go:177 +0x2d
2024-05-07T17:53:41.7175856Z
2024-05-07T17:53:41.7176737Z goroutine 1 [chan receive, 6 minutes]:
2024-05-07T17:53:41.7179254Z testing.(*T).Run(0xc00023a4e0, {0xd31eb5?, 0x0?}, 0xd7f708)
2024-05-07T17:53:41.7180975Z /opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:1750 +0x3ab
2024-05-07T17:53:41.7182332Z testing.runTests.func1(0xc00023a4e0)
2024-05-07T17:53:41.7184193Z /opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:2161 +0x37
2024-05-07T17:53:41.7185716Z testing.tRunner(0xc00023a4e0, 0xc000051c70)
2024-05-07T17:53:41.7187500Z /opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:1689 +0xfb
2024-05-07T17:53:41.7189301Z testing.runTests(0xc000218bd0, {0x1353500, 0x2, 0x2}, {0x1?, 0xc000215520?, 0x0?})
2024-05-07T17:53:41.7191239Z /opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:2159 +0x445
2024-05-07T17:53:41.7192468Z testing.(*M).Run(0xc000242460)
2024-05-07T17:53:41.7194161Z /opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:2027 +0x68b
2024-05-07T17:53:41.7195245Z main.main()
2024-05-07T17:53:41.7196194Z _testmain.go:49 +0x16c
2024-05-07T17:53:41.7196962Z
2024-05-07T17:53:41.7197867Z goroutine 18 [semacquire, 6 minutes]:
2024-05-07T17:53:41.7199102Z sync.runtime_Semacquire(0xc0000e1a40?)
2024-05-07T17:53:41.7200671Z /opt/hostedtoolcache/go/1.22.2/x64/src/runtime/sema.go:62 +0x25
2024-05-07T17:53:41.7201860Z sync.(*WaitGroup).Wait(0x0?)
2024-05-07T17:53:41.7203300Z /opt/hostedtoolcache/go/1.22.2/x64/src/sync/waitgroup.go:116 +0x48
2024-05-07T17:53:41.7204910Z golang.org/x/sync/errgroup.(*Group).Wait(0xc000380340)
2024-05-07T17:53:41.7207360Z /home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:53 +0x25
2024-05-07T17:53:41.7209268Z github.com/cockroachdb/pebble/metamorphic.Execute(0xc00024ac60)
2024-05-07T17:53:41.7210998Z /home/runner/work/pebble/pebble/metamorphic/meta.go:643 +0x116
2024-05-07T17:53:41.7213672Z github.com/cockroachdb/pebble/metamorphic.RunOnce({0xe52cf8, 0xc00023a680}, {0x7ffcbcf95c47, 0x2b}, 0x0, {0xc00026ef00, 0x33}, {0xc00022c3e0, 0x2, 0x2})
2024-05-07T17:53:41.7215949Z /home/runner/work/pebble/pebble/metamorphic/meta.go:562 +0xff8
2024-05-07T17:53:41.7217941Z github.com/cockroachdb/pebble/internal/metamorphic.runTestMeta(0xc00023a680, 0x0)
2024-05-07T17:53:41.7220119Z /home/runner/work/pebble/pebble/internal/metamorphic/meta_test.go:87 +0x179
2024-05-07T17:53:41.7222418Z github.com/cockroachdb/pebble/internal/metamorphic.TestMeta(0xc00023a680?)
2024-05-07T17:53:41.7224631Z /home/runner/work/pebble/pebble/internal/metamorphic/meta_test.go:54 +0x15
2024-05-07T17:53:41.7226153Z testing.tRunner(0xc00023a680, 0xd7f708)
2024-05-07T17:53:41.7227755Z /opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:1689 +0xfb
2024-05-07T17:53:41.7229176Z created by testing.(*T).Run in goroutine 1
2024-05-07T17:53:41.7230750Z /opt/hostedtoolcache/go/1.22.2/x64/src/testing/testing.go:1742 +0x390
2024-05-07T17:53:41.7231769Z
2024-05-07T17:53:41.7232597Z goroutine 8 [select]:
2024-05-07T17:53:41.7234269Z github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
2024-05-07T17:53:41.7236092Z /home/runner/work/pebble/pebble/vfs/disk_health.go:670 +0xf8
2024-05-07T17:53:41.7238172Z created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 18
2024-05-07T17:53:41.7240430Z /home/runner/work/pebble/pebble/vfs/disk_health.go:659 +0x6e
2024-05-07T17:53:41.7241385Z
2024-05-07T17:53:41.7242284Z goroutine 9 [select]:
2024-05-07T17:53:41.7243925Z github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
2024-05-07T17:53:41.7245728Z /home/runner/work/pebble/pebble/vfs/disk_health.go:253 +0xcb
2024-05-07T17:53:41.7247765Z created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 18
2024-05-07T17:53:41.7249729Z /home/runner/work/pebble/pebble/vfs/disk_health.go:248 +0x58
2024-05-07T17:53:41.7250713Z
2024-05-07T17:53:41.7251716Z goroutine 11 [select, 6 minutes]:
2024-05-07T17:53:41.7253259Z github.com/cockroachdb/pebble/wal.(*dirProber).probeLoop(0xc0003d00c8)
2024-05-07T17:53:41.7255099Z /home/runner/work/pebble/pebble/wal/failover_manager.go:99 +0xd2
2024-05-07T17:53:41.7256781Z github.com/cockroachdb/pebble/wal.(*stopper).runAsync.func1()
2024-05-07T17:53:41.7258487Z /home/runner/work/pebble/pebble/wal/failover_manager.go:813 +0x22
2024-05-07T17:53:41.7260374Z created by github.com/cockroachdb/pebble/wal.(*stopper).runAsync in goroutine 18
2024-05-07T17:53:41.7262265Z /home/runner/work/pebble/pebble/wal/failover_manager.go:812 +0x79
2024-05-07T17:53:41.7263644Z
2024-05-07T17:53:41.7264455Z goroutine 12 [select]:
2024-05-07T17:53:41.7266372Z github.com/cockroachdb/pebble/wal.(*failoverMonitor).monitorLoop(0xc0003d0000, 0xc000200360)
2024-05-07T17:53:41.7268312Z /home/runner/work/pebble/pebble/wal/failover_manager.go:334 +0x105
2024-05-07T17:53:41.7270018Z github.com/cockroachdb/pebble/wal.newFailoverMonitor.func1()
2024-05-07T17:53:41.7271847Z /home/runner/work/pebble/pebble/wal/failover_manager.go:258 +0x25
2024-05-07T17:53:41.7273458Z github.com/cockroachdb/pebble/wal.(*stopper).runAsync.func1()
2024-05-07T17:53:41.7275198Z /home/runner/work/pebble/pebble/wal/failover_manager.go:813 +0x22
2024-05-07T17:53:41.7277038Z created by github.com/cockroachdb/pebble/wal.(*stopper).runAsync in goroutine 18
2024-05-07T17:53:41.7278878Z /home/runner/work/pebble/pebble/wal/failover_manager.go:812 +0x79
2024-05-07T17:53:41.7279972Z
2024-05-07T17:53:41.7280797Z goroutine 13 [chan receive]:
2024-05-07T17:53:41.7282360Z github.com/cockroachdb/pebble.(*cleanupManager).mainLoop(0xc000242500)
2024-05-07T17:53:41.7284194Z /home/runner/work/pebble/pebble/obsolete_files.go:163 +0xe5
2024-05-07T17:53:41.7286343Z github.com/cockroachdb/pebble.openCleanupManager.func1.1({0xe559d0?, 0xc000358090?})
2024-05-07T17:53:41.7288242Z /home/runner/work/pebble/pebble/obsolete_files.go:98 +0x17
2024-05-07T17:53:41.7290065Z runtime/pprof.Do({0xe55910?, 0x14128a0?}, {{0xc00022c300?, 0x0?, 0x0?}}, 0xc00006cfc0)
2024-05-07T17:53:41.7292089Z /opt/hostedtoolcache/go/1.22.2/x64/src/runtime/pprof/runtime.go:51 +0x9d
2024-05-07T17:53:41.7293743Z github.com/cockroachdb/pebble.openCleanupManager.func1()
2024-05-07T17:53:41.7295361Z /home/runner/work/pebble/pebble/obsolete_files.go:97 +0x56
2024-05-07T17:53:41.7297166Z created by github.com/cockroachdb/pebble.openCleanupManager in goroutine 18
2024-05-07T17:53:41.7298885Z /home/runner/work/pebble/pebble/obsolete_files.go:96 +0x265
from pebble.
cockroach-teamcity
commented on August 20, 2024
github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ 98cadc9f9bcd:
// 04:08:45.696 INFO: [JOB 7927] sstable deleted 021223
// 04:08:45.696 INFO: [JOB 7927] sstable deleted 021224
// 04:08:45.696 INFO: [JOB 7927] sstable deleted 021225
// 04:08:45.696 INFO: [JOB 7927] sstable deleted 021226
// 04:08:45.700 INFO: [JOB 7927] sstable deleted 021227
// 04:08:45.700 INFO: [JOB 7927] sstable deleted 021228
// 04:08:45.700 INFO: [JOB 7927] sstable deleted 021229
// 04:08:45.700 INFO: [JOB 7956] sstable deleted 021450
// 04:08:45.700 INFO: [JOB 7961] sstable deleted 021451
// 04:08:45.700 INFO: [JOB 7962] sstable deleted 021452
// 04:08:45.700 INFO: [JOB 7967] sstable deleted 021453
// 04:08:45.700 INFO: [JOB 7968] sstable deleted 021454
// 04:08:45.700 INFO: [JOB 7968] sstable deleted 021455
// 04:08:45.700 INFO: [JOB 7975] sstable deleted 021456
// 04:08:45.700 INFO: [JOB 7978] sstable deleted 021457
// 04:08:45.700 INFO: [JOB 7981] sstable deleted 021458
// 04:08:45.700 INFO: [JOB 7984] sstable deleted 021459
// 04:08:45.700 INFO: [JOB 7985] sstable deleted 021460
// 04:08:45.700 INFO: [JOB 7985] sstable deleted 021461
// 04:08:45.700 INFO: [JOB 7987] sstable deleted 021462
// 04:08:45.700 INFO: [JOB 7995] sstable deleted 021463
// 04:08:45.700 INFO: [JOB 7996] sstable deleted 021464
// 04:08:45.700 INFO: [JOB 7998] sstable deleted 021465
// 04:08:45.700 INFO: [JOB 7998] sstable deleted 021466
// 04:08:45.700 INFO: [JOB 8004] sstable deleted 021467
// 04:08:45.700 INFO: [JOB 8007] sstable deleted 021468
// 04:08:45.700 INFO: [JOB 8007] sstable deleted 021469
// 04:08:45.700 INFO: [JOB 8012] sstable deleted 021470
// 04:08:45.700 INFO: [JOB 8015] sstable deleted 021471
// 04:08:45.700 INFO: [JOB 8017] sstable deleted 021472
// 04:08:45.700 INFO: [JOB 8017] sstable deleted 021473
// 04:08:45.700 INFO: [JOB 8023] sstable deleted 021474
// 04:08:45.700 INFO: [JOB 8027] sstable deleted 021475
// 04:08:45.706 INFO: [JOB 7933] MANIFEST created 021756
// 04:08:46.341 INFO: [JOB 7933] flushed 1 memtable (244B) to L0 [021502 021503 021504 021505 021506 021507 021508 021509 021510 021511 021512 021513 021514 021515 021516 021517 021518 021519 021520 021521 021522 021523 021524 021525 021526 021527 021528 021529 021530 021531 021532 021533 021534 021535 021536 021537 021538 021539 021540 021541 021542 021543 021544 021545 021546 021547 021548 021549 021550 021551 021552 021553 021554 021555 021556 021557 021558 021559 021560 021561 021562 021563 021564 021565 021566 021567 021568 021569 021570 021571 021572 021573 021574 021575 021576 021577 021578 021579 021580 021581 021582 021583 021584 021585 021586 021587 021588 021589 021590 021591 021592 021593 021594 021595 021596 021597 021598 021599 021600 021601 021602 021603 021604 021605 021606 021607 021608 021609 021610 021611 021612 021613 021614 021615 021616 021617 021618 021619 021620 021621 021622 021623 021624 021625 021626 021627 021628 021629 021630 021631 021632 021633 021634 021635 021636 021637 021638 021639 021640 021641 021642 021643 021644 021645 021646 021647 021648 021649 021650 021651 021652 021653 021654 021655 021656 021657 021658 021659 021660 021661 021662 021663 021664 021665 021666 021667 021668 021670 021672 021674 021675 021677 021678 021680 021681 021683 021685 021686 021688 021689 021690 021692 021693 021695 021696 021697 021699 021700 021702 021704 021705 021706 021707 021708 021711 021712 021713 021714 021715 021716 021717 021719 021721 021722 021723 021724 021726 021728 021729 021732 021733 021734 021736 021737 021739 021740 021742 021743 021744 021745 021747 021748 021750 021752 021753 021754] (191KB), in 42.1s (42.7s total), output rate 4.5KB/s
// 04:08:46.341 INFO: [JOB 7933] WAL deleted 021043
// 04:08:46.341 INFO: [JOB 7933] sstable deleted 021476
// 04:08:46.341 INFO: [JOB 7933] MANIFEST deleted 018632
// 04:08:46.342 INFO: [JOB 8028] compacted(default) L4 [021477] (1018B) Score=20.42 + L5 [] (0B) Score=0.93 -> L5 [021755] (1018B), in 1.2s (1.9s total), output rate 845B/s
// 04:08:46.964 INFO: [JOB 8028] sstable deleted 021477
// 04:08:46.966 INFO: [JOB 8031] compacted(default) L4 [021478] (1.0KB) Score=21.15 + L5 [] (0B) Score=0.95 -> L5 [021757] (1.0KB), in 0.0s (1.3s total), output rate 3.1MB/s
// 04:08:47.608 INFO: [JOB 8031] sstable deleted 021478
// 04:08:47.608 INFO: [JOB 8034] compacting(default) L3 [021490] (981B) Score=13.28 + L4 [] (0B) Score=20.12; OverlappingRatio: Single 0.00, Multi 0.00
// 04:08:47.608 INFO: [JOB 8034] compacting: sstable created 021758
// 04:08:47.609 INFO: [JOB 8035] compacting(default) L4 [021479] (1.0KB) Score=20.86 + L5 [] (0B) Score=0.92; OverlappingRatio: Single 0.00, Multi 0.00
// 04:08:47.609 INFO: [JOB 8034] compacting: sstable created 021759
// 04:08:47.609 INFO: [JOB 8035] compacting: sstable created 021760
To reduce: go test ./internal/metamorphic -tags invariants -run 'TestMeta$' --run-dir /artifacts/meta/240508-040626.0551582757471/random-029 --try-to-reduce -v
--- FAIL: TestMeta/execution/random-029 (142.03s)
Help
To reproduce, try:
go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMeta$' ./internal/metamorphic -seed 1715141186055399080 -ops "uniform:5000-10000"This test on roachdash | Improve this report!
from pebble.
itsbilal
commented on August 20, 2024
Investigating with seed 1715141186055399080 where we time out on an IngestExternalFileOp, it looks like we time out in makeRoomForWrite where we're trying to force a flush cuz we can't ingest external files as a flushable ingest:
goroutine 73 [sync.Cond.Wait]:
runtime.gopark(0xc18718ac7b00e410?, 0x14000600000?, 0x88?, 0x70?, 0x104cf4de0?)
/opt/homebrew/Cellar/go/1.21.1/libexec/src/runtime/proc.go:398 +0xc8 fp=0x1401d8a7040 sp=0x1401d8a7020 pc=0x104cc3578
runtime.goparkunlock(...)
/opt/homebrew/Cellar/go/1.21.1/libexec/src/runtime/proc.go:404
sync.runtime_notifyListWait(0x14000480af0, 0xb3)
/opt/homebrew/Cellar/go/1.21.1/libexec/src/runtime/sema.go:527 +0x154 fp=0x1401d8a7090 sp=0x1401d8a7040 pc=0x104cf3804
sync.(*Cond).Wait(0x14000480ae0)
/opt/homebrew/Cellar/go/1.21.1/libexec/src/sync/cond.go:70 +0xcc fp=0x1401d8a70d0 sp=0x1401d8a7090 pc=0x104d0012c
github.com/cockroachdb/pebble.(*DB).makeRoomForWrite(0x14000480900, 0x0)
/Users/bilal/go/src/github.com/cockroachdb/pebble/db.go:2471 +0x1a0 fp=0x1401d8a71c0 sp=0x1401d8a70d0 pc=0x1051fec30
github.com/cockroachdb/pebble.(*DB).ingest.func1(0x8b7)
/Users/bilal/go/src/github.com/cockroachdb/pebble/ingest.go:1601 +0x684 fp=0x1401d8a74e0 sp=0x1401d8a71c0 pc=0x105217964
github.com/cockroachdb/pebble.(*commitPipeline).AllocateSeqNum(0x140002a6000, 0x1, 0x14017f979b0, 0x1401d8a78f0)
/Users/bilal/go/src/github.com/cockroachdb/pebble/commit.go:415 +0x280 fp=0x1401d8a7550 sp=0x1401d8a74e0 pc=0x1051d9160
github.com/cockroachdb/pebble.(*DB).ingest(0x14000480900, {0x14017394360, 0x1, 0x1}, {0x0, 0x0, 0x0}, {{0x0, 0x0, 0x0}, ...}, ...)
/Users/bilal/go/src/github.com/cockroachdb/pebble/ingest.go:1685 +0x674 fp=0x1401d8a7a40 sp=0x1401d8a7550 pc=0x105216524
github.com/cockroachdb/pebble.(*DB).Ingest(0x0?, {0x14017394360?, 0x14000463ff0?, 0xe?})
/Users/bilal/go/src/github.com/cockroachdb/pebble/ingest.go:1106 +0xbc fp=0x1401d8a7af0 sp=0x1401d8a7a40 pc=0x105214f4c
github.com/cockroachdb/pebble/metamorphic.(*ingestExternalFilesOp).run(0x140003f85c0, 0x14000000900, {0x140001738f0?, 0x1402c9c7f20?, 0x0?})
/Users/bilal/go/src/github.com/cockroachdb/pebble/metamorphic/ops.go:1034 +0x6cc fp=0x1401d8a7e20 sp=0x1401d8a7af0 pc=0x1052ad66c
github.com/cockroachdb/pebble/metamorphic.(*Test).runOp(0x14000000900?, 0x140241bff00?, {0x140001738f0?, 0x0?, 0x0?})
/Users/bilal/go/src/github.com/cockroachdb/pebble/metamorphic/test.go:415 +0x118 fp=0x1401d8a7e90 sp=0x1401d8a7e20 pc=0x1052c6ea8
github.com/cockroachdb/pebble/metamorphic.Execute.func1()
/Users/bilal/go/src/github.com/cockroachdb/pebble/metamorphic/meta.go:627 +0x114 fp=0x1401d8a7f60 sp=0x1401d8a7e90 pc=0x1052a5df4
golang.org/x/sync/errgroup.(*Group).Go.func1()
from pebble.
cockroach-teamcity
commented on August 20, 2024
github.com/cockroachdb/pebble/internal/metamorphic.TestMeta failed with artifacts on refs/heads/master @ 4301b8f511d9:
=== RUN TestMeta/execution/random-014
=== PAUSE TestMeta/execution/random-014
=== CONT TestMeta/execution/random-014
=== RUN TestMeta/execution/random-006
=== PAUSE TestMeta/execution/random-006
=== CONT TestMeta/execution/random-006
=== RUN TestMeta/execution/random-021
=== PAUSE TestMeta/execution/random-021
=== CONT TestMeta/execution/random-021
=== RUN TestMeta/execution/random-022
=== PAUSE TestMeta/execution/random-022
=== CONT TestMeta/execution/random-022
=== RUN TestMeta/execution/random-024
=== PAUSE TestMeta/execution/random-024
=== CONT TestMeta/execution/random-024
=== RUN TestMeta/execution/random-029
=== PAUSE TestMeta/execution/random-029
=== CONT TestMeta/execution/random-029
=== RUN TestMeta/compare/standard-001
=== RUN TestMeta/compare/standard-004
=== RUN TestMeta/execution/random-007
=== PAUSE TestMeta/execution/random-007
=== CONT TestMeta/execution/random-007
=== RUN TestMeta/execution/random-025
=== PAUSE TestMeta/execution/random-025
=== CONT TestMeta/execution/random-025
=== RUN TestMeta/compare/standard-008
=== RUN TestMeta/execution/random-020
=== PAUSE TestMeta/execution/random-020
=== CONT TestMeta/execution/random-020
=== RUN TestMeta/execution/standard-007
=== PAUSE TestMeta/execution/standard-007
=== CONT TestMeta/execution/standard-007
=== RUN TestMeta/execution/standard-029
=== PAUSE TestMeta/execution/standard-029
=== CONT TestMeta/execution/standard-029
=== RUN TestMeta/compare/standard-003
=== RUN TestMeta/compare/standard-006
=== RUN TestMeta/execution/standard-001
=== PAUSE TestMeta/execution/standard-001
=== CONT TestMeta/execution/standard-001
=== RUN TestMeta/execution/standard-003
=== PAUSE TestMeta/execution/standard-003
=== CONT TestMeta/execution/standard-003
=== RUN TestMeta/execution/random-016
=== PAUSE TestMeta/execution/random-016
=== CONT TestMeta/execution/random-016
=== RUN TestMeta/execution/random-028
=== PAUSE TestMeta/execution/random-028
=== CONT TestMeta/execution/random-028
Help
To reproduce, try:
go test -tags 'invariants' -exec 'stress -p 1' -timeout 0 -test.v -run 'TestMeta$' ./internal/metamorphic -seed 1715227882649100225 -ops "uniform:5000-10000"This test on roachdash | Improve this report!
from pebble.
Related Issues (20)
- github.com/cockroachdb/pebble/internal/metamorphic: TestMetaTwoInstance failed
- don't use filter blocks for large flushable ingests HOT 1
- smart filter block loading
- improve performance of aggregations over a key range using range annotations
- db: strictly enforce prefix in internalIterator.SeekPrefixGE
- sstable: show filenum when sstable metadata is corrupted
- db: shared compaction concurrency limit across multiple Pebble instances HOT 2
- db: add ability to excise without ingest HOT 1
- github.com/cockroachdb/pebble/internal/metamorphic: TestMeta failed HOT 10
- add ValidateKey method to comparer
- ingest: verify ingestion sstables have legal boundary range keys
- internal/metamorphic/crossversion/crossversion_test: TestMetaCrossVersion failed HOT 1
- Backup HOT 1
- cmd/pebble: nightly benchmarks failing
- db: Open fails with 'open : no such file or directory' HOT 2
- db: assert that ingested sstables with synthetic suffixes have no duplicate keys
- Release fix for MIPS32 support HOT 5
- *: remove instances of go:linkname throughout
- tool: wal dump requires .log in current working directory
- wal: reader batch corruption
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pebble.