Comments (8)
syndtr
commented on May 29, 2024
CompactRange(leveldb.Range{}) will force compaction of entire sstable but
not memdb. If you want memdb compaction happen more often you can decrease
WriteBuffer size in the options. (Re-) opening the database will only force
memdb compaction but not sstable compaction, the sstable compaction will
happen on as needed basis.
On May 9, 2013 11:29 PM, "Sascha Matzke" [email protected] wrote:
When writing a lot of entries the database get's pretty big fast (I'm
doing a lot of appends).From what I observed a meaningful compaction is only done when (re-)
opening a database.Is there a way to trigger the compaction without closing/opening the
database?CompactRange(leveldb.Range{}) does not seem to have the same effect.
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/20
.
from goleveldb.
x6j8x
commented on May 29, 2024
I think I still don't understand how this is working... While my process is running, the number of files is only increasing (I run db.CompactRange(leveldb.Range{}) every 30 seconds and the WriteBuffer is set to 2 Mbytes).
Over the time I reach the following (triggering the maxOpenFiles ulimit):
2.0G 20130510-indexdb.ldb
Number of file in the database directory: 1040
When I now stop the process and restart it, those numbers go down to:
65M 20130510-indexdb.ldb
Number of file in the database directory: 38
What do I have to do/configure so that the number of files compacted during the runtime of the process and not just after a restart?
Here's the corresponding LevelDB log:
2013/05/0010 12:04:24.478059 JournalRecovery: started, min=1203
2013/05/0010 12:04:24.481911 JournalRecovery: recovering, num=1203
2013/05/0010 12:04:24.675362 Compaction: table created, source=mem level=0 num=1271 size=2026940 entries=6990 min="\x00\x14<\x92\x18:v:3312303" max="\xff\xfd\x18\x1a\x01:v:3316548"
2013/05/0010 12:04:54.461792 CompactRange: ordered, level=-1
2013/05/0010 12:04:54.461856 Compaction: compacting, level=0 tables=1, level=1 tables=0
2013/05/0010 12:04:54.531004 Compaction: table created, source=file level=1 num=1274 size=118435 entries=1760 min="\x00\x14<\x92\x18:v:3312303" max="72de7...e0158:v:3317752"
2013/05/0010 12:04:54.584905 Compaction: table created, source=file level=1 num=1275 size=61541 entries=1732 min="730d6...b2fea:v:3314568" max="f1908...fd8a4:v:3318016"
2013/05/0010 12:04:54.643517 Compaction: table created, source=file level=1 num=1276 size=44132 entries=2367 min="f1ff6...75614:v:3316527" max="\xff\xfd\x18\x1a\x01:v:3316548"
2013/05/0010 12:04:54.643559 Compaction: done
2013/05/0010 12:04:54.693517 Compaction: compacting, level=1 tables=3, level=2 tables=30
2013/05/0010 12:04:55.118781 Compaction: table created, source=file level=2 num=1277 size=2378345 entries=134568 min="\x00\x00\x01\x94\x18:v:2848297" max="\x13$\nV-:v:815058"
2013/05/0010 12:04:55.552147 Compaction: table created, source=file level=2 num=1278 size=2376126 entries=134494 min="\x13$$\xfd\x18:v:90642" max="&)AB\x00:v:413628"
2013/05/0010 12:04:55.902064 Compaction: table created, source=file level=2 num=1279 size=2297278 entries=89094 min="&)BB\x00:v:588215" max="08a1c...fe8a1:v:376972"
2013/05/0010 12:04:56.227204 Compaction: table created, source=file level=2 num=1280 size=2212137 entries=52574 min="08a1c...34db7:v:2678419" max="179d3...d84e8:v:2009924"
2013/05/0010 12:04:56.543971 Compaction: table created, source=file level=2 num=1281 size=2214812 entries=49999 min="179d3...3beb7:v:3121004" max="274b0...b2887:v:2829443"
2013/05/0010 12:04:56.835738 Compaction: table created, source=file level=2 num=1282 size=2219331 entries=49738 min="274b8...39651:v:2964652" max="37256...37117:v:2268017"
2013/05/0010 12:04:57.185757 Compaction: table created, source=file level=2 num=1283 size=2225528 entries=57297 min="37256...7490c:v:1574959" max="4541b...4b943:v:1507774"
2013/05/0010 12:04:57.535926 Compaction: table created, source=file level=2 num=1284 size=2205001 entries=48003 min="45421...924a9:v:714523" max="548b9...3a1e9:v:1785534"
2013/05/0010 12:04:57.877843 Compaction: table created, source=file level=2 num=1285 size=2215928 entries=49929 min="548b9...4fd6f:v:1265494" max="64335...3a4ec:v:2388203"
2013/05/0010 12:04:58.211110 Compaction: table created, source=file level=2 num=1286 size=2219279 entries=55873 min="6433b...9a37d:v:2844649" max="72a9b...f3123:v:2121812"
2013/05/0010 12:04:58.619344 Compaction: table created, source=file level=2 num=1287 size=2217955 entries=51619 min="72a9c...6375f:v:761795" max="82093...52396:v:2217728"
2013/05/0010 12:04:59.044418 Compaction: table created, source=file level=2 num=1288 size=2207878 entries=50665 min="82097...ca573:v:697090" max="91841...36cfb:v:2778347"
2013/05/0010 12:04:59.428237 Compaction: table created, source=file level=2 num=1289 size=2224420 entries=56423 min="91841...596eb:v:210060" max=";cx\n-:v:2918189"
2013/05/0010 12:05:00.012964 Compaction: table created, source=file level=2 num=1290 size=2381878 entries=135230 min=";c\xa7\t\x19:v:80932" max="P\xe2T\x16\x19:v:844234"
2013/05/0010 12:05:00.579362 Compaction: table created, source=file level=2 num=1291 size=2354029 entries=121004 min="P\xe2X%\x01:v:355362" max="a2aa7...dda27:v:911542"
2013/05/0010 12:05:00.962695 Compaction: table created, source=file level=2 num=1292 size=2213262 entries=50017 min="a2aaa...2ee1c:v:3057594" max="b25f3...e2c9d:v:1309575"
2013/05/0010 12:05:01.437547 Compaction: table created, source=file level=2 num=1293 size=2220028 entries=49548 min="b25f4...4568b:v:999262" max="c261a...62b2e:v:1770511"
2013/05/0010 12:05:02.203886 Compaction: table created, source=file level=2 num=1294 size=2226225 entries=55180 min="c261a...b552f:v:2525909" max="d1161...72b85:v:3090578"
2013/05/0010 12:05:02.795533 Compaction: table created, source=file level=2 num=1295 size=2211302 entries=48419 min="d1162...44645:v:2228557" max="e109f...82f3e:v:2053058"
2013/05/0010 12:05:03.370411 Compaction: table created, source=file level=2 num=1296 size=2216454 entries=49826 min="e109f...61645:v:1967638" max="f0ed8...7c2ea:v:251478"
2013/05/0010 12:05:03.961938 Compaction: table created, source=file level=2 num=1297 size=2213653 entries=52612 min="f0ed8...d8f14:v:3287124" max="f\xc8DB\x00:v:1111236"
2013/05/0010 12:05:04.378505 Compaction: table created, source=file level=2 num=1298 size=2368333 entries=129111 min="f\xc8DB\x1a:v:1491561" max="w\xce\xcf\x19\x00:v:18689"
2013/05/0010 12:05:04.744955 Compaction: table created, source=file level=2 num=1299 size=2380170 entries=135331 min="w\xce\xd0\x19\x00:v:289785" max="\x8b\x16@\x19\x00:v:3153055"
2013/05/0010 12:05:05.145129 Compaction: table created, source=file level=2 num=1300 size=2379193 entries=135236 min="\x8b\x16N?\x18:v:106125" max="\xa0\xaf!\x92\x18:v:1217003"
2013/05/0010 12:05:05.578307 Compaction: table created, source=file level=2 num=1301 size=2376665 entries=134601 min="\xa0\xaf\"\x92\x18:v:1300426" max="\xb3\xa4U)\x06:v:1828266"
2013/05/0010 12:05:06.036599 Compaction: table created, source=file level=2 num=1302 size=2375046 entries=134643 min="\xb3\xa4U)-:v:1828267" max="Ƙ\xe1\x98\x18:v:710174"
2013/05/0010 12:05:06.606999 Compaction: table created, source=file level=2 num=1303 size=2385581 entries=135282 min="Ƙ\xe1\x98-:v:710185" max="\xd9\x1c\xf70-:v:1062605"
2013/05/0010 12:05:06.995736 Compaction: table created, source=file level=2 num=1304 size=2377734 entries=135276 min="\xd9\x1d\x10\b\x19:v:2088155" max="\xee\"\xce\x02\x19:v:1968804"
2013/05/0010 12:05:07.345604 Compaction: table created, source=file level=2 num=1305 size=2058167 entries=116987 min="\xee#u\x12\x18:v:1316830" max="\xff\xff\xff\x19\x01:v:22923"
2013/05/0010 12:05:07.345678 Compaction: done
from goleveldb.
syndtr
commented on May 29, 2024
This might be a bug. I will look into it.
from goleveldb.
syndtr
commented on May 29, 2024
@x6j8x It is no need to do explicit compaction. Compaction should happen automatically. And please let me know if you still run into same issue.
from goleveldb.
x6j8x
commented on May 29, 2024
@syndtr Thanks for the fix. It's working perfect now.
68M gsb/20130510-indexdb.ldb
NrOfFiles: 60
74M gsb/20130510-indexdb.ldb
NrOfFiles: 50
from goleveldb.
marcopeereboom
commented on May 29, 2024
I updated to September 10th 2013 and I am seeing pretty much this exact behavior as described. Gigs of disk use, restart and it drops to Megs. When I add stuff it grows super fast again too. Did this issue get reintroduced?
from goleveldb.
syndtr
commented on May 29, 2024
@marcopeereboom Thanks for reporting. 291c883 should fix it.
from goleveldb.
marcopeereboom
commented on May 29, 2024
It did, thanks. Although I have seen a crash with the latest code as well but I'll write a new bug up for that.
from goleveldb.
Related Issues (20)
- Abandoned / Maintained? HOT 3
- Browser indexeddb in golang / wasm
- Does the 【Get】 method support * and ? fuzzy matching HOT 1
- long time no update release HOT 3
- Bad result in find by prefix
- Large Manifest file is repeatedly re-created HOT 2
- Locking the DB
- panic in `db.Stats()` during `db.Close()`
- data race
- some node will pass to f() function many times, is this a bug or a feature?
- slice bounds out of range [:18014398510996847] with capacity 4194304
- query efficiency and memory usage
- Opening a transaction will freeze all other db transactions HOT 2
- Is it possible to read the db from to different programs? HOT 3
- leveldb.(*DB).Close() took too much time to finish
- Panic: unexpected fault address
- Mem compression occurred during the creation of the iterator
- is the leveldb write (to disk) rate is 25MB ?
- Can I set file permissions? HOT 2
- How to read assigned start and end pos data?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goleveldb.