Comments (1)
bees doesn't currently build a complete extent map due to the limitations of LOGICAL_INO
(v1) and because bees doesn't have data structures to efficiently use the data. It currently works off partial information from the subvol tree.
If a single extent contains multiple identical blocks and a second extent contains another copy of these blocks, bees can thrash by constantly replacing some references to one of the blocks with references to the other two. The second extent triggers the problem because without it bees would never detect two instances of data in the same extent (such data is not added to the hash table until after the extent is scanned for duplicates to try to avoid this problem). The second extent is usually eliminated, but the first extent (which has two copies of the block) keeps getting new references to the "other" duplicate block because the old references are never fully deleted and there is no extent-tree lookup to see if both blocks belong to the same extent. The new extent refs feed back into the scanner, so the offending extent refs get deduped on every pass over a subvol, which is a waste of time.
bees can detect when it's doing this by looking up the physical addresses of both extents in dedup. If they share a physical extent bytenr, we know that we are doing a pointless dedup, and we can avoid feeding the extent back into the next pass.
So bees detects this pathological case and bombs out with an exception. That could be improved--we can just quietly log and return dedup failure. The exception was useful to collect the data to understand this phenomenon but now that I understand what is happening the exception is no longer necessary.
A proper extent-based scanner would not make this kind of mistake--it would fully map each physical extent, map and dedup on the physical extent level instead of the subvol extent ref or block levels, so that any block that is deduped is fully removed from the filesystem on the first pass and never visited again. Each matching block is mapped to its extent and any self-references ignored. When bees gets one of those, this issue just goes away.
from bees.
Related Issues (20)
- Mount prevent second run HOT 1
- bees seemingly cannot catch up with snapper snapshot creation HOT 16
- Initial run of bees appears to have resulted in data corruption HOT 7
- bees breaks existing reflinks? HOT 4
- bees "--loadavg=3" option causes load crazy 20+ and OOM kill HOT 1
- beesd script can't handle BEESHOME on a non-btrfs filesystem HOT 4
- Stopping one of multiple beesd services removes the /run/bees directory HOT 4
- 执行bee run之后一直卡出不动 HOT 2
- Lookup root 256 ino xxx failed: Bad address HOT 1
- Documentation: modern Ubuntu install HOT 1
- [Feature Request] Unmount mount points if beesd exits.
- [Feature Request] Beesd to run a full dedup cycle and then end HOT 6
- How to remove? HOT 10
- cancelling with CTRL+C need to be followed by umount /run/bees/mnt/$UUID HOT 3
- build fails on Fedora 40
- Demystifying needed options with QubesOS pool in btrfs reflink (multiple cow snapshots rotating, beesd dedup and load avg hitting 65+ on 12 cores setup) HOT 4
- btrfs send size HOT 3
- one question: multiple cloned sets, two proposals: de-duplicate on write, 'only new' option, HOT 5
- How to Force Rescan after increasing Hash Table Size? HOT 4
- optimization - de-optimization, fragmentation, how to use efficiently, lock down risk. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bees.