Comments (7)
we tried but restoration failed with fatal error:
INFO[0002] successfully fetched data of base snapshot in 1.5047977750000001 seconds [CompressionPolicy:gzip] actor=restorer unexpected fault address 0x776bdb053000 fatal error: fault [signal SIGBUS: bus error code=0x2 addr=0x776bdb053000 pc=0xf19fd6] goroutine 1 [running]: runtime.throw({0x207bb9d?, 0x0?})
It means the snapshot operation actually failed. So the received snapshot isn't a completed snapshot.
from etcd.
There are two possible reasons,
- The "snapshot" wasn't actually a snapshot, it might be just copied from the db file directly. Please run
etcdutl snapshot restore path-2-snapshot --skip-hash-check=true
to double check; - Some errors happened when generating the snapshot. The command mentioned above should fail. Did you see any error on either the client side or the server side when generating the snapshot?
from etcd.
The "snapshot" wasn't actually a snapshot, it might be just copied from the db file directly.
it was the snapshot
as we call snapshot api to take the snapshot (named it as full-snapshot
)
Please run etcdutl snapshot restore path-2-snapshot --skip-hash-check=true to double check;
we tried but restoration failed with fatal error:
INFO[0002] successfully fetched data of base snapshot in 1.5047977750000001 seconds [CompressionPolicy:gzip] actor=restorer
unexpected fault address 0x776bdb053000
fatal error: fault
[signal SIGBUS: bus error code=0x2 addr=0x776bdb053000 pc=0xf19fd6]
goroutine 1 [running]:
runtime.throw({0x207bb9d?, 0x0?})
Did you see any error on either the client side or the server side when generating the snapshot?
unfortunately I don't have logs, I saw this occurrence twice. First, in one of our test cluster which don't have observability stack, hence I'm unable to get logs and another occurrence is reported by one of our community user: gardener/etcd-backup-restore#749
from etcd.
So the received snapshot isn't a completed snapshot.
yes, it seems ... Is there a way to verify the integrity of snapshot either on etcd side before sending the snapshot or on etcd client side ?
For verifying the integrity of snapshot on etcd client side, I thought of this. It's a similar way how's restoration is verifying the snapshot before restoration.
Calculate the hash of snapshot (removing the appended hash), compare it with hash value which is appended by etcd on snapshot. If it matches then our snapshot integrity is intact else re-try to take the snapshot again till hash matches.
What do you think ?
from etcd.
You need to use the client side error to detect such failure.
Lines 216 to 226 in 9a55333
Also I just had a quick read on the server side implementation, it seems that there is a minor issue on the pb.SnapshotResponse.RemainingBytes
. The value of total doesn't include the sha256 checksum, so when the RemainingBytes == 0
, the server side may not have sent out the checksum yet. But your issue isn't caused by this minor issue.
etcd/server/etcdserver/api/v3rpc/maintenance.go
Lines 148 to 198 in d6c0127
from etcd.
You need to use the client side error to detect such failure.
we do handle the error at client side while taking the etcd snapshot but I guess client side didn't throw any error.
from etcd.
The value of total doesn't include the sha256 checksum, so when the RemainingBytes == 0, the server side may not have sent out the checksum yet. But your issue isn't caused by this minor issue.
why this issue is not caused by this ? TBH, to me it feels it caused by this as it sends the snapshot but failed to send the sha256 checksum and due to this there was no client side error detected as it feels snapshot taken was successful but it fails during restoration as it fails hash check/validation.
from etcd.
Related Issues (20)
- option go_package in proto files HOT 4
- Client v3 and GRPC versions conflict HOT 3
- Member remains learner after promotion HOT 1
- Improvement of OpenSSF Scorecard Score HOT 4
- Enable `unexported-return` linter rule HOT 7
- High memory usage (memory leak?) with 1 node cluster and small DB (but a lot of updates) HOT 8
- The second node took about 7 seconds to join in a single node cluster HOT 9
- Add hashKV checking to robustness test HOT 8
- Introduce a Go workspace HOT 2
- Need to update golang pkg.dev document HOT 4
- support compile `tools` in makefile HOT 2
- Detected disallowed custom content in v2store
- Confusing documentation for `WatchResponse.canceled`
- etcd cluster crash
- Bump Go to 1.21.13/1.22.6 HOT 10
- etcd Go & Java client SDK's retry mechanism may break `Serializable` HOT 5
- something wrong with simple_token HOT 7
- Update main branch to Go 1.23 / release branches to 1.22 HOT 7
- Tidy up infra docs to remove old documentation HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from etcd.