ralexstokes / ssz-rs Goto Github PK
View Code? Open in Web Editor NEWImplementation of ethereum's `ssz`
License: Apache License 2.0
Implementation of ethereum's `ssz`
License: Apache License 2.0
some errors where silently dropped while working on this PR #25 (comment)
and the CI didn't raise any issue...
we should set up some code coverage and strive for 100%
analogues of these:
ralexstokes/ethereum-consensus#151
ralexstokes/ethereum-consensus#152
for this repo :)
across this repo and https://github.com/ralexstokes/ethereum-consensus
there are many bytes-like types that have specific ways of printing/serializing/deserializing
let's unify across both repos
relevant data to consider:
support proving and verification of Merkle (multi)proofs
A series of tasks to improve the test gen, which reads the test artifacts to make a set of Rust "harnesses" which consume the data to exercise the implementation.
justfile
https://github.com/ralexstokes/ethereum-consensus/blob/main/justfileIssue raised in comment from ethereum-consensus
Idea is to avoid something like the below and allow for a new List
to be initialized without allocating a Vec
and then converting into a List
.
let rotate_participation = vec![ParticipationFlags::default(); state.validators.len()];
state.current_epoch_participation = rotate_participation
.try_into()
.expect("should convert from Vec to List");
For example, add impl of new
for something like List<T>::new(capacity: usize)
(or, in the example above, just allow for state.current_epoch_participation = List::<u8>::new(state.validators.len())
.
Based on some initial research, I'm not quite sure how to do this since List
obviously takes List<T, N>
where N
is a const usize
...so List<T>
would be invalid. And without knowing N
at compile time, that introduces concepts like raw pointers, unsafe
rust, Box
, Arc
, etc. that I couldn't figure out how to use or if they even solve this problem.
Not really your problem, but would appreciate some
When compiling to wasm32-unknown-unknown
, this line overflows, since 2^32 cannot fit inside of usize
. I hacked my way around this by just hard coding it to 2^32-1, but I imagine that is going to break something elsewhere. Guessing the better move would be to convert to u64
.
Trying to compute hash_tree_root
of BeaconState
deserialized from "consensus-spec-tests/tests/mainnet/phase0/operations/block_header/pyspec_tests/success_block_header/post.ssz_snappy"
results in panic:
thread 'phase0::state_transition::block_processing::tests::test_process_block_header' panicked at 'range end index 32 out of range for slice of length 0', C:\Users\cravt\.cargo\git\checkouts\ssz_rs-4493665f49c79d5a\af8292a\ssz_rs\src\list.rs:182:17
stack backtrace:
0: std::panicking::begin_panic_handler
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\/library\std\src\panicking.rs:517
1: core::panicking::panic_fmt
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\/library\core\src\panicking.rs:100
2: core::slice::index::slice_end_index_len_fail
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\/library\core\src\slice\index.rs:41
3: core::slice::index::impl$3::index_mut<u8>
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\library\core\src\slice\index.rs:251
4: core::slice::index::impl$1::index_mut<u8,core::ops::range::Range<usize> >
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\library\core\src\slice\index.rs:26
5: alloc::vec::impl$17::index_mut<u8,core::ops::range::Range<usize>,alloc::alloc::Global>
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\library\alloc\src\vec\mod.rs:2508
6: ssz_rs::list::List<ethereum_consensus::phase0::validator::Validator,1099511627776>::compute_hash_tree_root<ethereum_consensus::phase0::validator::Validator,1099511627776>
at C:\Users\cravt\.cargo\git\checkouts\ssz_rs-4493665f49c79d5a\af8292a\ssz_rs\src\list.rs:182
7: ssz_rs::list::impl$12::hash_tree_root<ethereum_consensus::phase0::validator::Validator,1099511627776>
at C:\Users\cravt\.cargo\git\checkouts\ssz_rs-4493665f49c79d5a\af8292a\ssz_rs\src\list.rs:248
8: ethereum_consensus::phase0::beacon_state::impl$14::hash_tree_root<8192,16777216,2048,1099511627776,65536,8192,2048,4096>
at .\src\phase0\beacon_state.rs:64
9: ethereum_consensus::phase0::state_transition::block_processing::tests::impl$0::execute
at .\src\phase0\state_transition\block_processing.rs:834
10: ethereum_consensus::phase0::state_transition::block_processing::tests::impl$0::execute_from_glob
at .\src\phase0\state_transition\block_processing.rs:865
11: ethereum_consensus::phase0::state_transition::block_processing::tests::test_process_block_header
at .\src\phase0\state_transition\block_processing.rs:872
12: ethereum_consensus::phase0::state_transition::block_processing::tests::test_process_block_header::closure$0
at .\src\phase0\state_transition\block_processing.rs:871
13: core::ops::function::FnOnce::call_once<ethereum_consensus::phase0::state_transition::block_processing::tests::test_process_block_header::closure$0,tuple$<> >
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\library\core\src\ops\function.rs:227
14: core::ops::function::FnOnce::call_once
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c\library\core\src\ops\function.rs:227
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Code for reproduction:
fn test_panic() {
let path = "consensus-spec-tests/tests/mainnet/phase0/operations/block_header/pyspec_tests/success_block_header/post.ssz_snappy";
let post_state = read_ssz_snappy_from_test_data(&path).expect("file exists");
let mut post_state: BeaconState = deserialize(&post_state).expect("can deserialize");
let result = post_state.hash_tree_root();
println!("{:?}", result);
}
there are several ways to construct valid SSZ objects, e.g. from deserializing some bytes, or using a FromIterator
implementation.
let's make sure that any way to consume or produce bytes from this library only allows for valid SSZ. in particular, this means making sure there is no Rust facility to construct "invalid" instances
Add a blanket implementation for &T
such that if T: SimpleSerialize
then &T
also implements SimpleSerialize
.
And extend to &mut T
as well unless there is an obvious reason not to do so.
impls like this: https://github.com/ralexstokes/ssz_rs/blob/main/ssz_rs/src/bitlist.rs#L33-L47
do not verify there is a leading 0x
and that can lead to accidentally skipping the first byte of data
look into fixing with trim_start_matches
https://doc.rust-lang.org/std/string/struct.String.html#method.trim_start_matches and return an error if there is no leading 0x
I was trying to write test cases for https://github.com/ralexstokes/ethereum_consensus to check if operations work correctly, but faced problems with the deserialization of BeaconBlock
from block.ssz_snappy
Here is code for reproducing panic:
use ethereum_consensus::phase0::mainnet::BeaconBlock;
use project_root;
use snap;
use ssz_rs::prelude::*;
use std::fs::File;
use std::io::Read;
use std::path::Path;
use std::path::PathBuf;
#[test]
fn test_invalid_parent_root() {
let path = Path::new("consensus-spec-tests/tests/mainnet/phase0/operations/block_header/pyspec_tests/invalid_parent_root/");
let block_path = path.join("block.ssz_snappy");
let block_path = block_path.to_str().expect("is valid path");
let block_data = read_ssz_snappy_from_test_data(block_path);
println!("block data from file {:?}", block_data);
// for clarity
let default_block = BeaconBlock::default();
let ser_default_block = ssz_rs::serialize(&default_block).expect("can serialize");
println!("default block data {:?}", ser_default_block);
let block: BeaconBlock = ssz_rs::deserialize(&block_data).expect("can deserialize");
println!("block from file {:?}", block);
}
// Return SSZ-encoded bytes from test file at `target_path`
pub fn read_ssz_snappy_from_test_data(target_path: &str) -> Vec<u8> {
let project_root = project_root::get_project_root().unwrap();
let target_path = PathBuf::from(target_path);
let data_path = project_root.join(&target_path);
let mut file = File::open(&data_path).expect("can read file");
let mut data = vec![];
let _ = file.read_to_end(&mut data).expect("can read file data");
let mut decoder = snap::raw::Decoder::new();
decoder
.decompress_vec(&data)
.expect("can decompress snappy")
}
Output:
block data from file [1, 0, 0, 0, 0, 0, 0, 0, 174, 0, 0, 0, 0, 0, 0, 0, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 132, 104, 188, 10, 245, 110, 113, 55, 111, 77, 65, 137, 20, 99, 112, 142, 241, 25, 41, 49, 219, 43, 36, 235, 53, 18, 216, 51, 83, 152, 0, 250, 94, 3, 9, 107, 173, 249, 30, 84, 252, 0, 232, 139, 156, 254, 184, 37, 18, 253, 41, 181, 18, 160, 148, 212, 215, 136, 104, 3, 185, 127, 64, 10, 147, 134, 151, 134, 219, 131, 237, 40, 151, 198, 107, 90, 44, 119, 157, 111, 151, 200, 255, 110, 221, 150, 109, 13, 144, 129, 255, 201, 55, 90, 18, 40, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0]
default block data [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 192, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0, 220, 0, 0, 0]
thread 'test_invalid_parent_root' panicked at 'range end index 4 out of range for slice of length 0', /home/cravtos/.cargo/git/checkouts/ssz_rs-4493665f49c79d5a/7a29e15/ssz_rs/src/de.rs:50:42
stack backtrace:
0: rust_begin_unwind
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/std/src/panicking.rs:517:5
1: core::panicking::panic_fmt
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/panicking.rs:100:14
2: core::slice::index::slice_end_index_len_fail
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/slice/index.rs:41:5
3: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::index
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/slice/index.rs:240:13
4: <core::ops::range::RangeTo<usize> as core::slice::index::SliceIndex<[T]>>::index
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/slice/index.rs:286:9
5: core::slice::index::<impl core::ops::index::Index<I> for [T]>::index
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/slice/index.rs:15:9
6: ssz_rs::de::deserialize_variable_homogeneous_composite
at /home/cravtos/.cargo/git/checkouts/ssz_rs-4493665f49c79d5a/7a29e15/ssz_rs/src/de.rs:50:42
7: ssz_rs::de::deserialize_homogeneous_composite
at /home/cravtos/.cargo/git/checkouts/ssz_rs-4493665f49c79d5a/7a29e15/ssz_rs/src/de.rs:78:9
8: <ssz_rs::list::List<T,_> as ssz_rs::de::Deserialize>::deserialize
at /home/cravtos/.cargo/git/checkouts/ssz_rs-4493665f49c79d5a/7a29e15/ssz_rs/src/list.rs:142:22
9: ethereum_consensus::phase0::beacon_block::BeaconBlockBody<_,_,_,_,_,_>::__ssz_rs_set_by_index
at ./src/phase0/beacon_block.rs:21:5
10: <ethereum_consensus::phase0::beacon_block::BeaconBlockBody<_,_,_,_,_,_> as ssz_rs::de::Deserialize>::deserialize
at ./src/phase0/beacon_block.rs:8:26
11: ethereum_consensus::phase0::beacon_block::BeaconBlock<_,_,_,_,_,_>::__ssz_rs_set_by_index
at ./src/phase0/beacon_block.rs:41:5
12: <ethereum_consensus::phase0::beacon_block::BeaconBlock<_,_,_,_,_,_> as ssz_rs::de::Deserialize>::deserialize
at ./src/phase0/beacon_block.rs:28:26
13: ssz_rs::deserialize
at /home/cravtos/.cargo/git/checkouts/ssz_rs-4493665f49c79d5a/7a29e15/ssz_rs/src/lib.rs:62:5
14: operations_tests::test_invalid_parent_root
at ./tests/operations_tests.rs:41:30
15: operations_tests::test_invalid_parent_root::{{closure}}
at ./tests/operations_tests.rs:27:1
16: core::ops::function::FnOnce::call_once
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:227:5
17: core::ops::function::FnOnce::call_once
at /rustc/f1edd0429582dd29cccacaf50fd134b05593bd9c/library/core/src/ops/function.rs:227:5
Same panic if I try to deserialize serialized BeaconBlock::default()
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.