appaquet / exocore Goto Github PK
View Code? Open in Web Editor NEWA distributed private application framework
License: Apache License 2.0
A distributed private application framework
License: Apache License 2.0
Allow embedding frames into frames.
Instead of having a seperate signature, we could just have a SignedFrame wrap a NonSignedFrame. This would allow preventing having an Option of the signature and be clear about what is what.
Right now, transport only support message passing. In order to optimize chain bootstrapping, we need to be able to support streaming.
The way it could be implemented is to add an optional stream / sink field in InMessage / OutMessage.
In libp2p implementation, we need a new ProtcolHandler to support this. We use the "OneShot" handler that works for message passing, but not for streaming
-Block depth => block height
-Block should perhaps be renamed BlockHeader ?
-Rename pending_operation to something else
-ChainBlockIterator -> BlockRefIterator
Right now, the pending store is just infinitly collecting operations, even if they had been committed to the chain store.
In order to cleanup, we need to implement a logic based on how deep (configurable) an operation is now in the chain, so that we are pretty confident that all nodes have that operation committed.
This logic will happen in the commit manager, since its the piece that is responsible for merging the pending and chain stores.
Exceptions:
We need to:
Questions
TODO
Chain synchronizer has to wait for maximum leader depth delta (max_leader_common_block_delta
) before trying to re-elect a master and find out all nodes have diverged.
We can do this faster in commit manager if we detect that majority of nodes are voting for blocks that are divergent from our local chain
This stream will be used by engine's users to be notified of changes in the different stores. The index layer will use this stream to know when it needs to reindex certain segments. This stream should probably throttled at the user level depending on the amplification (ex: we don't want to reindex on every change if they are happening at high rate).
A handle should also be notified if its stream had been discontinued since the channel used to back the stream will be bounded to prevent potential memory blow up.
Will be needed by the engine for replication and lookup what's in the chain right now vs pending
Need a way to store node's keys to disk
TODO:
TODO:
To think:
As mentioned in #41, the commit manager and the pending synchronizer need to rely on the chain synchronization status to know if they can proceed with their operations. This is crucial to prevent old operations from reviving if a node had been partitioned for a while.
This issue is about implementing the logic to detect when the local chain status is deemed uncertain because we're partitionned, we've diverged, we're lagging behind other nodes, etc.
Check for: https://docs.rs/kaos/0.1.1-alpha.2/kaos/
See https://github.com/appaquet/capnproto-rust/tree/thread-safe-arena-readlimiter
The atomic integers should be stabilized soon as the PR was merged on Jan 7th 2019: rust-lang/rust#57425
Right now, pending store is kept in memory. In order to increase resilience of it in case of crash, we need persist it.
The ideal persistence would be a WAL since the pending store is continuously being appended and cleaned up. We could directly mmap the operations data from it instead of storing them in memory.
Operations would still be in memory, but not their frame data.
TODO:
Depends on #34
Notes:
Questions
Queue messages until node is connected to prevent losing messages if peer isn't online yet
The commit manager stores operations by operation id. We should do binary lookup within the block instead of iterating through all operations to find an operation in Block
's get_operation
method.
Notes
We need a proper logic to propose blocks to be added to the chain so that we minimize odds of having multiple blocks generated at the same time by multiple nodes, which would result in splitting the concensus.
We also need a logic to select the best block to be signed when mutliple blocks are proposed.
We also need a logic to make proposed blocks timeout if no concesus happened after a while because of the presence of multiple blocks that split the concensus.
We also need a logic to detect stale commits, and propose another block (time based?)
A block should also have a limit amount of operations
Notes
See ring's build script: https://github.com/briansmith/ring/blob/ab0726d0cdb0c9aafd787e772592331b610acc75/mk/travis.sh
We should use the Failure's crate type of Error for better backtrace tracking and enforce proper display message.
See implementation in the framed.rs and the pending store Error for examples.
To convert:
Engine, Transport, Chain, extindex, extsort
Make sure we have #[fail(cause)] where we need it
TODO
First place to be used is for operation ID generation. They need to be unique in the cluster, and monotonically increasing. Clock doesn't need to be super precise, but enough to prevent block im commit manager, and prevent IDs collision.
Time should be:
Search for all set_with_caveats
on builder to make sure we are fine
Prevent having to use unwrap() everywhere.
The failure crate has a this catch-all Error that should be returned by all tests. The only downside is that we need to add Ok(()) at the end of each test
Depends on #9
This is needed to sign frames, blocks, messages, etc.
Framing support signature, but it's only hashing the message using SHA3 256 right now.
Possible signature source:
Problems:
Notes
Needs to opensource first
A Cell
should contains:
NodeID
& CellID
should be structs and should be renamed for Id
and not ID
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.