Comments (5)
Some other failures also appeared when running locally using taskset -c 0
, where most seem fixed with waitUntil
or derivates. One thing I want to do in the future is to abstract some of the boilerplate in the tests, something like an AbstractClusteredTest
.
from jgroups-raft.
I'll take this as a chance to do the above.
I'll create abstract classes to separate the test setup and combine the assertions and utilities. I'll update to include the Assertj, too. Although a little more verbose than using assert
directly, it is very nice to write test assertions. I'll add everything in different commits to not mix the history too much.
from jgroups-raft.
I added the last commit (9a05186) with everything discussed above. The commit creates a base structure for tests and updates the existing tests. All the assertions are shared, so there is no need to fix them across multiple files, and we could relax some of them. For example, we only wait for a majority to have a leader instead of all nodes in the cluster.
While working on the tests, I found some corner cases to fix. Since I didn't open dedicated issues to track the problems, I'll describe them here.
- Sending commits immediately after commit. It only affects it relying solely upon sending the entries after commit, but I believe everyone uses the task with the resend-interval configured. The issue happens with a message reordering. Since the entry is committed on a majority, when sending after a commit, one of the nodes might not know the entry to commit yet. This would cause the node to never commit since the commit message arrives before the actual entry message. I highly doubt this affects anyone because it needs to have the resend interval "disabled".
- Chaining the membership CF with the
.thenCompose
only. If the future ever fails exceptionally, the user is unable to issue membership changes after because the.thenCompose
is not invoked. - Completing the pending requests on the
RequestTable
when stepping down. Instead of letting requests time out, we complete them exceptionally. Previously, if a request doesn't have a timeout configured and the leader steps down, the request is never complete.
There is more future work, but I wanted to push what I had. We can add later a few utilities to help in some tests, something similar to the CheckPoint
class we have in ISPN to test and stop at specific events. Include a StateMachine implementation relying on the CheckPoint
to more easily test failure cases.
Some of the new tests to add include checking the commands complete when the leader steps down (exceptionally only for now). Assert leadership changes while applying operations, and verify redirecting during leader changes.
from jgroups-raft.
And to add a little more context. I am happy with the final result of the test suite. When we started, running with taskset and single core was frequently failing. After applying all the changes, I let it run in a while loop with while taskset -c 0 mvn clean test; do :; done
for around 2hrs, with each execution taking ~50s. It ran all this time without failures.
from jgroups-raft.
Perfect!
from jgroups-raft.
Related Issues (20)
- Remove AsyncCounter and SyncCounter interfaces HOT 1
- withOption method should create a new instance
- Add membership operations to RaftHandle
- Provide quorum reads for `ReplicatedStateMachine`
- `Client` and `ClientStub` issues during membership operations
- Liveness issue with ELECTION HOT 3
- can't remove leader by method of RaftHandle's removeServer HOT 9
- this.raftHandle.channel().disconnect() and reconnect trigger error "not found in retransmission table" HOT 7
- Questions HOT 2
- Ensure a single leader per term
- Configurable class loader for ReplicatedStateMachine block
- Remove ant and use only maven HOT 2
- Restarting node after membership change
- Leader stepping down with membership change HOT 2
- Fix longest log check during election
- Liveness issue joining during election thread execution
- Concurrency issue leads to nodes voting twice in same term
- Election problem HOT 2
- The election process may elect a leader who has left the group HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from jgroups-raft.