route-rs / route-rs Goto Github PK
View Code? Open in Web Editor NEWA multithreaded, modular software defined router library, written in rust. Safe, fast, and extensible.
Home Page: https://route-rs.github.io/route-rs/
License: MIT License
A multithreaded, modular software defined router library, written in rust. Safe, fast, and extensible.
Home Page: https://route-rs.github.io/route-rs/
License: MIT License
Revisited my test case and realize the panic with a zero length queue doesn't actually occur until tear-down. It is possible to create and start using an AsyncElementLink with queue capacity of 0, but it deadlocks. I also tested this on PR #42 and encountered the same issue.
Unless a different stream is needed to accurately test, we should prefer ImmediateStream in our basic test cases since it runs quicker, allowing us to run the the suite of tests in less time.
Define traits that can be reused over different implementations.
We should have an element a link that discards all packets that are passed into it. Since this is super generic, it should go in the runtime rather than a separate elements crate.
Edit: I realized that this should be a link type rather than an element, because it will poll but not emit anything, which is different from how everything else works.
This would allow us to remove the *ElementOverseer
since this might remove the deadlock.
The observation is that the deadlock occurs when both Consumer and Provider have both parked their tasks in their respective channels and gone to sleep, so neither of them can wake up. In other words, there is no milk in the fridge. If we enforce that at most 1 task can be queued up, then we ensure that the deadlock never occurs.
Right now graphgen doesn't handle any sort of joining or splitting behaviors, so it's hard to write an interesting router with it. We should support JoinElementLink and ClassifyElementLink, which requires a bunch of thought about how to autogenerate the dispatching logic.
We should add more tests, specifically we need tests that contain more output branches, have longer to compute split functions, have multiple layers of channels. etc etc.
Sized
, Display
, Debug
?
Graphgen creates pipelines from a graph file. Right now, we don't have any tests that this continues to work, and we only have one example crate to test against. We should have an extensible testing framework for these and more examples.
We need to fully debate the best way to denote the versions of our dependencies in our cargo.toml files. Ideally we would set a specific version, maybe we could use more of a meta-version if they are available 'latest-stable' or whatever.
Crossbeam provides a Select
type that will choose among multiple input channels based on whichever one becomes ready first. This could possibly help with task waking in the JoinElement
.
Some testing elements, like our IdentityElement, keep being redefined in each element link module. Let's define it in a common location.
See #72 for concept
The Classify function in ClassifyElement could probably just return the port the packet should be output on, rather than a tuple of (port, packet).
We need move tests to figure out if there are any additional bugs with our AsyncElement. We need help writing more tests that do more interesting things. Even basic cases like a stream that immediately returns None as input. Or seeing if there are any tests that should panic and writing those.
Right now, graphgen doesn't have a way of determining whether to use a SyncElementLink or AsyncElementLink to connect elements, because it doesn't know if the element is Async. We currently assume that everything is Sync.
We should be able to use the Rust Analyzer to find out if an element has the Async trait and act accordingly.
Most of the link still use names like join_element rather than join_link in their internal testing. This is confusing since these tests are primarily testing links rather than elements.
Yes
We'll need a link in the graph that can clone packets across multiple DownstreamProvider
s.
Right now, elements have no way to drop a packet, since they are forced to return a packet to the link due to the way their process function is implemented. We should change that, so that process returns the Option. Then links will unwrap the Option and forward the packet if it exists, if they get a None they will re-fetch and retry.
At this point, we've written several different types of Link structs, and they're shaping up to have a similar interface:
link.ingressor
is driven by Tokio to keep packets flowinglink.egressor
is polled by the next link to receive packetsWe should codify this interface in a trait. This also means that we can have accessor methods for the ingressor and egressor so that we're not calling into the struct's internals as much, which is kind of gross.
Let's add more tests, such as multiple layers of Join elements, add tests to ensure the join element can receive packets from all the other elements. Etc etc
Since we are just calling them links, lets change the struct to be AsyncLink instead of AsyncElementLink.
We wish that all of our various Elements would have the same trait bounds, especially the part of the trait that dictates that the Element inputs and outputs something of type Packet
. We should try to lean on traits composition to achieve this.
Same as #72
do client ip table?
Right now the API lives in a head crate, but we would like the repo to contain multiple independent crates. This will make testing easier, and keep dependencies cleaner in the project. Let's move the API to a crate sub-directory and give it a more descriptive name. #22
This is a bug which results when the channel between the AsyncElement consumer and provider fills to capacity, and is drained to empty repeatedly in quick succession. There is a race condition around writing the task handle to the other side and going to sleep. This can result in both the consumer and provider being asleep. We need to actually fix this bug.
To help with that we need:
An overseer part that can sidestep the issue for the time being
Better logging for the element. I'm thinking tokio-trace, with an emphasis on timestamps and such, so we know the order of events leading up to the deadlock.
Some sort of locking scheme that will prevent this issue, the better logging may help us figure out exactly what is in the race, and then lock that.
We noticed that among AsyncElementLink, ClassifyElementLink, and JoinElementLink, their subcomponents such as their consumers and producers, contain a lot of reoccurring logic. We should try to compose this in a helper function.
AF_PACKET
unit tests should use the lo
interface, making it possible to trivially run these tests on any Linux system.
Additionally, we should figure out how to make this work in CI. In particular, we'd need to enable the capability CAP_NET_RAW
on our test binary for this to work.
This is referring to the elements provided in the Click paper.
These are:
Note that some of these elements don't map exactly to elements in route-rs, generally because:
We need an icon, IIRC Ben volunteered for this, something using the route-66 sign.
Let's move this over like how we did for Async Element, shouldn't be a big change.
Reference PR #51
We decided that the Consumer
Provider
names aren't clear enough in internal discussion. Let's update the Consumer
s and Provider
s of AsyncElement
, ClassifyElement
, and JoinElement
with clearer names.
It's easier to explain how route-rs
works visually since it's a graph computation. Let's create a diagram that we can reference internally when we're building components, or show to people who are interested. It's debatable whether or not we want to expose internal components of the API, depending on the audience.
For testing, we should have an element that takes in an int
and outputs a char
, for example.
See #72 for concept.
See here for more detail.
Much like the task_park used in the classify and async elements, we want to be able to use the task_park for thread sleeping in the join element.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.