Comments (9)
Yeah I would like to figure out how to support a Source[T: Pickleable] in my apis.
from autowire.
Yeah, the "default" abstractions around Autowire has the following requirements:
- You can pickler
T
s intoV
s, whereT
is the input type andV
is a type of your choosing - You can ship the
V
s over the wire somehow, as bytes - You and reconstitute those bytes into a
V
on the other sideand - You can unpickle the
V
s intoT
s again.
uPickle clearly doesn't support binary data well by default (tho you could pickle binary data into base64 if you wish, at a 30% data-size penalty) and BooPickle seems not to support streaming stuff by default.
But if you look at the specification, presumably you can't efficiently collapse a Source[T]
into a "flat" T
without buffering it all up in memory, but could you:
- "pass through" the streaming
Source
structure and include it inside some kind ofV
- Pass the
V
containing the streamingSource
over the network in an efficient/streaming way - Re-constitute the
V
with a streamingSource
inside, without needing to wait for the entireSource
to turn up - Unpickle the
V
-containing-streaming-Source
into aSource[T]
on the other end
That seems like something that should work, even without needing any changes to Autowire itself. You'd just need to choose a different data-structure for V
that isn't just a strict Js.Value
or ByteString
as uPickle or BooPickle
do, but some kind of composite data structure with both strict and streaming portions, with your own T: VWriter: VReader
to convert your T
s into your V
s and back again.
You could still use uPickle
or BooPickle
as part of your implementation of VWriter
and VReader
for the individual items or the strict parts, and will just need your own logic to deal with preserving streaming things and streaming them over the network
from autowire.
I agree that support for streams would be useful. Slick 3.0.0 may not be the best example as its operations return Future
s which are already supported by autowire.
from autowire.
It can also return a Publisher for Akka Streams =>
http://slick.typesafe.com/doc/3.0.0/dbio.html#streaming
from autowire.
I'm wondering too whether this is supported.
The word stream
only occurs once in Autowire's code base and there it's java.io.ObjectInputStream
. So I guess it's not implemented yet.
from autowire.
We would have to implement something like protobuffers (grpc) for that I guess
from autowire.
I can pickle everything to byte arrays using boopickle. But trying to define a pickler for a Source[Bytestring] is where I got hung up I think.
from autowire.
Discussions about marshalling streams in Akka
and Alpakka
- akka/akka#21826
- akka/alpakka#63
- https://github.com/Tradeshift/ts-reaktive/tree/master/ts-reaktive-marshal
The above idea is more generic because it proposes marshalling the Source
as opposed all the stream
data itself.
from autowire.
gRPC
has some design elements that could be adopted in autowire
, I think.
It uses stream
key word on *.proto files to define streaming APIs
- https://grpc.io/docs/tutorials/basic/java.html#example-code-and-setup
- https://github.com/grpc/grpc-java/blob/master/examples/src/main/proto/route_guide.proto#L52
It has the concept of transport
https://github.com/grpc/grpc-java#transport: in autowire
this is only implied.
It (gRPC
) also has a runtime lib, io.grpc
, that implements StreamObserver
.
Then, it compiles proto code like this
rpc RouteChat(stream RouteNote) returns (stream RouteNote) {}
to this
/**
* <pre>
* A Bidirectional streaming RPC.
* Accepts a stream of RouteNotes sent while a route is being traversed,
* while receiving other RouteNotes (e.g. from other users).
* </pre>
*/
public io.grpc.stub.StreamObserver<io.grpc.examples.routeguide.RouteNote> routeChat(
io.grpc.stub.StreamObserver<io.grpc.examples.routeguide.RouteNote> responseObserver) {
return asyncUnimplementedStreamingCall(getRouteChatMethod(), responseObserver);
}
which then one can override to this
/**
* Receives a stream of message/location pairs, and responds with a stream of all previous
* messages at each of those locations.
*
* @param responseObserver an observer to receive the stream of previous messages.
* @return an observer to handle requested message/location pairs.
*/
@Override
public StreamObserver<RouteNote> routeChat(final StreamObserver<RouteNote> responseObserver) {
return new StreamObserver<RouteNote>() {
@Override
public void onNext(RouteNote note) {
List<RouteNote> notes = getOrCreateNotes(note.getLocation());
// Respond with all previous notes at this location.
for (RouteNote prevNote : notes.toArray(new RouteNote[0])) {
responseObserver.onNext(prevNote);
}
// Now add the new note to the list
notes.add(note);
}
@Override
public void onError(Throwable t) {
logger.log(Level.WARNING, "routeChat cancelled");
}
@Override
public void onCompleted() {
responseObserver.onCompleted();
}
};
}
@lihaoyi - thoughts on how welcome a design inspired from this would be in autowire
?
Wish transport
and something like StreamObserver
were explicit so that Ajax/HTTP and Kafka could be used to as implementations.
Also looking at https://github.com/cakesolutions/kafka-wire for inspiration.
from autowire.
Related Issues (20)
- Minimal example does not compile with upickle 0.4.4 HOT 3
- Pickle type for traits in method arguments HOT 1
- Custom header HOT 1
- simple example does not compile when using type parameters : could not find implicit value for evidence parameter of type MyReader[T] HOT 4
- cross-publish for 2.13.0-M1 HOT 5
- Can't Use API Factory Functions for Namespacing Purposes HOT 3
- Strange exception "autowire.Error$InvalidInput" with no message
- How to support server-side injection of data into API calls? HOT 4
- Possible issue using ClassTag in autowire calls HOT 1
- Support for implicit parameters in method?
- Can not assign server.route to a val and use that
- scala.MatchError of class autowire.Core$Request HOT 1
- Cannot find proxy for ...
- Parameterize effect type HOT 1
- Compilation fails in Scala 2.12.8 if one or more parameters have default values defined in the API trait HOT 3
- Need a license for this project HOT 2
- Allow customization of the API endpoint
- Autowire in AWS Lambda Java runtime HOT 1
- Autowire maintenance HOT 11
- Scala 3 support HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from autowire.