eigr / massa Goto Github PK
View Code? Open in Web Editor NEWA Stateful Serverless framework on the BEAM VM
Home Page: https://eigr.io
License: Apache License 2.0
A Stateful Serverless framework on the BEAM VM
Home Page: https://eigr.io
License: Apache License 2.0
During some tests @marcellanz found this error message:
Describe the bug
While I was working to add support for Forwards and Side Effects I ended up noticing that the gRPC client of the Elixir library that uses the Gun library had some errors when I tried to return a Stream mapped with the results of the requests to the user role as a return in calling a GenServer.
Error:
I was only able to solve the problem when I materialized the stream through an Enum.to_list() and returned a list instead of the Stream:
While this works for the moment and I keep this workaround for now I believe this approach is not the best approach to what we need to do.
The Elixir gRPC library is pretty bad to work with stream, its API is quite confusing and passive of errors in its use and from time to time I end up encountering these kinds of strange errors.
It is essential that we manage to work well with un-bounded streams (infinite streams) and we need to quickly find an alternative to this library. We recently forked this library and renamed it Falco, we need to speed up work on this version to resolve these issues
Hi!
First of all this is a very interesting project!
And I'd like to help out!
It's however a little unclear where efforts are best spent to move the project forward. What's already in progress etc. There are some open issues, I looked a little bit at #29. But couldn't find a reference to it in the cloudstate docs or an example client so I'm looking for a bit of guidance on how to get started, so any examples, docs I might have missed are much appreciated!
Add information about the project, our serveless vision, goals, and if possible some diagrams. We can take advantage of Marcel's presentations or material from eigr.io
During tests @ralphlaude found some errors during discovery phase.
This may be related to the issue #20
iex --name [email protected] -S mix
Erlang/OTP 23 [erts-11.1.8] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe] [dtrace]
2021-03-07 21:31:44.033 [[email protected]]:[pid=<0.655.0> ]:[warn]: [libcluster:dev] unable to connect to :"[email protected]"
2021-03-07 21:31:44.034 [[email protected]]:[pid=<0.655.0> ]:[warn]: [libcluster:dev] unable to connect to :"[email protected]"
2021-03-07 21:31:44.034 [[email protected]]:[pid=<0.664.0> ]:[info]: Starting Horde.RegistryImpl with name MassaProxy.GlobalRegistry
2021-03-07 21:31:44.035 [[email protected]]:[pid=<0.667.0> ]:[info]: Starting Horde.DynamicSupervisorImpl with name MassaProxy.GlobalSupervisor
2021-03-07 21:31:44.035 [[email protected]]:[pid=<0.671.0> ]:[info]: Starting Proxy Cluster...
2021-03-07 21:31:44.035 [[email protected]]:[pid=<0.671.0> ]:[info]: [massa proxy on :"[email protected]"]: Connecting Horde to :"[email protected]"
2021-03-07 21:31:44.035 [[email protected]]:[pid=<0.671.0> ]:[info]: [massa proxy on :"[email protected]"]: Connecting Horde to :"[email protected]"
2021-03-07 21:31:44.036 [[email protected]]:[pid=<0.671.0> ]:[debug]:Starting Supervisors...
2021-03-07 21:31:44.039 [[email protected]]:[pid=<0.551.0> ]:[info]: Starting Registry for EventSourced
2021-03-07 21:31:44.039 [[email protected]]:[pid=<0.677.0> ]:[info]: [MassaProxy on :"[email protected]"][EntityRegistry]: Initializing...
2021-03-07 21:31:44.039 [[email protected]]:[pid=<0.675.0> ]:[warn]: CRDT: %{}
2021-03-07 21:31:44.039 [[email protected]]:[pid=<0.675.0> ]:[warn]: Picked up [] for EventSourced
2021-03-07 21:31:44.039 [[email protected]]:[pid=<0.551.0> ]:[info]: Starting Registry for CRDT
2021-03-07 21:31:44.040 [[email protected]]:[pid=<0.678.0> ]:[info]: [MassaProxy on :"[email protected]"][EntityRegistry]: Initializing...
2021-03-07 21:31:44.040 [[email protected]]:[pid=<0.675.0> ]:[warn]: CRDT: %{}
2021-03-07 21:31:44.040 [[email protected]]:[pid=<0.675.0> ]:[warn]: Picked up [] for CRDT
2021-03-07 21:31:44.040 [[email protected]]:[pid=<0.551.0> ]:[info]: Starting Registry for Stateless
Interactive Elixir (1.11.3) - press Ctrl+C to exit (type h() ENTER for help)
2021-03-07 21:31:44.040 [[email protected]]:[pid=<0.679.0> ]:[info]: [MassaProxy on :"[email protected]"][EntityRegistry]: Initializing...
2021-03-07 21:31:44.040 [[email protected]]:[pid=<0.675.0> ]:[warn]: CRDT: %{}
2021-03-07 21:31:44.040 [[email protected]]:[pid=<0.675.0> ]:[warn]: Picked up [] for Stateless
iex([email protected])1> 2021-03-07 21:31:44.041 [[email protected]]:[pid=<0.551.0> ]:[info]: Starting Elixir.Discovery.Worker on target function address tcp://127.0.0.1:8080
2021-03-07 21:31:50.043 [[email protected]]:[pid=<0.682.0> ]:[error]:GenServer Discovery.Worker terminating
** (stop) bad return value: %ArgumentError{message: "argument error"}
Last message: :work
State: []
2021-03-07 21:31:50.043 [[email protected]]:[pid=<0.551.0> ]:[info]: Starting Elixir.Discovery.Worker on target function address tcp://127.0.0.1:8080
2021-03-07 21:31:56.046 [[email protected]]:[pid=<0.745.0> ]:[error]:GenServer Discovery.Worker terminating
** (stop) bad return value: %ArgumentError{message: "argument error"}
Last message: :work
State: []
2021-03-07 21:31:56.046 [[email protected]]:[pid=<0.551.0> ]:[info]: Starting Elixir.Discovery.Worker on target function address tcp://127.0.0.1:8080
2021-03-07 21:32:02.049 [[email protected]]:[pid=<0.808.0> ]:[error]:GenServer Discovery.Worker terminating
** (stop) bad return value: %ArgumentError{message: "argument error"}
Last message: :work
State: []
This is a feature proposal.
We can add activation rules processing for entities, so we could assess whether a given entity should be activated (called) only if it meets the requirements proposed by the rules.
What should we change for this to work?
Move priv/protos files and respectives Elixir modules in the massa_proxy to the another app
There appears to be a bug in the initial support for kubernetes when using the DNS strategy.
The proxy does not identify any nodes even though the hostnames look correct. I suspect it is something with the configuration of the libcluster and headless.
It might be better to try the Cluster.Strategy.Kubernetes.DNSSRV strategy
Hi,
Just raising the suggestion following a discord discussion with @sleipnir. Vaxine (https://vaxine.io) is a project developing a rich-CRDT database, building on https://antidotedb.eu, with the core database in Erlang and the higher levels in Elixir.
Hopefully Vaxine could be one of the pluggable storage options underpinning the Eigr proxy CRDT entity types. Happy to feed in on interface design and put some time into a working demo integration if useful. On which, it would be useful to understand:
Thanks,
James.
For mongoose we will need a Config.Provider that reads the configuration entries via kubernetes configmap
References:
https://hexdocs.pm/elixir/master/Config.Provider.html
https://hexdocs.pm/distillery/extensibility/config_providers.html
We need to change fetch-cloudstate-pb.sh to pull files from Permastate instead of CloudState repository
Implement gRPC transcoding to provide an HTTP server to reflect the user function
@marcellanz @ralphlaude This is a good first issue to start
https://github.com/bake-bake-bake/bakeware/releases/tag/v0.2.0
It may be possible to cache the modules that were created after the discovery and subsequent creation of the gRPC server. This would be useful for restarts of the container optimizing the boot time of the application, it could also be useful in case of initialization of replicas, if the cache is distributed.
Check if we can use: code.load_binary for this purpose, something like:
iex([email protected])2> {_, mod, bytecode, _} = defmodule Foo do
...([email protected])2> def thing(), do: "hello"
...([email protected])2> end
{:module, Foo,
<<70, 79, 82, 49, 0, 0, 4, 208, 66, 69, 65, 77, 65, 116, 85, 56, 0, 0, 0, 123,
0, 0, 0, 13, 10, 69, 108, 105, 120, 105, 114, 46, 70, 111, 111, 8, 95, 95,
105, 110, 102, 111, 95, 95, 10, 97, 116, ...>>, {:thing, 0}}
iex([email protected])3>
iex([email protected])4> :code.purge(Foo)
false
iex([email protected])6> :code.delete(Foo)
true
iex([email protected])8> :code.load_binary(mod, 'filename', bytecode)
{:module, Foo}
iex([email protected])10> Foo.thing
"hello"
iex([email protected])11>
Running mix test
with a fresh pull of massa on OTP 24 and OTP 23 produces an error like the following:
ASDF .tool-versions
erlang 24.2.1
elixir 1.13.2-otp-24
Error
** (Mix) Could not start application massa_proxy: exited in: MassaProxy.start(:normal, [])
** (EXIT) an exception was raised:
** (ArgumentError) errors were found at the given arguments:
* 1st argument: the node name is not part of a distributed system
:erlang.set_cookie(:nonode@nohost, :massa_proxy)
(massa_proxy 0.1.0) lib/massa_proxy/util.ex:54: MassaProxy.Util.setup/0
(massa_proxy 0.1.0) lib/massa_proxy.ex:10: MassaProxy.start/2
(kernel 8.2) application_master.erl:293: :application_master.start_it_old/4
ASDF .tool-versions
erlang 23.3.4.11
elixir 1.13.3-otp-23
Error
** (Mix) Could not start application massa_proxy: exited in: MassaProxy.start(:normal, [])
** (EXIT) an exception was raised:
** (FunctionClauseError) no function clause matching in :erlang.set_cookie/2
:erlang.set_cookie(:nonode@nohost, :"6eycE1E/S341t4Bcto262ffyFWklCWHQIKloJDJYR7Y=")
(massa_proxy 0.1.0) lib/massa_proxy/util.ex:54: MassaProxy.Util.setup/0
(massa_proxy 0.1.0) lib/massa_proxy.ex:10: MassaProxy.start/2
(kernel 7.3.1.4) application_master.erl:277: :application_master.start_it_old/4
May relate to erlang/otp#5402 which seems to have a fix in main pending release: erlang/otp#5670
Server Reflection is a gRPC feature that allows ‘dynamic’ clients, such as command-line tools for debugging, to discover the protocol used by a gRPC server at run time. They can then use this metadata to implement things like completion and sending arbitrary commands.
However, for the proxy that implements the CloudState protocol, it goes a little further by making the proxy capable of creating an implementation based on the client services specification (user functions). For this reason I consider reflection to be two items to be implemented:
For the first item we have to consider implementing the following issue:
elixir-grpc/grpc#148
For the second item, we have to implement the following steps:
1 - Create the FileDescriptor from what arrived at EntityDiscovery
2 - Use the internal apis of the elixir protobuf library to generate the source code of the Elixir modules of these Protos.
3 - Compile these codes with Code.compile_string or compile_file
4 - Dynamically generate the module with the client's gRPC implementation, something like
defmodule Mongoose.ModuleName
use GRPC.Server, service: ServiceNameHere
// define the calling method here, this method should call a Mongoose handler, that is, it is just a facade
end
5 - Create the module with the Endpoint (or leave a pre-defined module that injects the endpoints via metaprogramming)
6 - Compile the generated module
7 - Bootstrap the entire gRPC code over your own supervision tree.
Task implementation Index:
There are important things in this version
https://moosecode.nl/blog/whats_new_horde_0_8_0
Example can be found here
We need to register protobuf extension IDs for extension we use with our proxy protocol:
https://github.com/protocolbuffers/protobuf/blob/master/docs/options.md
I've requested extensions: 1110-1114.
I would like to have our own protobufs without depending on the Cloudstate protocol, I think certain things will have to evolve without this dependence on Cloudstate.
Obviously I appreciate the compatibility with the Cloudstate protocol, after all I invested a lot of time in it. But I think we can guarantee this as a compatibility mode in the proxy, that is, if the client chooses to use the Cloudstate protocol, then the proxy could select the adapter for this protocol but that was not our default mode, opting primarily for our protocol itself instead.
Just for reference as a little has been discussed here #14
Describe the bug
During the implementation of item #19, it became clear that the registration attempt approach cannot proceed until the stage of creating the gRPC server, as this means that in addition to the modules being compiled many times in an unnecessary way, the gRPC server itself binds to a door already activates what is not beneficial.
Below is a print of the error
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The Discovery service heratbeat should not attempt to re-register entities already registered, nor should it attempt to initialize a new gRPC server
It would be interesting to use Bakeware to build the application.
Below is an example of how to configure the application.
This can have an effect on the local build and it may be interesting to only exist in one release directory if you experience problems with the environment of many developers.
https://github.com/eigr/Astreu/blob/5fc1927a132d1b6c4129db0a8894d6d15f653711/mix.exs#L33
https://github.com/eigr/Astreu/blob/5fc1927a132d1b6c4129db0a8894d6d15f653711/mix.exs#L52
Bye bye litle Mongoose :'(
New node joining does not receive entities from other nodes
A node joining a cluster does only dispatch a {:join, entities}
to the other nodes of the cluster
but does not receive any information about the entities which the other nodes hold. So if a node joins after
another node the first node will not receive any entity information.
Edit: I noticed the orchestrator runs the discovery periodically, which will fix the issue I'm describing. It's however
once every minute, so perhaps another strategy can be considered as a new node joining the cluster possible won't have the correct topology for ~1 minute.
To Reproduce
# Will discover user function on 8080
NODE_COOKIE=secret PROXY_PORT=5000 PROXY_HTTP_PORT=5001 iex --name [email protected] -S mix
# Will have no entities
USER_FUNCTION_PORT=4000 NODE_COOKIE=secret iex --name [email protected] --cookie secret -S mix
GenServer.call(MassaProxy.Entity.EntityRegistry, {:get, "cloudstate.action.ActionProtocol"})
to see that it's emptyExpected behavior
A node joining a cluster should receive information about the entities that is registered stred on that node.
I.e when a EntityRegistry
receives a join message from it should respond to the node with the entities.
Create docker-compose file with a proxy and one user-function in de deployments folder to local tests
We may have to have our own elixir image based on erts 24 with BeamJIT enabled
Complementing the #13
Describe the bug
When a node in the cluster terminates the entities of the node that left the cluster are not removed from the other nodes
To Reproduce
Steps to reproduce the behavior:
Start more than one instance and then remove one of those instances.
Expected behavior
The callback for the message: leave will not be performed. No message indicating your call will be printed in the log. Correct behavior should be the opposite of this.
Starting massa with:
PROXY_HEARTBEAT_INTERVAL=3000 iex --name [email protected] -S mix
leads to
2021-12-28 12:28:19.269 [[email protected]]:[pid=<0.652.0> ]:[error]:GenServer MassaProxy.Orchestrator terminating
** (ArgumentError) errors were found at the given arguments:
* 1st argument: not an integer
:erlang.send_after("3000", #PID<0.652.0>, :work)
(massa_proxy 0.1.0) lib/massa_proxy/orchestrator.ex:52: MassaProxy.Orchestrator.schedule_work/1
(massa_proxy 0.1.0) lib/massa_proxy/orchestrator.ex:47: MassaProxy.Orchestrator.handle_info/2
(stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4
(stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6
(stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Last message: :work
State: []
``
With the arrival of the first version of Massa close, we need to start developing Operator. It would be interesting to have both proxy and operator functional in the same release.
Previously I imagined using the pre-existing Operator, but maybe it is better to start something that has a fine adjustment with our proxy.
In terms of technologies I see two main options, the first using Go and the second, less traditional but very simple, using Elixir with the Bonny library.
I've been looking at the examples and it seems to me to be very easy to implement an Operator with Bonny, I didn't think there was a need for the experience in Elixir not even with the development of operators to use this library.
I suggest that you take a look at this to help the decision.
What is the best way to implement passivation and activation of entities?
Allow to send side effects and forwards inter cluster
I think we could start to work on the TCK. We start with the action protocol, find a way to run TCK tests. We can use any of the TCK user functions already implemented.
Topics to work on
gcr.io/eigr-io/eigr-go-tck-action
but we might separate them also by code as this image was done ad-hoc.Describe the bug
When making a request via stream, the first payload sent in the stream is not sent to the user function.
Example:
/home/sleipnir/go/bin/grpcurl -d @ --plaintext localhost:9000 cloudstate.tck.model.action.ActionTckModel/ProcessStreamed <<EOF
{
"groups":{
"steps":[
{
"reply":{
"message":"The north face was first climbed on July 24, 1938"
}
}
]
}
}
{
"groups":{
"steps":[
{
"reply":{
"message":"The north face was first climbed on July 24, 1939"
}
}
]
}
}
{
"groups":{
"steps":[
{
"reply":{
"message":"The north face was first climbed on July 24, 1940"
}
}
]
}
}
{
"groups":{
"steps":[
{
"reply":{
"message":"The north face was first climbed on July 24, 1941"
}
}
]
}
}
EOF
The logs resulting from this request is:
2022-02-13 06:41:27.112 [[email protected]]:[pid=<0.3878.0> ]:[debug]:Running client stream #Function<59.58486609/2 in Stream.transform/3>
2022-02-13 06:41:27.112 [[email protected]]:[pid=<0.3878.0> ]:[debug]:send_stream_msg %Cloudstate.Action.ActionCommand{metadata: %Cloudstate.Metadata{entries: []}, name: "ProcessStreamed", payload: nil, service_name: "cloudstate.tck.model.action.ActionTckModel"}
2022-02-13 06:41:27.113 [[email protected]]:[pid=<0.3878.0> ]:[debug]:send_stream_msg %Cloudstate.Action.ActionCommand{metadata: nil, name: "", payload: %Google.Protobuf.Any{type_url: "type.googleapis.com/cloudstate.tck.model.action.Request", value: "\n7\n5\n3\n1The north face was first climbed on July 24, 1939"}, service_name: ""}
2022-02-13 06:41:27.113 [[email protected]]:[pid=<0.3878.0> ]:[debug]:send_stream_msg %Cloudstate.Action.ActionCommand{metadata: nil, name: "", payload: %Google.Protobuf.Any{type_url: "type.googleapis.com/cloudstate.tck.model.action.Request", value: "\n7\n5\n3\n1The north face was first climbed on July 24, 1940"}, service_name: ""}
2022-02-13 06:41:27.114 [[email protected]]:[pid=<0.3878.0> ]:[debug]:send_stream_msg %Cloudstate.Action.ActionCommand{metadata: nil, name: "", payload: %Google.Protobuf.Any{type_url: "type.googleapis.com/cloudstate.tck.model.action.Request", value: "\n7\n5\n3\n1The north face was first climbed on July 24, 1941"}, service_name: ""}
As you can see from the payload: nil the first message goes with the payload attribute empty, and that's why in the reacquisition return only three messages are returned instead of four:
...
EOF
{
"message": "The north face was first climbed on July 24, 1939"
}
{
"message": "The north face was first climbed on July 24, 1940"
}
{
"message": "The north face was first climbed on July 24, 1941"
}
To Reproduce
Steps to reproduce the behavior:
docker run -it --rm -p 8090:8080 gcr.io/eigr-io/eigr-go-tck-action
MIX_ENV=prod USER_FUNCTION_PORT=8090 iex --name [email protected] -S mix
Expected behavior
That the response of all messages return in the console.
I'd be great to have a simple and pragmatic deployment story right from the beginning.
devcontainer
.mg deploy --repository https://github.com/mungotinae/shoppingcart-example/ --auth [email protected]
make build fails with:
% make build
docker build -f Dockerfile-Bakeware -t eigr/massa-proxy:0.1.36 .
[+] Building 1.1s (14/21)
=> [internal] load build definition from Dockerfile-Bakeware 0.0s
=> => transferring dockerfile: 1.03kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:3 0.0s
=> [internal] load metadata for docker.io/library/elixir:1.12-alpine 0.6s
=> [builder 1/8] FROM docker.io/library/elixir:1.12-alpine@sha256:557c1597322d0b10f255d3c208b24e51755ce64b9faee675940ca0f06ffe093c 0.0s
=> [internal] load build context 0.2s
=> => transferring context: 2.86MB 0.1s
=> [stage-1 1/8] FROM docker.io/library/alpine:3 0.0s
=> CACHED [stage-1 2/8] RUN apk add --no-cache --update openssl zstd 0.0s
=> CACHED [stage-1 3/8] WORKDIR /home/app 0.0s
=> CACHED [builder 2/8] RUN mkdir -p /app/massa_proxy 0.0s
=> CACHED [builder 3/8] WORKDIR /app/massa_proxy 0.0s
=> CACHED [builder 4/8] RUN apk add --no-cache --update git build-base zstd 0.0s
=> [builder 5/8] COPY . /app/massa_proxy 0.0s
=> ERROR [builder 6/8] RUN rm -rf /app/apps/massa_proxy/mix.exs && mv /app/apps/massa_proxy/mix-bakeware.exs /app/apps/massa_proxy/mix.exs 0.2s
------
> [builder 6/8] RUN rm -rf /app/apps/massa_proxy/mix.exs && mv /app/apps/massa_proxy/mix-bakeware.exs /app/apps/massa_proxy/mix.exs:
#14 0.186 mv: can't rename '/app/apps/massa_proxy/mix-bakeware.exs': No such file or directory
------
executor failed running [/bin/sh -c rm -rf /app/apps/massa_proxy/mix.exs && mv /app/apps/massa_proxy/mix-bakeware.exs /app/apps/massa_proxy/mix.exs]: exit code: 1
make: *** [build] Error 1
Change default strategy from epmd to Cluster.Strategy.Gossip like we have on Atreu project https://github.com/eigr/Astreu/blob/master/config/config.exs
Today by the Cloudstate protocol you can emit many side effects (a list of them) but the proxy handles each side effect call individually.
It would be interesting to allow the user role to emit a side effect for all entities registered in each of the Nodes of the cluster of a certain type. That is, to allow a call in Broadcast mode to be possible.
For that I see that we would have to change the SideEffect message to include a boolean flag to indicate this behavior. Something like:
// A side effect to be done after this command is handled.
message SideEffect {
// The name of the service to perform the side effect on.
string service_name = 1;
// The name of the command.
string command_name = 2;
// The payload of the command.
google.protobuf.Any payload = 3;
// Whether this side effect should be performed synchronously, ie, before the reply is eventually
// sent, or not.
bool synchronous = 4;
// Send side effect to all entities of the given type.
bool broadcast = 6;
// The metadata to include with the side effect
Metadata metadata = 5;
}
``
I'm registering the desire to change the protocol. Instead of using the Cloudstate protocol as a base, I propose to create our own protocol.
I will try in the next few days to make a draft and publish it as a draft so that everyone can comment and contribute.
I will keep everyone informed of my progress.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.