lightbend / kalix-jvm-sdk Goto Github PK

Reproducer https://github.com/octonato/counter-action-view

In the above reproducer, ref_id.proto is a standalone definition that is imported in both counter_api.proto and counter_action.proto.

The generate protobuf for RefId is com.example.RefIdProto, but in the generated code we reference it as Com.example.RefId.placeholder. Note as well the uppercase C at beginning.

For example, the generated action is:

public abstract class CounterActionInterface {
    @Handler
    public abstract Reply<Com.example.RefId.placeholder.RefId> newCounter(Empty event, ActionContext ctx);
}

but it should be

public abstract class CounterActionInterface {
    @Handler
    public abstract Reply<com.example.RefIdProto.RefId> newCounter(Empty event, ActionContext ctx);
}

Or in MainComponentRegistrations

registerValueEntity(
    CounterImpl.class,
    CounterApi.getDescriptor().findServiceByName("CounterService"),
    CounterDomain.getDescriptor(),
    Com.example.RefId.placeholder.getDescriptor(),
    EmptyProto.getDescriptor()
)

should be

registerValueEntity(
    CounterImpl.class,
    CounterApi.getDescriptor().findServiceByName("CounterService"),
    CounterDomain.getDescriptor(),
    com.example.RefIdProto.getDescriptor(),
    EmptyProto.getDescriptor()
)

One thing that calls attention is how different are the APIs for Value Entity, Event Sourced Entity, View, ReplicatedEntity.

This has been discussed in https://github.com/lightbend/akkaserverless/discussions/4306#discussioncomment-872700 and several other places so I will not repeat the motivation here.

Actions:

• Manage state for EventSourcedEntity instead of instance variables in user code
• Remove @Snapshot / @SnapshotHandler
• Access current state via method parameter instead of context
• Emit event and ValueEnitty state update via returned Effect instead of via context
• Remove context.failure since returning Effect.failure covers that

This also means that the current Reply type will be renamed to Effect, and it is also used for defining the reply, forward and failure.

From @ennru: Both with and without the google.api.http annotations.

For example

rpc AddItem(AddLineItem) returns (google.protobuf.Empty) {
    option (google.api.http) = {
        post: "/cart/{cart_id}/items/add"
        body: "*"
    };
}

From @raboof:
There is a 'Transcoding HTTP' section now, but I agree it would be good to flesh this out more.

Perhaps it could be moved to its own page, something like "Providing HTTP API's". There we would also explain that Akka Serverless is not intended to be used to implement arbitrary HTTP API's, but is limited to those parts of HTTP that can be described by transcoding.

For example:

AFAICS there is no way to influence the HTTP response status code
POST requests are supported. All messages that go to an entity must have an entity id, though - so also POST requests to an entity. If you want to create new entities, it might be better to use PUT, which has the additional advantage that it can be idempotent.

We've been toying with the idea of alternate editor for protobuf and Visual Studio Code projects (VSC). VSC has been targeted specifically as it also permits us to explore the ability to provide the same editing experience for web-based applications such as the Akka Serverless web UI.

Our goal continues here to be one of reducing the number of new things a developer must take on to become successful with Akka Serverless. Following on from leveraging the native tooling a developer should already be familiar with (npm/js == npx, Maven/Java == archetypes), we think that have an extension for VS Code focused sharply on protobuf editing in the context of Akka Serverless, should feel intuitive. Furthermore, the experience of this editor, while we can promote its usage and easy-installation within VSC, is opt-in. There will certainly be developers preferring to edit their own protobuf.

This issue has been designed to encourage feedback. To provoke that, here's a short video of what we've prototyped to-date:

https://drive.google.com/file/d/1Gicx-PqI5tMB2wP_Z_NAenZAOSyg-n34/view?usp=sharing

If there is sufficient interest and approval, we will then develop a solution design to capture this thinking and feedback.

When I added the initial effects I only added support for emitting one event. We should also have support for many. emitEvents taking a List or vararg parameter. Might not be possible to have overload with both List and and vararg so might be enough with List.

Would it be possible to make the codegen service do what the Maven archetype does so that users don't need to update their settings.xml to get a Maven starting point?
We want them to install akkasls anyway.

@mckeeh3 commented on Mon Jun 07 2021

In the Create unit tests section, the following sentence

the Akka Serverless Maven plugin creates a unit test stub for each class.

Capitalize "the". But the main thing in this sentence is "for each class" at the end. What classes? It would help to be more specific.

@rstento commented on Thu Jul 08 2021

"The" has been capitalized, but the question about what is meant by "class" remains.

When migrating the maven plugin and archetypes here, I keep it as maven project. But it requires some hacking to build it using the same version as the sbt project.

Lagom has a maven plugin and archetype that are build with sbt. I think we should do the same here.

Then we don't need to hack things around and when publish we only need to deal with sbt release. That will make our life easier.

...and use those in the docs to show the test kit and unit testing.

Right now, the abstract base class for the Event Sourced Entity gets the suffix Interface which is misleading.

The example with a single value being updated doesn't illustrate the principle of Event Sourcing well. We can either use the "counter" example as in the Value entity or come up with something else.

Add ReplicatedMultiMap to cover ReplicatedMap of ReplicatedSet values.

Note that in Akka (proxy impl) there is a specialized MultiMap, which isn't only ORMap of ORSet.

@rstento commented on Thu Jul 08 2021

@mckeeh3 commented on Mon Jun 07 2021

In The service proto file section of the docs.

This sentence:

The package specifies a namespace for this proto file and its imports—names must be unique within each namespace:

What is this namespace used for? This is confusing. How is this namespace used? Does it only apply to the proto files? Or is the namespace used in code as well?

WriteConsistency is a too low level concern in this context.

It should be WriteMajority by default.

Possibly it could be a configuration property.

they seem to be using the released sdk still

seen in https://app.circleci.com/pipelines/github/lightbend/akkaserverless-java-sdk/350/workflows/39cf07e0-c28b-43a4-9e72-189efcc15438/jobs/2008

Update: we decided to not offer it as an option, but pick juni5.

(keeping previous description for the record)

Our Java SDK testkit supports both JUnit 4 and JUnit 5.
The current codegen generates JUnit 4 code.

We'd possibly need a configuration to allow selecting the desired JUnit style (and support for it in codegen).

This should be documented for proto and the view impl.

For example, I have a view that only utilizes 2 of many events and the others are ignored. Anything that subscribes to an event sourced entity may need a CatchOthers.

Building this application serves two goals:

Understanding how the developer experience will be for Open Beta users and identifying any issues early in the process.

Validating early decisions made wrt to developer experience using Akka Serverless to build applications that require event publishing and consuming and viewing aggregate data.

@mckeeh3 commented on Mon Jun 07 2021

In the Writing gRPC descriptors section of the docs, the proto file naming convention is inconsistent with other parts of the documentation.

Use "API" and "domain" suffixes for proto file names. Also, use dash versus underscore in file names.

Examples

• shopping-cart-api.proto
• shopping-cart-domain.proto

We're reviewing and simplifying the APIs for replicated entities
https://github.com/lightbend/akkaserverless-framework/issues/698
to make them available for users during the open beta phase.

How can we support using Replicated Entities with codegen?

From @patriknw: Can't see that we describe how to publish to a topic in topic-eventing.adoc?

From @ennru: Correct, that is pending. My idea would be to add an external publish on cart checkout.
That's a good place to show entity id access in a listener.

From @viktorklang: @rstento I want to raise this as a very high priority for Open Beta. Not being able to publish messages makes the Eventing functionality very, very, limited.

In #15, we discussed how we could manage the publishing of samples.

I don't think we ever discussed a strategy for that. Hence this issue.

I guess what we need is an automated process to published the sample images each time we release a need SDK version. We want to offer the images in the console so user can install them and try it out. Eventually, we configure the console to point to LATEST, so users get the lastest as soon as we publish them.

lightbend/akkaserverless-framework#250 introduced the capability to listen to updates of Value Entities similar to an Event log of Event Sourced entities.

This should be illustrated for publishing to topics.

Since we are moving the maven plugin, the archetypes and the codegen-java under the java-sdk, we will be able to better glue the pieces and run several tests.

I will sketch here what I have in mind so others can comment and bring in other ideas.

We have few parts to integrate

• java-sdk (sbt project)
• codegen - depends on java-sdk (sbt project)
• plugin - depends on codegen
• archetypes - depends on all the above
• samples - depends on sdk and plugin (indirectly on codegen)

What do we want? Test all the things!

Plugin and Codegen

The plugin is now testing the codegen based on a simple descriptor. We generate code and then we verify if the files were emitted. The codegen itself has more tests and we should keep and improve as we go.

Samples

The samples are now tested against the current SDK (current as in the branch, not published artifacts) and uses a published codegen.

Once we move the codegen here, the samples test should run against the current codegen as well.

Archetypes

Archetypes are minimally tested. As of today, we only generated a new project from it an run mvn verify.

A better test would be to generate a new project, copy some implementation (domain + tests) and run the tests and integration tests. For example, on each new build, we could generate a new valueentity-counter, copy key files from samples/valueentity-counter and run the full tests.

That will confirm a two things: codegen generates the right pieces and archetype pom.xml is what needs to be.

Note, this is not the same as testing the samples. The idea of copying implementation classes from the samples it to force a start from scratch and be able to assess if all the new generated code plus the pom.xml from the archetype is correct.

Another set of tests, would be to build the user service container, deploy it with the proxy and run some integration tests.

@retgits commented on Thu May 27 2021

Many of the developers would come from a CRUD approach and it would be hard for them to get a grasp on Event Sourcing and the Java Classes related to it. Maybe we could do something to help them out? They could have questions like, why am I using akkaserverless.field? We have context like EntityContext, CommandContext, EffectContext etc but what exactly is this context?

Since the codegen is the selected approach we should remove the Java annotation based alternative and its reflection based implementation.

• Value Entities: Effect API (no implementation) #62
• Value Entities: Implementation of Effect API #91
• Value Entities: Update codegen to produce Effect-based interface, initial implementation and tests #120
• Value Entities: 'manual' prototype of 'handler' code to replace reflection-based implementation #73
• Value Entities: Update codegen to produce 'handler' code to replace reflection-based implementation #148 and #243
• Event Sourced Entities: Effect API (no implementation) #62
• Event Sourced Entities: Implementation of Effect API #91
• Event Sourced Entities: Update codegen to produce Effect-based interface, initial implementation and tests #132
• Event Sourced Entities: Update codegen to produce 'handler' code to replace reflection-based implementation #163 (potential additional cleanup in #150), actual codegen of handler in #231
• Views: Effect API (no implementation) #230
• Views: Implementation of Effect API #240
• Views: Update codegen to produce Effect-based interface, initial implementation and tests #240
• Views: Update codegen to produce 'handler' code to replace reflection-based implementation #240
• Actions: Effect API (no implementation) #258
• Actions: Implementation of Effect API #258
• Actions: Update codegen to produce Effect-based interface, initial implementation and tests #258
• Actions: Update codegen to produce 'handler' code to replace reflection-based implementation #258
• Protobuf option to support JSON API's that convert to method calls with Jackson-serialized objects
• Remove annotation-based implementation
• Re-enable all tests/samples/CI

Small changes required in pom.xml also user will need to install protoc locally in order to use the maven archetype.

This needs to be documented properly.

as per #42

The examples contain Maven builds so that they can serve as example projects outside of the Java SDK build.
The Maven builds should run in CI.

We can only build replies using a strict value or a forward. We should be able to buidl a reply from a CompletionStage.

This will be useful when calling another service or local components (see below) through gRPC or any other interface. This also adds the ability to transform the response or compose a response from several calls to other components/services.

@mckeeh3 commented on Wed Jun 09 2021

Need a gRPC API example in Implementing Event Sourced Entities in Java.

This is needed to help to relate the API definition to the event source entity implementation code examples.

By @mckeeh3
In Implementing Views for both JS and Java a full view proto file example is needed. The current examples do not include syntax, package, imports, and view specific option details.

Without these details, it is impossible to create views.

@mckeeh3 commented on Mon Jun 07 2021

This initial section in both the Implementing Event Sourced Entities in Java and Javascript needs to be reworked.

The third paragraph "When you need to read state in your service..." is confusing. The links are confusing.

The fourth paragraph, after the TODO, "To load an Entity...", implies that the developer is somehow responsible to load an entity.

The "Event Sourced Entities offer strong consistency guarantees..." paragraph mentions nodes. What nodes? I don't care about any nodes. This description should focus on how entity instances scale up and down - logically - without exposing too much detail about the inner workings. The takeaway from this section should be about the lifecycle of entities and how entity instances handle state.

By @mckeeh3

In Implementing Views in both javaScript and Java, change the three main subsections in the bullet list to links to their associated subsections. I know there are links on the right nav list, but things like these links are a nice touch when re-referencing specific documentation.

Event Sourced Entity events
Value Entities state changes
Messages received from subscribing to topics on a broker
Als0, order the bullets in the same order as the subsections (for us OCD folks).

We need to stop the build if the component reference from a service points to a non-existent component - same for other references too.

As discussed, we plan to merge into this project the maven plugin, the maven archetypes and the java codegen.

The execution plan is as follow:

Phase 1:

• move: bring maven-java under java-sdk (preserving commit history) #28
• build: must be integrated in CircleCI (similar to what we are doing to the samples already) #36
• versioning: we want the maven plugin and archetypes to have the same version of the java-sdk. #65
• publishing: archetypes and plugin must be published under a new name because of version misalignment #45 and #65
• docs: documentation must be adapted since users won't need to change anything in their settings.xml file and we will have a new artifact name #96
• github issues: move all open issues from maven-java to java-sdk

Phase 2:

First we need to split codegen into codegen-core, codegen-java and codegen-js (preserving commit history) See https://github.com/lightbend/akkaserverless-codegen/issues/147

• move codegen-java under java-sdk (preserving commit history)

Please, feel free to add here any other point of action that you believe we will need.

So that the java compiler will notice when a new rpc is added, or changed.

Currently it generates 3 methods for each command handler and the user has to choose which one to override. That is just confusing and missing implementation will only be detected at runtime instead of compile time.

Additionally, abstract methods are nice in an IDE because it's easy to click the compilation error and add the missing method.

We need a TestKit for running EventSourcedEntity and ValueEntity implementations in a unit test. The previous Mockito approach of mocking the context will not work for the effects, and it's anyway exposing to much internals to the user.

The goal is to feed the implementation with a command and verify the outcome. Much similar to the EventSourcedBehaviorTestKit in Akka.

The assertions should not be on the returned effect but on a test specific Result, which is populated with things like:

• emitted events
• optional reply
• optional errors
• optional forward

On the implementation side of this it should probably not take the same approach as EventSourcedBehaviorTestKit. EventSourcedBehaviorTestKit is running the EventSourcedBehavior as a real actor, because that's the only realistic way to make sure that is doing the real thing. In Akka Serverless we can have a synchronous test runner instead of the real EventSourcedEntitiesImpl. It is simple enough to have a second implementation for testing purposes.

One thing we can consider adding would be conversion methods to/from storage representation, so the business logic in command and event handlers can use a more convenient state representation than the proto objects (or json). That class would be somewhat difficult to define in the proto codegen options though, unless it's defined by it's fully qualified java class name.

Originally posted by @patriknw in #62 (comment)

Each component impl class includes a register method

  public static void register(AkkaServerless akkaServerless) {
    akkaServerless.registerValueEntity(
      CounterImpl.class,
      CounterApi.getDescriptor().findServiceByName("CounterService"),
      CounterDomain.getDescriptor()
    );
  }

In the Main.main( ... ) method

    final var service = new AkkaServerless();

    CounterImpl.register(service);
    MyActionImpl.register(service);
    ProductsByPostalViewImpl.register(service);
    CustomerOrdersViewImpl.register(service);

    service.start().toCompletableFuture().get();

We need to create internal documentation for developers who will maintain the code generation tooling.

Design documentation is referenced below and attached for download:

• https://docs.google.com/document/d/1YIGOYhXGNWKSRCMw9AjUtpm15XTKtVo_adRVUnEQIIo/
• https://docs.google.com/document/d/1TBB_uth8b_DcBtxCQ59V7S0t2qp9k3PA6xA66MxRL9c/

Internal documentation should describe, among other things, the logical arrangement of components, the role of each component and "what to change when..."

For example, what to change in case you want to add dependencies to the generated Java project.

https://app.zenhub.com/files/327973294/3966fff0-5f4b-4de7-a0d3-1cc7c00f86b9/download

https://app.zenhub.com/files/327973294/68b0ffea-bd3a-45b6-9351-53971118297e/download

@rstento commented on Fri Jul 09 2021

@mckeeh3 commented on Sun Jun 13 2021

In Implementing Views in both JavaScript and Java, move the Querying a View (JS, Java) subsection after all of three main subsections (Value Entity, Event Source Entity, Topics).

At the end of each of the three main subsections, add links to the "Querying a View" subsection. Something like "See Querying a View for more about how to select data."

At the end of each of the three main "Creating a View from a X" subsections, add a registering subsection specific to that View type.

We are starting to run mvn for the samples (see #8) and I notice that it's download all the artifacts each time.

That's strange, because CircleCI mention that it caches the artifacts. There must be something missing in the yml.

Various issues that were logged in the akkaserverless-docs repo:

By @mckeeh3
In Implementing Views, use a complete protobuf file example including the file names for the example value entity protobuf, event-sourced entity, and from a topic.The reason for this request is that there are references from the view protobuf files to the source protobuf files, i.e. domain.CustomerState, and it must be clear how the view to source names are determined. It is not clear why the reference domain.CustomerState has the domain. prefix.

By @mckeeh3
In the Creating Views section, there are examples of queries, but there are no examples of code implementations that reference queries. For example, define an API that performs a query on a view and returns a stream and another example that returns a single row. What seems to be missing is a section that explains how views are used.

By @mckeeh3
In Implementing Views in both JavaScript and Java, add a more detailed description about streams.

View queries that return streams contain the data available at the time the query is performed. We have already had questions about these streams from users (Jeremy). It was not clear what data is returned in these streams.

By @mckeeh3
In Creating a View from a Value Entity (JS, Java), change the first sentence:

Using an example of a customer registry, you can define a Customer Value Entity in Protobuf as:

to

Using an example of a customer Value Entity, it defines a domain Protobuf as:

Phrase the revised sentence as needed. The point is to make it clear that the view is using a domain Protobuf that has been defined for a value entity. The value entity and a view for the value entity share a common domain Protobuf.

By @mckeeh3
In Implementing Views for both JS and Java it should be clear that Update Transformation is possible for Value Entity views, Event Sourced Entity Views, and Views from topics.

The "Update transformation class" (Java, JS) subsection is included within the Creating a View from an Event Sourced Entity, which implies that transformation is not allowed with Value Entity Views and Topic Views.

There should b a single "Update transformation functions" subsection referenced from each of the three View from Value Entity, View from Event Sourced Entity, and Topic Views subsections.

lightbend/akkaserverless-framework#573 added support to connect a locally running proxy to a locally running Google Pub/Sub emulator.

This needs to be documented.

Marking as a bug because #573 was closed, but no documentation was provided.

Use the more specialized map implementations that are built on top of ReplicatedMap

• ReplicatedCounterMap
• ReplicatedRegisterMap

We should probably add ReplicatedMultiMap to cover ReplicatedMap of ReplicatedSet values.

• Reduce the number of different ReplicatedData types:
  • ReplicatedCounter instead of GCounter and PNCounter - it would be a PNCounter
  • ReplicatedSet instead of GSet and ORSet
  • Flag isn't needed, a Counter is enough to represent that
• Use less theoretical naming, such as ReplicatedCounter instead of PNCounter