Security Policy Outside Collaborators is out of compliance, status:
Found 1 outside collaborators with admin access.

Issue created by Allstar. https://github.com/ossf/allstar

I'd like to install Allstar https://github.com/ossf/allstar https://github.com/apps/allstar-app on this repo as a trail for eventually enabling on all envoyproxy org repos.

Allstar checks repos for violations against configured security policies, and takes actions when out of compliance:

Policies:

• Branch Protection settings
• SECURITY.md present
• No non-org Admins (outside collaborators)
• No binary artifacts.

Actions:

• Create a GitHub Issue
• Fix the issue (being developed)

Which policies to enable and which action to take are configured via config files in either an org-level repo named .allstar or files in the individual repo. This lets org owners control the main repo to manage settings.

I'll work with the org-owners to get it installed and configured with settings appropriate for the Envoy community.
cc @lizan @htuch @mattklein123

Relevant discussion: #35 (comment)

Ideally we would have access to the Node during the stream open and close callbacks. Due to the stream observer design, there are some tradeoffs in order to support that.

For onStreamOpen, currently we're triggering that callback outside of the request StreamObserver. At that point, we haven't actually received a DiscoveryRequest yet, so we haven't been given the Node yet. A potential workaround would be to delay the onStreamOpen event until we receive the first request message. In theory there could be an arbitrarily large gap between when the stream is actually opened and when the first request is sent and with this approach we lose the ability to measure that. In practice, is that something that we really care about for this use case? With the ADS/xDS protocol design, the first message is always sent from the client and it occurs immediately after the stream is opened.

For onStreamClose and onStreamCloseWithError, we can just cache the Node from the last request message. Consumers will just need to be aware that the node parameter in this scenario is @Nullable because the stream could close without ever receiving a request.

Previous discussion: #105 (comment).

DiscoveryRequestStreamObserver#onNext may receive messages after the responseObserver is closed (e.g. via responseObserver#onError). Calling responseObserver#onError twice causes an error, so we need some way to guard against calling it again.

Potential solution: check isClosing at the beginning of each onNext call. If true, log an error and return immediately.

afaict the last update with the data-plane-api was #63, Jul 17.

Can we bring them up to date? I'm specifically interested in the work done in Envoy issue #3555 which was addressed in Envoy PR #4015.

(basing this on #2)

What are people's thoughts on migrating this repo to use bazel instead of maven? Not only would it match the build tooling that envoy uses, but it would also simplify the problem of importing the necessary protos, as we'd be able to depend on data-plane-api instead of copying all the protos into this repo. Bazel also has fairly good native support for Java and the IJ plugin works well for small repos.

Thoughts? I'd be happy to put in the work to move it over (I have some experience migrating java repos to bazel), but I'd like to know where we stand on this before starting that work.

Please see:
https://github.com/envoyproxy/java-control-plane/blob/master/api/src/main/proto/envoy/service/ratelimit/v3/rls.proto#L95
whereas, in the envoy data plane api, the rls.proto defines:
https://github.com/envoyproxy/data-plane-api/blob/master/envoy/service/ratelimit/v3/rls.proto#L117

Is this a bug? or was this value intentionally left out?

With Envoy v1.13.0 release v2 API became deprecated and we need to move on to v3.
Is there any schedule on this?

I tried to set up java-control-plane on my local machine;
Cloned and $ mvn clean package

The Build is successful but IntelliJ fails to recognize
import io.envoyproxy.envoy.api.v2.DiscoveryResponse;

Am I missing anything?

[INFO] Reactor Summary:
[INFO]
[INFO] java-control-plane ................................. SUCCESS [ 4.516 s]
[INFO] api ................................................ SUCCESS [04:44 min]
[INFO] cache .............................................. SUCCESS [ 14.967 s]
[INFO] server ............................................. SUCCESS [ 19.204 s]
[INFO] test ............................................... SUCCESS [ 0.504 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

The CheckResponse builder does not provide any setHttpResponse() methods.

As per the 1.8.0 envoy release, Secret Discovery Service has been added.

DiscoveryServer.java does not have this implemented. Can you please add SDS to the DiscoveryServer implementation?

Hi i use v 0.1.28 of java control plane with slight modifications ( i load my routes from db), and i changed the code such that it uses all the V3 objects ( and not v2 backeard compatible fileds bare being used to best of my knowledge).

the java control plane is able to talk to data plane,using v3 apis, and i see http filters, listener, config etc being loaded correctly,
however i see that Route info is missing in my envoy config , and i get the following erro in enovy proxy log

DATA PLANE LOGS /ENVOY PROXY LOGS

[2021-05-25 08:36:15.379][10][warning][config] [source/common/config/grpc_subscription_impl.cc:127] gRPC config for type.googleapis.com/envoy.api.v2.RouteConfiguration rejected: The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021. You may make use of v2 in Q4 2020 by following the advice in https://www.envoyproxy.io/docs/envoy/latest/faq/api/transition. (envoy.api.v2.RouteConfiguration) [2021-05-25 08:36:15.379][10][debug][config] [source/common/config/grpc_mux_impl.cc:129] Resuming discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration (previous count 1) [2021-05-25 08:36:15.382][10][debug][config] [source/common/config/grpc_mux_impl.cc:158] Received gRPC message for type.googleapis.com/envoy.api.v2.RouteConfiguration at version 1621931747625 [2021-05-25 08:36:15.382][10][debug][config] [source/common/config/grpc_mux_impl.cc:122] Pausing discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration (previous count 0)

java control plane logs
[2021-05-25 08:36:15.379][10][warning][config] [source/common/config/grpc_subscription_impl.cc:127] gRPC config for type.googleapis.com/envoy.api.v2.RouteConfiguration rejected: The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021. You may make use of v2 in Q4 2020 by following the advice in https://www.envoyproxy.io/docs/envoy/latest/faq/api/transition. (envoy.api.v2.RouteConfiguration) [2021-05-25 08:36:15.379][10][debug][config] [source/common/config/grpc_mux_impl.cc:129] Resuming discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration (previous count 1) [2021-05-25 08:36:15.382][10][debug][config] [source/common/config/grpc_mux_impl.cc:158] Received gRPC message for type.googleapis.com/envoy.api.v2.RouteConfiguration at version 1621931747625 [2021-05-25 08:36:15.382][10][debug][config] [source/common/config/grpc_mux_impl.cc:122] Pausing discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration (previous count 0)

I am not sure why java contorl plane use v2 routeconfiguration when v3 object is there in the code.

My GRPC daemon look like
GRPCDamemon.java

ServerBuilder builder = NettyServerBuilder.forPort(config.getGrpcPort()) // removed all v2 stuff .addService(discoveryServer.getAggregatedDiscoveryServiceImpl()) // removed all v2 stuff .addService(discoveryServer.getClusterDiscoveryServiceImpl()) // removed all v2 stuff .addService(discoveryServer.getEndpointDiscoveryServiceImpl()) // removed all v2 stuff .addService(discoveryServer.getListenerDiscoveryServiceImpl()) // removed all v2 stuff .addService(discoveryServer.getRouteDiscoveryServiceImpl()) .addService(v3DiscoveryServer.getAggregatedDiscoveryServiceImpl()) .addService(v3DiscoveryServer.getClusterDiscoveryServiceImpl()) .addService(v3DiscoveryServer.getEndpointDiscoveryServiceImpl()) .addService(v3DiscoveryServer.getListenerDiscoveryServiceImpl()) .addService(v3DiscoveryServer.getRouteDiscoveryServiceImpl()); server = builder.build(); server.start();

I create my Simple as as follow

cache = new SimpleCache<>(new NodeGroup<String>() { @Override public String hash(Node node) { throw new IllegalStateException("not supported/disabled"); } @Override public String hash(io.envoyproxy.envoy.config.core.v3.Node node) { return routerConfiguration.getGroupName(); } });
and this is how initialize my cache manger

ConfigSource rdsSource = ConfigSource.newBuilder().setAds(AggregatedConfigSource.getDefaultInstance()) .setApiConfigSource(ApiConfigSource.newBuilder().setTransportApiVersion(ApiVersion.V3) .setApiType(ApiType.GRPC) .addGrpcServices(GrpcService.newBuilder().setEnvoyGrpc( EnvoyGrpc.newBuilder().setClusterName(routerConfiguration.getXdsClusterName())))) .build()

Not sure if iam missing something, this seems to be a bug to me.

Sometimes we need to validate DiscoveryRequest (mostly node metadata). Currently we can do it in DiscoveryServerCallbacks.onStreamRequest, but there is no simple way to return custom error response. Throwing exception in that method will result in returning status UNKNOWN without any description.

It would be nice to return custom status with description what actually is wrong with the request. This way in Envoy logs we will see that description instead of enigmatic line: gRPC config stream closed: 2,.

I prepared a PR with implementation.

Reactive gRPC makes gRPC reactive and easy to implement logic. Anybody agrees with this feature? It's easy, and just add reactive-grpc protocPlugin in api module. Details link: https://github.com/salesforce/reactive-grpc

When I set the snapshot with the instance in locality dc1 with priority 0 and then set another one with instance in locality dc1 with priority 1, but without the first one, there are two instances in Envoy.

I've written following test
https://github.com/jakubdyszkiewicz/java-control-plane/commit/39b76bc71fe48ab8f727d66714ef3f3f427dc039

But then it turned out that the problem seems to be even more general.
When I set a snapshot with endpoint with priority 0, then set another one with priority 1, but without the first one - both stays in Envoy.
https://github.com/jakubdyszkiewicz/java-control-plane/blob/a2a6b4b5498fa1de734ed76aafcdf768067f525a/server/src/test/java/io/envoyproxy/controlplane/server/MoveInstanceToAnotherLocalityTest.java

Not sure whether this is an issue with java-control-plane or Envoy itself or this is just an expected behaviour.

I've seen issues in Envoy that may be connected with this
envoyproxy/envoy#3327
envoyproxy/envoy#3312

@snowp it would be helpful if you could take a look at this, because you fixed above issues. Thank you.

•   * TCP Proxy filter configuration using deprecated V1 format. This is required for complex
    * routing until filter chain matching in the listener is implemented.
    * .. attention::
    *   Using this field will lead to `problems loading the configuration
    *   <https://github.com/envoyproxy/envoy/issues/2441>`_. If you want to configure the filter
    *   using v1 config structure, please make this field a boolean with value ``true`` and configure
    *   via the opaque ``value`` field like is suggested in :api:`envoy/config/filter/README.md`.
    * 

  • .envoy.config.filter.network.tcp_proxy.v2.TcpProxy.DeprecatedV1 deprecated_v1 = 6 [deprecated = true];
    */
    @java.lang.Deprecated public envoy.config.filter.network.tcp_proxy.v2.TcpProxyOuterClass.TcpProxy.DeprecatedV1OrBuilder getDeprecatedV1OrBuilder() {
    return getDeprecatedV1();
    }

so the following code can't work ?

envoy.config.filter.network.tcp_proxy.v2.TcpProxyOuterClass.TcpProxy.Builder tcpProxyBuilder=envoy.config.filter.network.tcp_proxy.v2.TcpProxyOuterClass.TcpProxy.newBuilder();
envoy.config.filter.network.tcp_proxy.v2.TcpProxyOuterClass.TcpProxy.DeprecatedV1.Builder routeBuiler=tcpProxyBuilder.setStatPrefix("ingress_tcp").getDeprecatedV1Builder();
routeBuiler.addRoutes(value)

• README
• CLA
• CONTRIBUTING.md
• CircleCI

Feel free to copy stuff from the main repo.

Relevant discussion: #33 (comment)

Ideally the cache would transparently handle any differences that arise from running in ADS vs. xDS mode, without needing the consumer to create separate caches.

I've been thinking about how to improve the performance of our control plane, and I've realized that with EDS in particular, the control plane does a lot of work generating ClusterLoadAssignments that could theoretically be shared for many different Nodes.

If 3 different microservices depend on the "foo" cluster for example, we can likely generate just one ClusterLoadAssignment for "foo" one time and share it for all.

Right now we're using ADS for everything, I'm thinking about using ADS just for Listeners, Routes, and Clusters, and having a separate EDS only control plane that can create a NodeGroup based on the resource names (from DiscoveryRequest) for the ClusterLoadAssignment in addition to the Node.

What do y'all think about me adding that capability?

Stated another way, instead of only providing

public interface NodeGroup<T> {
  T hash(Node var1);
}

We could also allow implementing

public interface NodeAndResourceNamesGroup<T> {
  T hash(Node var1, Set<String> resourceNames);
}

Our experience building a control plane out of java-control-plane has shown that it's easy to overwhelm the service if there's many Envoys querying the control plane for many services.

Especially in the bootstrap phase, when deploying multiple Envoys, they won't be able to serve as a proxy if they don't get the initial configuration. Killing the control plane leads to even worse consequences, where the cluster cannot be stabilised at all.
We dealt with the issues by whitelisting services and using ADS to limit the number of connections.

However, in case of some error on our side (or Envoy's), we might end up with far too many streams/requests being issued to the control plane than an instance can handle.

I'm suggesting adding three things:

• rate limiting of concurrent requests handled by java-control-plane
• limiting number of streams that can be open at the same time
• utilising HTTP2's flow control by manually controlling transmission

The goal of the above is simply to make the SPoF component (which java-control-plane might end up being) resilient in spite of unexpected rise of demand. This also prevents cascading failures, where a misconfiguration in part of the infrastructure could kill the entire cluster.

I'm happy to submit a PR with these changes if you think this is something worth introducing here. Otherwise, we'd need to keep a fork/copy of DiscoveryServer.

Hello,

it looks like java-control-plane fell behind envoy api updates.

I have a need of some v1.14 features, so I updated protos to v1.14.4. It was quite straightforward, but it means I have to maintain my own build of the library. I can make a PR with those changes.

Update to v1.15.0 (still supporting v2) will be more involving as there are much more changes to address in repository versions; I could look into that too if there was a demand.

Are there any plans on updating protos with each envoy release?

Hello!
I try to build the project in intellij but it seems that it cant find the package envoy.api. in general and this creates a lot o errors. Do you know anything about that??
Thank you in advance

When config update fails at envoy, currently we are just logging (as per #14) the error and sending the same response again. It is likely to fail again immediately based on kind of error we get. Ideally based on error we should be able to decide whether to send the request again or not. However, we should at least implement some backoff strategy or rate limiting these failed responses.

Hi guys, I'm using ControlPlane to build complex Envoy configurations with thousands of routes & clusters. I've discovered that after SimpleCache.setSnapshot call huge configuration remains in the memory (100MB or even more) as the following maps are keeping config & statuses:

private final Map<T, Snapshot> snapshots = new HashMap<>();
private final Map<T, CacheStatusInfo<T>> statuses = new HashMap<>();

Is there a way to publish Envoy configuration without SimpleCache? I would like to build snapshot and push it to Envoy config dump and do not leave any 'big' traces in RAM.

Hello,

At Allegro, we are building our control plane on top of java-control-plane. We integrated it with Consul Discovery Service which on our dev environment has about 800 unique services and 2000 service instances. In our tests we observed issues with performance of the provided implementation.

A single instance of our control plane with 2GB of memory and 2 cores is struggling to handle 100 connected Envoys.
We’ve diagnosed that the main problem is Snapshot versioning. Right now, there is one version for all endpoints. When even one service instance is changed (added/removed) the version of all endpoints is changed and all endpoints have to be sent to all envoys. This results in a huge spike in a gRPC executor in tens of thousands of jobs as well as creating a huge number of objects on the heap. Consequently, the GC runs so often that application can be stuck in GC for even 40% of the time.

We’ve introduced a PoC change in java-control-plane that lets us version every cluster in EDS individually. That means that if one instance is changed, we only sent the state of one cluster. After this change, our control plane can easily handle those 100 Envoys with the same machine with ~10% CPU usage.

An initial proposal for the change can be found at:
https://github.com/jakubdyszkiewicz/java-control-plane/commit/e96f23afe4cb45698a53b58d394e4d8a524310c0

Can we prepare a proper PR from it? If you have suggestions how to solve it in a better way please let us know.
We’d appreciate sharing your experiences with java-control-plane’s scalability.

Are there any plans to add an implementation of the delta xDS? I see that there's an open PR in go-control-plane to implement the delta xDS.

Currently java-control-plane uses an EmitterProcessor to notify subscribers that a watch has changed. Since this executes the the subscription callbacks synchronously, we end up spending a substantial time constructing the gRPC responses while holding the SimpleCache write lock during setSnapshot, essentially making the cache inaccessible until all the watches have been triggered.

It would be nice to use an async processor instead (maybe WorkQueueProcessor?) which would allow us to decouple the response construction from watch triggers, reducing overall lock contention.

Spent a bit of time figuring this out but ran into some deadlocks and issues in test, so filing this for now to track the the issue in case anyone else runs into this.

Setting a new snapshot that removes a cluster fails with the following Exception:

controlplane_1  | 2018-11-02 19:29:37 ERROR c.m.fury.controlplane.xds.XdsCache - xds cache rebuild failed
controlplane_1  | io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
controlplane_1  |  at io.grpc.Status.asRuntimeException(Status.java:517)
controlplane_1  |  at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:335)
controlplane_1  |  at io.envoyproxy.controlplane.server.DiscoveryServer$7.send(DiscoveryServer.java:270)
controlplane_1  |  at io.envoyproxy.controlplane.server.DiscoveryServer$7.lambda$null$1(DiscoveryServer.java:214)
controlplane_1  |  at io.envoyproxy.controlplane.cache.Watch.respond(Watch.java:77)
controlplane_1  |  at io.envoyproxy.controlplane.cache.SimpleCache.respond(SimpleCache.java:283)
controlplane_1  |  at io.envoyproxy.controlplane.cache.SimpleCache.lambda$setSnapshot$2(SimpleCache.java:209)
controlplane_1  |  at io.envoyproxy.controlplane.cache.CacheStatusInfo.lambda$watchesRemoveIf$0(CacheStatusInfo.java:142)
controlplane_1  |  at java.base/java.util.Collection.removeIf(Collection.java:492)
controlplane_1  |  at io.envoyproxy.controlplane.cache.CacheStatusInfo.watchesRemoveIf(CacheStatusInfo.java:142)
controlplane_1  |  at io.envoyproxy.controlplane.cache.SimpleCache.setSnapshot(SimpleCache.java:200)
controlplane_1  |  at myapp.controlplane.xds.XdsCache.rebuild(XdsCache.java:125)
controlplane_1  |  at myapp.controlplane.xds.XdsCache.lambda$new$1(XdsCache.java:57)
controlplane_1  |  at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
controlplane_1  |  at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
controlplane_1  |  at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
controlplane_1  |  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
controlplane_1  |  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
controlplane_1  |  at java.base/java.lang.Thread.run(Thread.java:844)

It would seem that when removing a cluster, envoy cancels the EDS stream asociated with that cluster, but the control plane still tries to synchronously respond to that stream and fails because it has already been cancelled.
I would expect this scenario to be handled better, at least not fail the setSnapshot as this prevents further responses to be sent.

It possible that RDS might be sent before LDS. When there is many changes in state of application it might happens that RDS watch will be answeared in createWatch with new version.

E.g.:

1. Envoy connects to control-plane
2. Snapshot already exists in control-plane <- other instance share same group
3. Control-plane respond with LDS in createWatch method
4. There is snapshot update which change LDS and RDS versions
5. Envoy sends RDS request
6. Control-plane respond with RDS in createWatch method
7. Envoy resume LDS and RDS requests.
8. Envoy sends request LDS
9. Control plane respond with LDS in createWatch method (new version)
10. Envoy sends RDS requests
11. Control plane doesn't respond because version hasn't changed
12. Listeners of a service stays in warming phase

Wanted to use the reference implementation to adapt eureka service discovery to envoy cluster configuration.

When I clone the repo and do a plain maven build on the test module to try the tests, maven fails to compile the generated java files from protoc-plugin.

Using IntelliJ 2018.3 community with Java11 on mac. Also tried setting gRPC version to LATEST/RELEASE and Java8. Always the same outcome.

[INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project api: Compilation failure: Compilation failure: [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/rbac/v2alpha/RBAC.java:[6,14] class Rbac is public, should be declared in a file named Rbac.java [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/network/client_ssl_auth/v2/ClientSSLAuth.java:[6,14] class ClientSslAuth is public, should be declared in a file named ClientSslAuth.java [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/accesslog/v2/AccessLog.java:[6,14] class Accesslog is public, should be declared in a file named Accesslog.java [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/http/ip_tagging/v2/IPTagging.java:[6,14] class IpTagging is public, should be declared in a file named IpTagging.java [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/network/rbac/v2/RBAC.java:[6,14] class Rbac is public, should be declared in a file named Rbac.java [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/http/rbac/v2/RBAC.java:[6,14] class Rbac is public, should be declared in a file named Rbac.java [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/network/rbac/v2/RBACOrBuilder.java:[27,42] cannot access io.envoyproxy.envoy.config.rbac.v2alpha.RBAC [ERROR] bad source file: /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/rbac/v2alpha/RBAC.java [ERROR] file does not contain class io.envoyproxy.envoy.config.rbac.v2alpha.RBAC [ERROR] Please remove or make sure it appears in the correct subdirectory of the sourcepath. [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/network/rbac/v2/RBACOrBuilder.java:[110,52] cannot access io.envoyproxy.envoy.config.filter.network.rbac.v2.RBAC [ERROR] bad source file: /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/network/rbac/v2/RBAC.java [ERROR] file does not contain class io.envoyproxy.envoy.config.filter.network.rbac.v2.RBAC [ERROR] Please remove or make sure it appears in the correct subdirectory of the sourcepath. [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/network/tcp_proxy/v2/TcpProxy.java:[5304,72] cannot access io.envoyproxy.envoy.config.filter.accesslog.v2.AccessLog [ERROR] bad source file: /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/accesslog/v2/AccessLog.java [ERROR] file does not contain class io.envoyproxy.envoy.config.filter.accesslog.v2.AccessLog [ERROR] Please remove or make sure it appears in the correct subdirectory of the sourcepath. [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/http/rbac/v2/RBACPerRoute.java:[90,57] cannot access io.envoyproxy.envoy.config.filter.http.rbac.v2.RBAC [ERROR] bad source file: /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/http/rbac/v2/RBAC.java [ERROR] file does not contain class io.envoyproxy.envoy.config.filter.http.rbac.v2.RBAC [ERROR] Please remove or make sure it appears in the correct subdirectory of the sourcepath. [ERROR] /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/http/ip_tagging/v2/IPTaggingOrBuilder.java:[25,55] cannot access io.envoyproxy.envoy.config.filter.http.ip_tagging.v2.IPTagging [ERROR] bad source file: /Users/maikmewes/IdeaProjects/java-control-plane/api/target/generated-sources/protobuf/java/io/envoyproxy/envoy/config/filter/http/ip_tagging/v2/IPTagging.java [ERROR] file does not contain class io.envoyproxy.envoy.config.filter.http.ip_tagging.v2.IPTagging [ERROR] Please remove or make sure it appears in the correct subdirectory of the sourcepath. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
May you have advise on what I'm missing?

I've been having a problem with my project which is based on java-control-plane. After running my server for some time, it would stop updating the envoy configuration. The envoy logs didn't report any problems, and the stats showed an open attempt at updating the resources. So I looked at java-control-plane and I seem to have found a problem here with nonces and watches.

I'm using envoy v1.8.0 and java-control-plane v0.1.9 with ADS.

I added logs (Can only reproduce in production) to DiscoveryServer in onNext(DiscoveryRequest) (here) and on get/put for the latestResponse map. Below are the logs I gathered while the problem occurs:

Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.cache.SimpleCache lambda$setSnapshot$2
INFO: responding to open watch 2[my-cluster] with new version 15389724330002
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.cache.SimpleCache respond
INFO: responding for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment from node MyNode at version 15389724270001 with version 15389724330002
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 send
INFO: [0] response type.googleapis.com/envoy.api.v2.ClusterLoadAssignment with nonce 4 version 15389724330002
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [0] request type.googleapis.com/envoy.api.v2.ClusterLoadAssignment[my-cluster] with nonce 4 from version 15389724330002
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [latestResponse get] type: type.googleapis.com/envoy.api.v2.Cluster, nonce: 0
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [latestResponse get] type: type.googleapis.com/envoy.api.v2.ClusterLoadAssignment, nonce: 1
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [0] requestOnNext type.googleapis.com/envoy.api.v2.ClusterLoadAssignment nonce mismatch; latestResponseNonce:1 requestNonce:4
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [latestResponse get] type: type.googleapis.com/envoy.api.v2.Listener, nonce: 2
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [latestResponse get] type: type.googleapis.com/envoy.api.v2.RouteConfiguration, nonce: 3
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 lambda$null$0
INFO: [latestResponse put] type: type.googleapis.com/envoy.api.v2.ClusterLoadAssignment, nonce: 4
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.cache.SimpleCache lambda$setSnapshot$2
INFO: responding to open watch 4[my-routes] with new version 15389724330002
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.cache.SimpleCache respond
INFO: responding for type.googleapis.com/envoy.api.v2.RouteConfiguration from node MyNode at version 15389724270001 with version 15389724330002
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 send
INFO: [0] response type.googleapis.com/envoy.api.v2.RouteConfiguration with nonce 5 version 15389724330002
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [0] request type.googleapis.com/envoy.api.v2.RouteConfiguration[my-routes] with nonce 5 from version 15389724330002
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [latestResponse get] type: type.googleapis.com/envoy.api.v2.Cluster, nonce: 0
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [latestResponse get] type: type.googleapis.com/envoy.api.v2.ClusterLoadAssignment, nonce: 4
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [latestResponse get] type: type.googleapis.com/envoy.api.v2.Listener, nonce: 2
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [latestResponse get] type: type.googleapis.com/envoy.api.v2.RouteConfiguration, nonce: 3
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 onNext
INFO: [0] requestOnNext type.googleapis.com/envoy.api.v2.RouteConfiguration nonce mismatch; latestResponseNonce:3 requestNonce:5
Oct 08, 2018 4:20:33 AM io.envoyproxy.controlplane.server.DiscoveryServer$7 lambda$null$0
INFO: [latestResponse put] type: type.googleapis.com/envoy.api.v2.RouteConfiguration, nonce: 5

After the nonce mismatch occurs (when this check fails for all types), my service no longer updates envoy's config for that resource. I'm not really familiar enough with the protocol here, so I would appreciate it if anyone could take a look.

In order to be able to monitor the config changes (and alert on updates stuck due to NACK loops) it'd be nice to have some hooks in DiscoveryServer to let us observe the result. Something generic like a callback that just receives the request object whenever one is sent would be sufficient for our use cases, but I'm open to other options.

@jyotimahapatra I've switched Envoy from 1.11.1 to 1.14.1 and discovered that control plane's DiscoveryServerCallbacks.onStreamRequest is no longer triggered on Envoy app start. It used to workin in Envoy 1.11.1 and was triggered every time Envoy started. I've checked Envoy config_dump and found that data class & data structure has changed (see below). Is it there a way to fix it? I'm using most recent control plane 0.1.23

Envoy 1.11.1 config dump

"configs": [
    {
      "@type": "type.googleapis.com/envoy.admin.v2alpha.BootstrapConfigDump",
      "bootstrap": {
        "node": {
          "id": "0c993f87e006",
          "cluster": "aaa-envoy",
          "metadata": {
            "host": "0c993f87e006",
            "admin_port": 10000,
            "port": 11000,
            "instance": "0c993f87e006aaa-envoy"
          },
          "build_version": "7a12f379e47a9f7cf7211c727fe8fc70b6a0a9ab/1.11.2/Clean/RELEASE/BoringSSL"
        }...

Envoy 1.14.1 config dump

"configs": [
    {
      "@type": "type.googleapis.com/envoy.admin.v3.BootstrapConfigDump",
      "bootstrap": {
        "node": {
          "id": "f108b1f075bd",
          "cluster": "aaa-envoy",
          "metadata": {
            "instance": "f108b1f075bdaaa-envoy",
            "port": 10000,
            "admin_port": 11000,
            "host": "f108b1f075bd"
          },
          "hidden_envoy_deprecated_build_version": "3504d40f752eb5c20bc2883053547717bcb92fd8/1.14.1/Clean/RELEASE/BoringSSL",
          "user_agent_name": "envoy",
          "user_agent_build_version": {
            "version": {
              "major_number": 1,
              "minor_number": 14,
              "patch": 1
            },
            "metadata": {
              "build.type": "RELEASE",
              "ssl.version": "BoringSSL",
              "revision.status": "Clean",
              "revision.sha": "3504d40f752eb5c20bc2883053547717bcb92fd8"
            }
          },

Listener

@Component
class ServiceDiscoveryCallback(
...
) : DiscoveryServerCallbacks {


    override fun onStreamRequest(streamId: Long, request: DiscoveryRequest) {
        logger.info("streamId: $streamId")
...
        }
    }

Xds api v2 has been "softly" deprecated in latest Envoy release. Our integration tests fail with errors like:

gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021. You may make use of v2 in Q3 2020 by setting '--bootstrap-version 2' on the CLI for a v2 bootstrap file and also enabling the runtime envoy.reloadable_features.enable_deprecated_v2_api flag. (envoy.api.v2.Listener)
[tc-okhttp-stream-439430225] INFO 🐳 [envoyproxy/envoy-alpine-dev:latest] - STDERR: 

Short term fix would be to add -bootstrap-version 2 to Envoy container used in tests, long term to get rid of all v2 tests/code. This blocks all incoming PRs.

There are a lot of upcoming changes in envoyproxy/go-control-plane#45. We should track those changes and consider porting them to java-control-plane.

Hi i trield downloading java control plane
rand the proto updates
compiled the project using mvn clean package and was able to compile .
However, when i try to import and run this in eclipse, i see many compilation issues and not able to run the TestMain.

Especailly i see issues with "AutoValue_LatestDiscoveryResponse" and many "Auto_XXx" classes.
not sure what i am missing.

Currently there is limited count of resource types in xDSv2/xDSv3, and should be enough for common scenarios.

But if we have some customized resource types, e.g. private module in envoy which relies on specific configs, then resource type extension support is needed.

Like that the pattern UDPA suggests: separate transport(UDPA-TP) and data-models(UDPA-DM), and it should be easy to extend data models.

Hello! I'm trying to incorporate this package in my project but I'm having some issues building my project because of unshaded dependencies. My project has some of the same dependencies in this project, such as Opencensus, which causes build issues since both classes end up in the classpath. Would you be able to shade the dependencies or what would you suggest I do to fix this? Thanks!

As multiple threads/watchers can send updates on the same stream for ADS and EDS, need to synchronize responseObserver in send method of DiscoveryService

After pulling the source code and executing:

mvn clean package 

In the test class TestMain the following files fail to import:

import io.envoyproxy.envoy.api.v2.Cluster;
import io.envoyproxy.envoy.api.v2.Cluster.DiscoveryType;
import io.envoyproxy.envoy.api.v2.core.Address;
import io.envoyproxy.envoy.api.v2.core.SocketAddress;

Tested this on Windows and Ubuntu. The same end result.

Security Policy SECURITY.md is out of compliance, status:
SECURITY.md not found.
Go to https://github.com/envoyproxy/java-control-plane/security/policy to enable.

Issue created by Allstar. https://github.com/ossf/allstar

DiscoveryServer is logging ERROR when client Envoy closes connection.

I have an Envoy connected to my java-control-plane implementation and whenever I restart Envoy, the control plane logs an ERROR from io.envoyproxy.controlplane.server.DiscoveryServer

Strange thing is that the code here: https://github.com/envoyproxy/java-control-plane/blob/master/server/src/main/java/io/envoyproxy/controlplane/server/DiscoveryServer.java#L224-L226
says that it should log ERROR if the status is other than CANCELLED, however in my case, the stack trace says:

io.grpc.StatusRuntimeException: CANCELLED: cancelled before receiving half close
	at io.grpc.Status.asRuntimeException(Status.java:517)
	at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:272)
	at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:293)
	at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:741)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

and the message is [3] stream closed with error together with logger_name io.envoyproxy.controlplane.server.DiscoveryServer, indicating that the log line must be coming from the Discovery server there.

(Basing this on #2)

Noticed quite a bit of redundant calls to .build() when making non-trivial proto messages: protoc generates an overload that accepts a Message.Builder instead of a Message when setting fields on a proto builder. This allows you to omit the build() when passing a proto to a setter.

I noticed this in server/src/test/java/io/envoyproxy/controlplane/server/Resources.java, but there might be other occurrences.

This tracks cleaning that up, as it just adds noise to the code.

Hello,
I have a problem with java-control-plane regarding removal of route configurations.
Everything works well until a route configuration (together with all other objects: endpoints, clusters and listener) is removed from snapshot. Java control plane refuses to respond to envoy (not responding in ADS mode...) because envoy-requested route is not present in snapshot. From now on further updates on RDS channel do not work, even when missing route is reinstantiated. Setting new route snapshot version is ignored by data plane. To resolve this, one needs a restart of a data plane (or envoy).

I do not know how to handle such situation. How to remove listeners/route configurations/clusters/endpoints gracefully having no feedback from the library that update has been silently dropped? How to remove something envoy requests?

Quick look into the SimpleCache's implementation shows that for update to be sent to envoy its version must match requested version. Would it be bad to loosen this condition and allow also newer versions? I'm not an expert in envoy's XDS protocol, so I do not know if it would be correct.

java-control-plane version: 0.1.19

Relevant logs:

2019-12-09 14:47:24.058 DEBUG 13917 --- [   scheduling-1] p.w.s.dominator.envoy.SnapshotPublisher  : Routes changed
2019-12-09 14:47:24.060 DEBUG 13917 --- [   scheduling-1] p.w.s.d.envoy.EnvoyDiscoveryServer       : Routes names in version 1575899241-1-1: auto-virtual-hosts, diagnostics, custom-50055-http, custom-10011-http, custom-0.0.0.0:14503-http, custom-10021-http
2019-12-09 14:48:00.438 DEBUG 13917 --- [ault-executor-0] i.e.controlplane.server.DiscoveryServer  : [0] request type.googleapis.com/envoy.api.v2.RouteConfiguration[custom-10011-http, custom-10021-http, custom-50055-http, custom-0.0.0.0:14503-http, auto-virtual-hosts, diagnostics] with nonce  from version
2019-12-09 14:48:00.439 DEBUG 13917 --- [ault-executor-0] i.e.controlplane.cache.SimpleCache       : responding for type.googleapis.com/envoy.api.v2.RouteConfiguration from node default at version  with version 1575899241-1-1
2019-12-09 14:48:00.441 DEBUG 13917 --- [ault-executor-0] i.e.controlplane.server.DiscoveryServer  : [0] response type.googleapis.com/envoy.api.v2.RouteConfiguration with nonce 3 version 1575899241-1-1
2019-12-09 14:48:00.445 DEBUG 13917 --- [ault-executor-0] i.e.controlplane.server.DiscoveryServer  : [0] request type.googleapis.com/envoy.api.v2.RouteConfiguration[custom-10011-http, custom-10021-http, custom-50055-http, custom-0.0.0.0:14503-http, auto-virtual-hosts, diagnostics] with nonce 3 from version 1575899241-1-1
2019-12-09 14:48:00.447 DEBUG 13917 --- [ault-executor-0] i.e.controlplane.cache.SimpleCache       : open watch 4 for type.googleapis.com/envoy.api.v2.RouteConfiguration[custom-10011-http, custom-10021-http, custom-50055-http, custom-0.0.0.0:14503-http, auto-virtual-hosts, diagnostics] from node default for version 1575899241-1-1
2019-12-09 14:48:27.306 DEBUG 13917 --- [   scheduling-1] p.w.s.dominator.envoy.SnapshotPublisher  : Routes changed
2019-12-09 14:48:27.306 DEBUG 13917 --- [   scheduling-1] p.w.s.d.envoy.EnvoyDiscoveryServer       : Routes names in version 1575899241-1-2: auto-virtual-hosts, diagnostics, custom-50055-http, custom-10011-http, custom-10021-http
2019-12-09 14:48:27.308  INFO 13917 --- [   scheduling-1] i.e.controlplane.cache.SimpleCache       : not responding in ADS mode for type.googleapis.com/envoy.api.v2.RouteConfiguration from node default at version 1575899241-1-2 for request [custom-10011-http, custom-10021-http, custom-50055-http, custom-0.0.0.0:14503-http, auto-virtual-hosts, diagnostics] since [custom-0.0.0.0:14503-http] not in snapshot
2019-12-09 14:48:43.371 DEBUG 13917 --- [   scheduling-1] p.w.s.dominator.envoy.SnapshotPublisher  : Routes changed
2019-12-09 14:48:43.372 DEBUG 13917 --- [   scheduling-1] p.w.s.d.envoy.EnvoyDiscoveryServer       : Routes names in version 1575899241-1-3: auto-virtual-hosts, diagnostics, custom-50055-http, custom-10011-http, custom-0.0.0.0:14503-http, custom-10021-http
2019-12-09 14:53:15.528 DEBUG 13917 --- [   scheduling-1] p.w.s.dominator.envoy.SnapshotPublisher  : Routes changed
2019-12-09 14:53:15.528 DEBUG 13917 --- [   scheduling-1] p.w.s.d.envoy.EnvoyDiscoveryServer       : Routes names in version 1575899241-1-4: auto-virtual-hosts, diagnostics, custom-50055-http, custom-10011-http, custom-10021-http
2019-12-09 14:53:18.556 DEBUG 13917 --- [   scheduling-1] p.w.s.dominator.envoy.SnapshotPublisher  : Routes changed
2019-12-09 14:53:18.557 DEBUG 13917 --- [   scheduling-1] p.w.s.d.envoy.EnvoyDiscoveryServer       : Routes names in version 1575899241-1-5: auto-virtual-hosts, diagnostics, custom-50055-http, custom-10011-http, custom-0.0.0.0:14503-http, custom-10021-http

GRPC protobuff attribute_context.proto does not contain field "body" with index 11.

You can see difference in the next files:

Java-control-plane repo definition:
`
...
message HttpRequest {
// The unique ID for a request, which can be propagated to downstream
// systems. The ID should have low probability of collision
// within a single day for a specific service.
// For HTTP requests, it should be X-Request-ID or equivalent.
string id = 1;

// The HTTP request method, such as `GET`, `POST`.
string method = 2;

// The HTTP request headers. If multiple headers share the same key, they
// must be merged according to the HTTP spec. All header keys must be
// lowercased, because HTTP header keys are case-insensitive.
map<string, string> headers = 3;

// The HTTP URL path.
string path = 4;

// The HTTP request `Host` or 'Authority` header value.
string host = 5;

// The HTTP URL scheme, such as `http` and `https`.
string scheme = 6;

// The HTTP URL query in the format of `name1=value`&name2=value2`, as it
// appears in the first line of the HTTP request. No decoding is performed.
string query = 7;

// The HTTP URL fragment, excluding leading `#`. No URL decoding is performed.
string fragment = 8;

// The HTTP request size in bytes. If unknown, it must be -1.
int64 size = 9;

// The network protocol used with the request, such as
// "http/1.1", "spdy/3", "h2", "h2c"
string protocol = 10;

}
....
`

Envoy proxy repo definition:

`
...
message HttpRequest {
// The unique ID for a request, which can be propagated to downstream
// systems. The ID should have low probability of collision
// within a single day for a specific service.
// For HTTP requests, it should be X-Request-ID or equivalent.
string id = 1;

// The HTTP request method, such as `GET`, `POST`.
string method = 2;

// The HTTP request headers. If multiple headers share the same key, they
// must be merged according to the HTTP spec. All header keys must be
// lowercased, because HTTP header keys are case-insensitive.
map<string, string> headers = 3;

// The request target, as it appears in the first line of the HTTP request. This includes
// the URL path and query-string. No decoding is performed.
string path = 4;

// The HTTP request `Host` or 'Authority` header value.
string host = 5;

// The HTTP URL scheme, such as `http` and `https`.
string scheme = 6;

// This field is always empty, and exists for compatibility reasons. The HTTP URL query is
// included in `path` field.
string query = 7;

// This field is always empty, and exists for compatibility reasons. The URL fragment is
// not submitted as part of HTTP requests; it is unknowable.
string fragment = 8;

// The HTTP request size in bytes. If unknown, it must be -1.
int64 size = 9;

// The network protocol used with the request, such as "HTTP/1.0", "HTTP/1.1", or "HTTP/2".
//
// See :repo:`headers.h:ProtocolStrings <source/common/http/headers.h>` for a list of all
// possible values.
string protocol = 10;

// The HTTP request body.
string body = 11;

}
...
`

Although it is possible to use Java gRPC Stubs on a Kotlin project, the interface is not idiomatic. It would be nice to also ship Kotlin gRPC Stubs in the published library.

This is just documenting the information that may be necessary if java-control-plane wants to start validating regular expression max_program_sizes. This is unnecessary if the regular expressions are considered "trusted" (known to be non-malicious).

I had this information and wanted to dump it somewhere that it may be useful if the need arises. It seems there is no explicit management of GoogleRE2 in this repo. Feel free to close.

In envoyproxy/envoy#10971, Envoy deprecated the max_program_size configuration field in favor of the control plane being responsible. This is great, and works for C++ and Go languages, but leaves some TODOs for Java.

The C++ re2 exposes RE2::ProgramSize() which reports the number of instructions of the compiled regular expression. This is the "program size" that Envoy was referring to.

Go's regexp/syntax package allows getting something similar:

regex, err := syntax.Parse("INPUT HERE", 0)
prog, err := syntax.Compile(regex)
len(prog.Inst) // this is the program size

But it does not appear re2j supports such a feature. Basically, you will need a new Pattern method that returns the length of Prog.inst (via re2.prog).

Upgrade to new data-plane-api version and log the Status returned in the discovery request.

Would it be possible to get a maven artifact uploaded so that we can consume it internally? Couldn't find anything on http://search.maven.org

We're eager to start playing around with this to see how this works in practice. If you feel this repo ready for an initial version yet I'll publish a jar internally, but I figured others might be interested in trying it out as well.

SimpleCache uses a ReentrantReadWriteLock for guarding Snapshot map and StatusInfo access, in a scenario where a server has thousands of clients and updates being made, this proves a bottleneck.

We measured our setSnapshot time to sometimes take up to 300ms, this could be, because other createWatches are locking and preventing this to happen, or this could be real time taken because we do have about 500k streams per control plane server. If the latter is true, no watches can be created for 300ms because the write lock is taken, this is definitely not good.

I imagine the most needed synchronisation need is to prevent watches from being created while a new snapshot is being set and being left with an old version. Though the current locking scheme is much more restrictive, one cannot even create two watches concurrently.

We should take a look at this and try to handle it a better way.

Some ideas on how to improve this:

• CacheStatusInfo is internally guarded by a lock that doesn't seem very necessary, we should drop it and use a ConcurrentMap.

• SimpleCache is the most tricky, but we could:

  • Use a readLock to guard createWatch since we don't really mind watches being created concurrently, this is kind of counter intuitive since we're using a read lock but actually writing data, but i've seen this kind of use before. Then use writeLock only in setSnapshot. This strategy would improve things a bit, but it doesn't really solve our problem.
  • We could probably readLock entire createWatch but writeLock only snapshots.put on setSnapshot. I have to think about the implications of this a little more, but it would seem that if we do that, there would be no conflicts since createWatch would always have a consistent view of snapshots. Currently creating watches would see and old version of snapshot but manage to create it's watches before setSnapshot can lock and insert the new snapshot, after that, all new watches actually see that version and should have no problemas, while setSnapshot continues to iterate through watches and responding accordingly.
  • We could almost remove all locks and provide another way to ensure that watches are being responded to correctly when new snapshots are being created concurrently. Maybe make createWatch double check if a new snapshot has been created.

More ideas are welcome, but it would seem this locking mechanism should be reworked to allow better performance, we're not even stressing our cpus right now.

Edit: Another idea would be to not drop watches everytime we respond to them, this would decrease the lock contention a lot, has anyone thought about this before?