This will allow distinct surface stability guarantees and also will allow non-grpc APIs to not pull in grpc as a dependency.

Not sure if this bug should go in googleapis/toolkit instead, but logging this issue here for reference.

When looking at the JavaDoc of a specific GAPIC generated class, we should be able to access the underlying types (e.g. when clicking on the parameters or return values of a specific method).

For Cloud Pub/Sub, the maximum possible PullResponse is 20 MB large. The default gRPC Channel maxMessageSize is 4 MB. This should be configured per API, or some messages will be silently undeliverable.

The new(ish) PropertiesProvider writes exceptions and other messages to System.err instead of using a logger. This creates a significant amount of noise for what appear to be non-critical errors.

For example, loadProperty invokes System.err.printStackTrace on line 81, but the return null after the catch suggests exceptions there are non-critical. By using a logger instead of System.err, users can suppress these messages if desired.

Example stderr output:

java.lang.NullPointerException
	at java.util.Properties$LineReader.readLine(Properties.java:434)
	at java.util.Properties.load0(Properties.java:353)
	at java.util.Properties.load(Properties.java:341)
	at com.google.api.gax.core.PropertiesProvider.loadProperty(PropertiesProvider.java:69)

Thanks

e.g. -> RetryParams.newBuilder() vs ConnectionSettings.builder()

My preference would be to use builder().

Currently, streaming methods don't translate exceptions, so the user just sees the raw StatusRuntimeException. We need to add exception translation like unary methods have.

Please ensure that the header we use with GAX is x-goog-client-api and that the format is similar to the one used for Go and Python.

Go: https://github.com/GoogleCloudPlatform/gcloud-golang/blob/master/logging/apiv2/logging/client.go#L125

Autovalue should be used as "compileOnly" dependency not as "compile" in the gradle config. That implies your jar should not depend on "com.google.auto.value:auto-value"

See here:
https://mvnrepository.com/artifact/com.google.api/gax/1.15.0

In opencensus-java we also use auto-value see here:
https://github.com/census-instrumentation/opencensus-java/blob/4aa6e80c8efac571e3017b70d599eb52db23a6f3/api/build.gradle

This includes classes, methods, variables, docs, etc.

cc @garrettjonesgoogle
If we call getChannel(), the provider gets an executor from a call to executorProvider.getExecutor(). Depending on the implementation of executorProvider, we might create a new executor in the process.

Who's responsible for shutting down this executor? My best guess is that the returned channel should shut down the executor whenever the channel itself is shut down. If this is the case, I think it should be explicitly documented.

@garrettjonesgoogle, @michaelbausor I found the ProtoReflectionUtils class (https://github.com/googleapis/gax-java/blob/05d4c20d2a302fee410001fe9e34fb7f59a64d7b/src/main/java/com/google/api/gax/protobuf/ProtoReflectionUtil.java) while looking for a way to get the Java Enum class of a proto enum field reflectively.

I see that the ProtoReflectionUtils class was removed in 964d3ab

Is that class or its functionality moved somewhere else, or is it retired for good?

Thank you,
Nikolay

This was briefly discussed internally. As per title we should drop the grcp-all dependency and favor cherry-picking gRPC dependencies.

This should help us getting rid of some dependencies as well as duplicated classes (e.g. the ones pulled by both protobuf-java and protobuf-lite).

ServiceApiSettings.createChannelProvider only uses the executor for ClientAuthInterceptor, but it needs to be used for the Channel too.

The user doesn't know what call causes exceptions:

Exception in thread "main" com.google.api.gax.grpc.ApiException: io.grpc.StatusRuntimeException: ALREADY_EXISTS: Resource already exists in the project (resource=zipkin).

A service API (e.g. PublisherApi) can be configured with its own specific PublisherSettings settings.

Some of PublisherSettings configuration are inherited from ServiceApiSettings.

To allow chaining this is returned from the settings methods, however because PublisherSettings
does not override its parent methods the returned this would be ServiceSettings and not PublisherSettings (which breaks chaining).

Also, I think it would have been nicer if ServiceApiSettings was immutable and using a Builder for construction.

Following this comment from Garrett I am trying to reuse the same channel to instantiate multiple service objects.

I can do that by creating the channel myself and passing it to ServiceApiSettings.provideChannelWith(ManagedChannel, boolean), this however has several drawbacks:

• I need to duplicate the code in createChannelProvider
• The channel is no longer created lazily

A method like ServiceApiSettings.setChannelProvider(ChannelProvider) could really help on this. So that I can pass ServiceApiSettings.getChannelProvider() of a previously created settings object.

Related to this issue, the semantics of channel providers created with ServiceApiSettings.createChannelProvider(ConnectionSettings) is a bit confusing to me. The method getOrBuildChannel seems to create a new channel every time it's called, rather than creating it only the first time. Is this behavior intended?

@garrettjonesgoogle @davidtorres

From our meeting last Thursday, I think the decision was

• These fields should be set and non-zero
  • elementCountThreshold
  • requestByteThreshold
  • delayThreshold
• Delete the rest

I wanted to make sure I get it right to avoid code churn since this is tightly coupled with work in google-cloud-java.

Dear Google team,

During our internal performance testing we found out that there's a resource leak (which is causing abnormal native memory consumption) in com.google.api.gax.core.PropertiesProvider class. Test app with google datastore api consumed more than 20 GB of RAM and java process got killed by linux's OOM killer.

Methods

PropertiesProvider#loadProperty(Class<?> loadedClass, String propertiesPath, String key)
and
PropertiesProvider#loadProperty(Properties properties, String propertiesPath, String key)

are loading resources but not closing them(input stream is not closed). So native resources are held using java.util.zip.Inflater (See java.util.zip.GZIPInputStream).

Another proposal is to cache results of property loading by key. Even dumb implementation with synchronization of entire method and HashMap(as cache provider) will work quicker than loading of resource each time.

Tested on linux with latest JDK 8.

BR,
Dionis

Consequences:

1. Guava can't be exposed on the public surface of GAX
2. gRPC can't be exposed on the public surface of GAX

Number 2 will be the larger effort, because Guava is already in the process of being hidden, but gRPC is currently exposed.

You can get a PageAccessor through an async call, but then it's not possible to do further async calls, so providing a getNextPageAsync() is necessary for completeness.

In particular, grpc-google-common-protos exposes Guava. Depends on googleapis/gapic-generator#1145 .

The FlowController:: reserve doesn't take the initial size of the semaphore into account when blocking.

Background: trying to provide limits to the pub/sub subscriber - wanted one-at-a-time. With the size of the semaphore set to 1, pub/sub subscriber code will usually pull back more than one message (could be as many as 1000 if I'm reading the code correctly). Each thread that processes pub/sub messages gets stuck on this call [and will never unblock].

Specifically, in https://github.com/googleapis/gax-java/blob/master/src/main/java/com/google/api/gax/grpc/InstantiatingChannelProvider.java . NettyChannelBuilder is @Experimental so we shouldn't depend on it directly; all of the same functionality should be available in ManagedChannelBuilder.

Also consider an initial wait period that is relative to the size of the input (would be API-specific).

Copying over some possible issues found in googleapis/google-cloud-java#1432:

1. com.google.api.gax.bundling.ThresholdBundler has an unbounded queue for closedBundles that will fill endlessly if an application is publishing faster than the queue can be drained, or if draining/publishing can't occur at all for some reason. We've seen on a few instances (3-5 applications out of 1000s) that this eventually leads to a OutOfMemoryError as the heap space fills up.

2. It seems like if the thread that ThresholdBundlingForwarder starts dies for any reason, it won't be restarted, and the messages accumulating in the Bundler will not be processed at all (eventually leading to the OOM above).

I am only familiar with the bundling behavior as it is used in google-cloud-pubsub, but for the first issue I think it may make sense for the Bundler to allow for specifying a maximum number of bundles/items that it will hold onto, and then either reject new items from being added or drop the oldest bundles/items when this limit is hit.

Currently, only gax-grpc is showing up in the javadocs.

Move the pom version logic from generated code into the MetadataProvider class.

The docs for gax-java are out of date. They seem to be stuck on version 1.9.1 while the library is on 1.14.0.

PubSub library for Java doesn't use the new endpoint pubsub.googleapis.com. Let's regenerate the auto-gen layer so that the new endpoint is used.

@garrettjonesgoogle

In an upcoming PR, gax is split into individual projects. Google-java-format only works on the first project it encounters, and not the rest. This probably requires a fix in google-java-format.

Currently the default gax version is hard-coded.

When creating a PubSub's PublisherApi object I am getting the following IllegalStateException error:

java.io.IOException: java.lang.IllegalStateException: Missing required properties: totalTimeout
    at com.google.cloud.pubsub.spi.DefaultPubSubRpc.<init>(DefaultPubSubRpc.java:87)
    at com.google.cloud.pubsub.PubSubOptions$DefaultPubSubRpcFactory.create(PubSubOptions.java:57)
    ... 23 more
Caused by: java.lang.IllegalStateException: Missing required properties: totalTimeout
    at com.google.api.gax.core.AutoValue_RetrySettings$Builder.autoBuild(AutoValue_RetrySettings.java:264)
    at com.google.api.gax.core.RetrySettings$Builder.build(RetrySettings.java:220)
    at com.google.api.gax.grpc.SimpleCallSettings$Builder.build(SimpleCallSettings.java:73)
    at com.google.cloud.pubsub.spi.v1.PublisherSettings.<init>(PublisherSettings.java:219)
    at com.google.cloud.pubsub.spi.v1.PublisherSettings.<init>(PublisherSettings.java:103)
    at com.google.cloud.pubsub.spi.v1.PublisherSettings$Builder.build(PublisherSettings.java:556)
    at com.google.cloud.pubsub.spi.DefaultPubSubRpc.<init>(DefaultPubSubRpc.java:78)

However we do set the totalTimeout value: see here.

If I understood it correctly this might be caused by RetrySettings.Builder.merge which does not merge the totalTimeout field.

In the source tree, we don't have version tags. Given an gax-java version, it's pretty hard or near to impossible to find the source associated w/ that version in the source tree.

Partially implemented in #462, but we need a new way to create in-process mock server.

At the same time, make sure that cancellation with mayInterruptIfRunning = false cancels the local execution but not the remote operation.

ServiceApiSettings.applyToAllApiMethods(Iterable<ApiCallSettings.Builder> methodSettingsBuilders, ApiCallSettings.Builder newSettingsBuilder) sets newSettingsBuilder.getRetryableCodes() and newSettingsBuilder.getRetrySettingsBuilder() for all API method settings in methodSettingsBuilders.

applyToAllApiMethods avoids using newSettingsBuilder.getRetryableCodes() if its null, however, there's no way to create an ApiCallSettings.Builder with null retryable codes (always initialized to empty set).

The result is that if I want to use applyToAllApiMethods to set common RetrySettings I am forced to override also retryable codes. Could you add a way to just set RetrySettings for all methods?

/cc @shinfan @garrettjonesgoogle

Using grpc-all creates a larger dependency tree.

Both the upstream dependency gRPC, and downstream consumer google-cloud-java are Apache 2.0 license. This one in the middle glueing the two is BSD.

See grpc/grpc-java#2391 (comment) - ClientAuthInterceptor is deprecated, and doesn't upgrade service accounts to JWTs appropriately in all cases.

ApiFutures.addCallback(future, new ApiFutureCallback<String>() {
          @Override
          public void onFailure(Throwable throwable) {
                   if (throwable instanceof ApiException) {
                     ApiException e = (ApiException) throwable;
                     System.out.println("Api Error Code : " + e.getStatusCode());
                   }
                   System.out.println("Error publishing message : " + message);
          }

e.getStatusCode() currently retrieves :

com.google.api.gax.grpc.GrpcStatusCode@3fce1ed0

Users should be able to not have to perform the following cast to retrieve the status code:

((GrpcStatusCode)e.getStatusCode()).getCode()

Still manual:

• Update version.txt

Full process:

• verify that there are no local changes before starting the process
• update docs (there is already a task for this step - updateDocsWithCurrentVersion)
• commit generated gh-pages (updateDocsWithCurrentVersion puts this under tmp_gh-pages/)
• upload built archives (uploadArchives is the command that needs to be invoked when being performed manually)
• close & release the archive on Sonatype
• commit the auto-generated doc updates
• push the auto-generated doc updates to github