Connecting to a TLS-based Kafka instance under AWS MSK? about kaffe HOT 18 CLOSED

Promising news: it turns out that the previous error was due to a bug on our side, and after some preliminary testing, it looks like we are indeed able to connect to MSK via Kaffe when using the heroku configuration!

We loaded a list of topics into MSK outside of Kaffe, and then we started up our application. One of the first things it does is to get a list of those topics and subscribe to them, and all of that worked.

We have to do more extensive testing of other features of our application, but now we are optimistic that we can get our MSK Kafka instance working via Kaffe.

I will post more news later, once we do more testing, to give you (hopefully!) final confirmation that this is working.

This should occur within the next few days.

from kaffe.

It turns out that we are indeed able to run our application with MSK when configuring up Kaffe in "heroku" mode and setting the appropriate environment variables, including those for the cert and cert key.

This is a relief for us, because we didn't want to go down the road of refactoring our application to use something other than Kaffe.

We might still hit edge cases later on, since we are still in the early stages of putting this in place, but now it seems likely that we will be able to continue with this.

A suggestion: in the future, it might be helpful to offer another setting besides "heroku_kafka_env: true" in order to trigger SSL mode ... perhaps naming it something like "ssl_kafka_env" or "tls_kafka_env". The "heroku_kafka_env" setting could remain and be aliased to this new variable, for backward compatibility.

Anyway, thank you again for all your help!

from kaffe.

@HippoMan Right now connecting to MSK isn't officially supported. It very well might work, but we (Spreedly) haven't tried it out yet. That said, hopefully in the near future we'll be looking at alternatives to Heroku Kafka and will likely need to update Kaffe accordingly. In the meantime, feel free to try a connection to MSK and/or open a PR if need be. We'll keep this issue open for tracking purposes.

cc @hdeters @fatcatt316 @schwarzgeist

from kaffe.

Thank you. I will be trying the environment-variable approach, and I'll report my results, hopefully over the next few days.

from kaffe.

Given the following code in kaffe/config.ex, it appears that kafka+ssl:// isn't necessarily needed at the head of the URL's supplied in KAFKA_URL , because that gets stripped off, anyway ... correct?

  def parse_endpoints(url) when is_binary(url) do
    url
    |> String.replace("kafka+ssl://", "")
    |> String.replace("kafka://", "")
    |> String.split(",")
    |> Enum.map(&url_endpoint_to_tuple/1)

from kaffe.

OK. Just wanted to make sure about TLS. I'll do more checking on our side, to make sure we aren't doing something incorrectly, and I'll report back.

from kaffe.

All of the configuration should be driven by environment variables. If you can create environment variables with the connection information for MSK, it might work. 🤞

from kaffe.

I have a question about this environment-variable approach.
To be clear, is the suggestion to try the following? ...
Pretend that we're running under heroku and set heroku_kafka_env to true,
set KAFKA_CLIENT_CERT and KAFKA_CLIENT_CERT_KEY appropriately,
and then set KAFKA_URL to the address of the MSK Kafka broker?

Would there be anything to do in addition to this in order to try to connect to an MSK Kafka instance?

Ideally, we do not want to alter the Kaffe code, and we just want to use what is already there. Given that SSL is only handled via the heroku-related code, that's why I assume that I'd have to set heroku_kafka_env to true in order for these env vars to be recognized. But in that case, will the fact that I'm telling Kaffe that we're running under heroku when we actually are not running that way mean that other things might break?

Thank you.

from kaffe.

@HippoMan Yes, that's the suggestion. I don't recall anything that's really Heroku specific with that configuration and think it should "just work" if the connectivity is the same. That was created as just a convenience for us since that's how we run. The certs are entirely contained in the ENV vars in Heroku:

KAFKA_CLIENT_CERT:           -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

The URLs look like this:

KAFKA_URL:                   kafka+ssl://***.compute-1.amazonaws.com:9096,kafka+ssl://***.compute-1.amazonaws.com:9096,kafka+ssl://***.compute-1.amazonaws.com:9096

from kaffe.

@HippoMan Yes, I think you're right.

from kaffe.

When setting up as described above, we are getting the following error ...
{"time":"2020-06-22 18:43:01.074","pid":"#PID<0.3334.0>","msg":":brod_client [#PID<0.3334.0>] :ourappname is terminating\nreason: {{{{:query_api_versions_error, {:badmatch, {:error, :enomem}}}, [{:brod_sock, :inactive_request_sync, 5, [file: 'src/brod_sock.erl', line: 258]}, {:brod_sock, :query_api_versions, 4, [file: 'src/brod_sock.erl', line: 235]}, {:brod_sock, :maybe_query_api_versions, 6, [file: 'src/brod_sock.erl', line: 226]}, {:brod_sock, :do_init, 4, [file: 'src/brod_sock.erl', line: 208]}, {:brod_sock, :init, 5, [file: 'src/brod_sock.erl', line: 176]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}, [{'mskur1.com', 9094}, {'mskurl2.com', 9094}]}, [{:brod_client, :start_metadata_socket, 5, [file: 'src/brod_client.erl', line: 708]}, {:brod_client, :handle_info, 2, [file: 'src/brod_client.erl', line: 301]}, {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 637]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 711]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}]}\n","level":"warn"}

I anonymized our application name and the AWS MSK url's in the error message, but everything else is verbatim.

The version of Kafka used under AWS MSK is 2.2.1. Could this be the reason for the "query_api_versions_error" message? Or could this be due to our doing something incorrectly on our end?

The Kafka endpoints work for producing and consuming when accessed via standard MSK-specific utilities. Also, our software works when not pointing to MSK, but to our older Kafka provider's endpoints.

Any ideas as to why this isn't working? Thank you very much!

from kaffe.

I've never seen that error. My guess would be it's due to a fundamental connection error. That's probably just one of the first operations the underlying Brod client performs.

We're using Kafka version 2.2.2, so I think you'd be fine there.

Those ports aren't standard, I'm assuming they're correct, however. 🤔

Yeah, AWS says:

To communicate with producers and consumers in plaintext, brokers use port 9092.
To communicate with producers and consumers in TLS, brokers use port 9094.

So that seems good.

from kaffe.

Thank you. When configured to connect to Heroku, does Kaffe/brod use TLS, or just plain, old-fashioned SSL?

from kaffe.

I'd be surprised if any version of Kafka or anything in Heroku supports true OG SSL. Has to all be TLS. But those client certs are definitely required by Kaffe.

from kaffe.

Will be interested to hear. We're planning on using MSK in the future (not sure how distant).

from kaffe.

@HippoMan Great news! Very excited that you were able to get it working.

Yes the choice of heroku_kafka_env was controversial from the beginning! 😄

Please keep in touch as you experience issues, I'd love to know how it goes!

from kaffe.

Thank you again! If we hit issues, we'll definitely let you know. And if you don't hear anything, you can go by the watchword, "No news is good news."

from kaffe.

Closing since, as Mr. @HippoMan so wisely put it, "No news is good news."

from kaffe.