tyro / rabbit-amazon-bridge Goto Github PK

This may vary from organisation to organisation so we should allow it to be configured.

Recently we got a connection issue, but the logs are missing some key information that would help us debug what's going on. For example, this is what we got:
{"exception_class":"com.amazonaws.SdkClientException","exception_message":"Unable to execute HTTP request: Connect to **** [****/****] failed: Connection refused (Connection refused)","stacktrace":"com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1136), com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1082), com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745), com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719), com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701), com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669), com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651), com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515), com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2147), com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2116), com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2105), com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1559), com.amazonaws.services.sqs.AmazonSQSAsyncClient$14.call(AmazonSQSAsyncClient.java:1057), com.amazonaws.services.sqs.AmazonSQSAsyncClient$14.call(AmazonSQSAsyncClient.java:1051), java.util.concurrent.FutureTask.run(FutureTask.java:266), java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149), java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624), java.lang.Thread.run(Thread.java:748)"}}

It would be useful to know what we were polling when the error happened

It appears that the @Profile settings for the AWSSqsConfig vs the AWSSqsTestConfig are clashing unless we provide both the test and docker-integration-test profiles when running the application.

We just need to use more sensible profile expressions to prevent them from clashing with each other.

I also noted that we're not annotating the sqs beans with a shutdown method like spring does in SqsClientConfiguration and we should try to be as similar as possible as we are rolling our own.

We currently support transformation when going from rabbit to AWS.
Though we currently do not use it in our on-prem solution, we can imagine that it would be useful to some.

Missed in #4 and initial import.
This just needs to be cleaned up.

The build won't run as it is missing some setup in the pom.

In some environments the user may not wish to dead-letter on an error.
Provide options to drop the message, nack it so it gets replayed, or dead-letter it.
A question is should there be a default behaviour and what would it be?

The build image we are pulling in right now is not available publicly.

We originally put this rule in place to force us to do a particular transformation on all bridges. This made sense for our use case in Tyro, but does not make sense for the base rabbit-amazon-bridge. We think this would make more sense for users to handle via a unit or integration test of their bridge config.

We think we should remove this behaviour for now.

This is to reduce the need to duplicate the configuration to disable or enable particular queues when testing new queues prior to a production deploy.

Lots of improvements can be made to the documentation:

• How to run it
• Diagrams
• Links to jolt
• Custom properties that we added
• Useful Spring properties that can be used to customise it

It would make sense to add alternatives to just deadlettering every message if Amazon is unavailable.

For this I would propose:

• Catching all recoverable exceptions, e.g. AmazonClientException or HttpException
• Allowing the setting of a TTL on deadletter queues or; Enable spring-retry by default, or and throwing the recoverable exceptions. Of just document the properties in the README
  spring.rabbitmq.listener.simple.retry.enabled=true spring.rabbitmq.listener.simple.retry.initial-interval=5000 spring.rabbitmq.listener.simple.retry.max-attempts=5 spring.rabbitmq.listener.simple.retry.multiplier=1.5 spring.rabbitmq.listener.simple.retry.max-interval=30000

We currently reference a private.properties file. This is something that we had to do for our application due our on-prem setup. If we can't change our on-prem setup and remove this property file we can look at making its path a configuration option.

We should ideally run the build in docker so that we are less dependent on the developers environment.

This got messed up when we copied it from another project.

The build process is a bit complicated and hard to understand.
We could simplify things by:

• Build using maven, rather than maven in docker
• Test and integration test using maven, rather than maven in docker-compose. We already spin up required containers using docker-maven-plugin

We should also include a additional CVE scans in the pipeline.

We started using Gson when we started this project, then pulled in Jolt that uses Jackson under the hoods. We should just stick with Jackson. It is Springs default reducing configuration.

There is also a .jackson-kotlin-module that respects kotlins null safety during deserialisation which will eliminate some custom validation in our bridge file parser.

This will allow a smoother release process

This was left in by accident when we split the From and To definitions for Rabbit. It can be removed as publishers to do not know about queues.

Consider combining the .build and .deploy Dockerfiles into a single Dockerfile and using a multi-stage build: https://docs.docker.com/develop/develop-images/multistage-build/

For ease of deployment we should look at producing a docker image.
Once we have one, we can then consider pushing to docker hub as well as maven central

We agreed that we prefer our logging to be structured as JSON so we will enable this by default.
This can of course be overridable using standard spring properties.

We are a couple or releases behind. We should get it up to date.

I've seen some errors when it runs with java 9. We should make sure it works with Java 11.

We can just use the default spring ones.

cloud.aws.credentials.accessKey=${aws.credentials.accessKey}
cloud.aws.credentials.secretKey=${aws.credentials.secretKey}

We migrated to jolt and can remove this now. Jolt provides more flexibility as well.

In order to provide for more flexibility for later changes and improve the base model.

e.g.

 "from" : {
      "rabbit": {
        "exchange": "exchange-name-1",
        "queueName": "queue-name-1",
        "routingKey": "routing-key-1"
      }
    },
    "transformationSpecs": {
       "type": "jolt",
       "transformations":  [{
         "operation": "shift",
         "spec": {
           "fieldA": "fieldA",
           "fieldB": "fieldB"
         }
     }]
    },
    "to" : {
      "sns": {
        "name":"rabbit-bridge-test-topic"
      }
    }
  }