AwsDiscoveryStrategy.discoverNodes throws exception causes node to shutdown immediately about hazelcast-aws HOT 12 CLOSED

The issue will be fixed by #71.

@ps-jay , the issue is really only about the retries. The log you see is actually wrong, Hazelcast does not wait "300000 ms" before printing this message. I've sent a fix for that (hazelcast/hazelcast#13164).

from hazelcast-aws.

Please see hazelcast/hazelcast#12581

This could need to be fixed on core side instead of AWS.

from hazelcast-aws.

Related Zendesk issue: https://hazelcast.zendesk.com/agent/tickets/3809

from hazelcast-aws.

related to #28

from hazelcast-aws.

@mesutcelik , @leszko do you think that this should be fixed in aws repo or in the core, since it affects all SPI discoveries?

from hazelcast-aws.

@gokhanoner, why do you think it affects other environments than AWS?

from hazelcast-aws.

@leszko We have another ZD ticket related to this.
I guess what Gokhan means is; the same behaviour which was fixed on client side via hazelcast/hazelcast#12457 is also observed on member side aws discovery as seen on the ticket: https://hazelcast.zendesk.com/agent/tickets/3926

from hazelcast-aws.

Ok, if both are observed only in AWS, I'd assume it's related to AWS only. So, we can keep this issue here and link it form Zendesk.

from hazelcast-aws.

I think the current behavior seems correct. the difference between this issue and client is that client retry mechanism wasn't working because of unhandled exception on the client side. I think we need to shutdown the member if it can't fetch IPs of the other cluster members. However, we need to define the cases of such failures and put proper logging otherwise Ops Guys would not know how to correct the cause behind fail fast shutdown situation.

@leszko I suggest you to investigate with @gokhanoner and try to reproduce his case...

from hazelcast-aws.

Hi @mesutcelik,

I think there is a problem with the current behaviour, as it seems to "trick" Hazelcast into thinking that 300000ms has elapsed.

From the logs in the first comment on this issue, you can see only 10s elapsed, but Hazelcast shutdown:

2018-04-06 16:08:18,927Z INFO  [main] com.hazelcast.instance.Node - [10.205.89.32]:11415 [dev] [3.9.3] Activating Discovery SPI Joiner

<...snip...>

2018-04-06 16:08:29,213Z ERROR [main] com.hazelcast.internal.cluster.impl.DiscoveryJoiner - [10.205.89.32]:11415 [dev] [3.9.3] java.net.SocketTimeoutException: connect timed out

<...snip...>

2018-04-06 16:08:29,216Z ERROR [main] com.hazelcast.instance.Node - [10.205.89.32]:11415 [dev] [3.9.3] Could not join cluster in 300000 ms. Shutting down now!

I have observed the same behaviour, and can post those logs too if you need.
In my case, it's not a SocketTimeoutException, it's a HazelcastException because of a 503 API Rate Limit Exceeded error from Amazon.

Regards,
Phil

from hazelcast-aws.

So the fix will be that Hazelcast won't shutdown if enountering AWS throttling on startup?

from hazelcast-aws.

yes, but also it will retry the number of times specified by the parameter and only then return an empty list. So in result, if there is no connection to AWS, you may end up with the Split Brain, but Hazelcast won't shutdown.

from hazelcast-aws.