Comments (13)
feanor07
commented on September 14, 2024
1
@mesutcelik planning on writing a small how-to article if I find some time soon. I will post it to here when I am done 🤞
from hazelcast-aws.
leszko
commented on September 14, 2024
Hi @pkgonan,
Thank for reporting it. We'll check that.
from hazelcast-aws.
mesutcelik
commented on September 14, 2024
Hi @pkgonan
We actually did not test hazelcast-aws in AWS ElasticBeanstalk Environment so I can't judge the effort required. However, your contributions would be very welcome in this.
from hazelcast-aws.
pkgonan
commented on September 14, 2024
Hi, I would like to use hazelcast in AWS Beanstalk environment.
Is there another alternative to use?
from hazelcast-aws.
mesutcelik
commented on September 14, 2024
Hazelcast is not tested in AWS Beanstalk env.
please feel free to test and provide feedback.
You can also use hazelcast in kubernetes env. too
please check for more info.
https://github.com/hazelcast/hazelcast-kubernetes
from hazelcast-aws.
feanor07
commented on September 14, 2024
I tested and used aws elastic beanstalk via deploying a relatively simple application. It worked fine. One thing I need to mention is that you need to ensure security groups you use allow inbound traffic for ports hazelcast is relying on.
from hazelcast-aws.
mesutcelik
commented on September 14, 2024
@feanor07 Tesekkurler!
It would be useful for others if you share you codesample on github so that other people can benefit from it.
The other option is to write a guide that will be hosted by hazelcast but not sure how to automate integration testing with Elasticbeanstalk to check the validity of the guide over time.
https://github.com/hazelcast-guides/base-guide/wiki/How-to-write-a-guide
from hazelcast-aws.
jklingsporn
commented on September 14, 2024
@feanor07 If you have the time, can you please explain to me how you did it (just some bullet points)? Like any deviations from the default configuration? I am trying to configure it defining an IAM-role and using tags. I've also setup the security groups to allow any TCP communication on any port and provided the IAM-role with the necessary EC2-permissions but without luck. Your help would be really appreciated.
from hazelcast-aws.
jklingsporn
commented on September 14, 2024
Hazelcast-version 4.0.2 / hazelcast-aws-version 3.12
This is my ElasticBeanstalk configuration:
- one application
- two environments (A & B)
- web server tier
- using the same VPC
- having security group hazelcast-dev
- using the same EC2 instance role
- instance type t3.small
- java8 se platform
- EC2 instance role is granted AmazonEC2FullAccess
- hazelcast-dev security group allows all TCP traffic having itself as source
I am using vertx 4.0.0.Beta1 with the following module: https://vertx-web-site.github.io/docs/vertx-hazelcast/java/
This is my hazelcast-configuration:
<hazelcast xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.hazelcast.com/schema/config
http://www.hazelcast.com/schema/config/hazelcast-config-4.0.xsd">
<properties>
<property name="hazelcast.wait.seconds.before.join">0</property>
<property name="hazelcast.jmx">true</property>
<property name="hazelcast.member.list.publish.interval.seconds">5</property>
</properties>
<cluster-name>dev</cluster-name>
<network>
<port auto-increment="false">5701</port>
<join>
<multicast enabled="false"/>
<aws enabled="true">
<tag-key>cluster</tag-key>
<tag-value>dev</tag-value>
<region>eu-central-1</region>
<iam-role>aws-elasticbeanstalk-ec2-role</iam-role>
<hz-port>5701</hz-port>
<connection-timeout-seconds>10</connection-timeout-seconds>
</aws>
</join>
</network>
<multimap name="__vertx.subs">
<backup-count>1</backup-count>
<value-collection-type>SET</value-collection-type>
</multimap>
<map name="__vertx.haInfo">
<backup-count>1</backup-count>
</map>
<map name="__vertx.nodeInfo">
<backup-count>1</backup-count>
</map>
<cp-subsystem>
<cp-member-count>0</cp-member-count>
<semaphores>
<semaphore>
<name>__vertx.*</name>
<jdk-compatible>false</jdk-compatible>
</semaphore>
</semaphores>
</cp-subsystem>
</hazelcast>
The observed behavior is as follows: upon application start the nodes do not detect each other. This can be proven when configuring one application as a lite-member. The boot will fail with the following message:
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.267 [vert.x-worker-thread-0] INFO c.hazelcast.aws.AwsDiscoveryStrategy - Availability zone found: 'eu-central-1c'
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.267 [vert.x-worker-thread-0] DEBUG c.h.internal.cluster.ClusterService - [172.31.8.247]:5701 [dev] [4.0.2] Setting master address to null
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.268 [vert.x-worker-thread-0] DEBUG c.h.i.cluster.impl.DiscoveryJoiner - [172.31.8.247]:5701 [dev] [4.0.2] This node will assume master role since none of the possible members accepted join request.
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.268 [vert.x-worker-thread-0] TRACE c.h.i.c.impl.ClusterJoinManager - [172.31.8.247]:5701 [dev] [4.0.2] This node is being set as the master
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.268 [vert.x-worker-thread-0] DEBUG c.h.internal.cluster.ClusterService - [172.31.8.247]:5701 [dev] [4.0.2] Setting master address to [172.31.8.247]:5701
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.269 [vert.x-worker-thread-0] DEBUG c.h.i.cluster.impl.MembershipManager - [172.31.8.247]:5701 [dev] [4.0.2] Local member list join version is set to 1
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.269 [vert.x-worker-thread-0] DEBUG c.h.i.cluster.impl.DiscoveryJoiner - [172.31.8.247]:5701 [dev] [4.0.2] PostJoin master: [172.31.8.247]:5701, isMaster: true
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.270 [vert.x-worker-thread-0] INFO c.h.internal.cluster.ClusterService - [172.31.8.247]:5701 [dev] [4.0.2]
Aug 21 15:05:08 ip-172-31-8-247 web: Members {size:1, ver:1} [
Aug 21 15:05:08 ip-172-31-8-247 web: Member [172.31.8.247]:5701 - f96cf760-7a1b-4da1-9812-093c6164179d this lite
Aug 21 15:05:08 ip-172-31-8-247 web: ]
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.278 [vert.x-worker-thread-0] TRACE c.h.i.m.ManagementCenterService - [172.31.8.247]:5701 [dev] [4.0.2] Creating new executor for Management Center service tasks with threadCount=2
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.278 [vert.x-worker-thread-0] TRACE com.hazelcast.config.Config - No configuration found for PhoneHome, using default config!
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.279 [vert.x-worker-thread-0] TRACE com.hazelcast.config.Config - No configuration found for PhoneHome, using default config!
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.287 [vert.x-worker-thread-0] TRACE c.h.i.diagnostics.HealthMonitor - [172.31.8.247]:5701 [dev] [4.0.2] HealthMonitor started
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.288 [vert.x-worker-thread-0] INFO c.h.internal.jmx.ManagementService - [172.31.8.247]:5701 [dev] [4.0.2] Hazelcast JMX agent enabled.
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.309 [vert.x-worker-thread-0] INFO com.hazelcast.core.LifecycleService - [172.31.8.247]:5701 [dev] [4.0.2] [172.31.8.247]:5701 is STARTED
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.330 [vert.x-worker-thread-0] TRACE c.h.internal.metrics.MetricsRegistry - [172.31.8.247]:5701 [dev] [4.0.2] Registered probeInstance [service=hz:impl:multiMapService,unit=count,metric=event.listenerCount]
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.330 [vert.x-worker-thread-0] TRACE c.h.internal.metrics.MetricsRegistry - [172.31.8.247]:5701 [dev] [4.0.2] Registered probeInstance [service=hz:impl:multiMapService,unit=count,metric=event.publicationCount]
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.453 [vert.x-acceptor-thread-0] TRACE io.vertx.core.net.impl.NetServerImpl - Net server listening on 172.31.8.247:/172.31.8.247:42189
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.465 [vert.x-worker-thread-2] WARN c.h.i.p.impl.PartitionStateManager - [172.31.8.247]:5701 [dev] [4.0.2] No member group is available to assign partition ownership...
Aug 21 15:05:08 ip-172-31-8-247 web: 15:05:08.469 [vert.x-eventloop-thread-0] ERROR io.vertx.core.impl.VertxImpl - Failed to initialize clustered Vert.x
Aug 21 15:05:08 ip-172-31-8-247 web: com.hazelcast.partition.NoDataMemberInClusterException: Target of invocation cannot be found! Partition owner is null but partitions can't be assigned since all nodes in the cluster are lite members.
Aug 21 15:05:08 ip-172-31-8-247 web: at com.hazelcast.spi.impl.operationservice.impl.PartitionInvocation.newTargetNullException(PartitionInvocation.java:90)
Aug 21 15:05:08 ip-172-31-8-247 web: at com.hazelcast.spi.impl.operationservice.impl.Invocation.initInvocationTarget(Invocation.java:270)
Aug 21 15:05:08 ip-172-31-8-247 web: at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:562)
Aug 21 15:05:08 ip-172-31-8-247 web: at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:540)
Aug 21 15:05:08 ip-172-31-8-247 web: at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:237)
Aug 21 15:05:08 ip-172-31-8-247 web: at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:59)
Aug 21 15:05:08 ip-172-31-8-247 web: at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:468)
Aug 21 15:05:08 ip-172-31-8-247 web: at com.hazelcast.map.impl.proxy.MapProxySupport.putInternal(MapProxySupport.java:407)
Aug 21 15:05:08 ip-172-31-8-247 web: at com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:121)
Aug 21 15:05:08 ip-172-31-8-247 web: at com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:111)
Aug 21 15:05:08 ip-172-31-8-247 web: at io.vertx.spi.cluster.hazelcast.HazelcastClusterManager.lambda$setNodeInfo$2(HazelcastClusterManager.java:173)
Aug 21 15:05:08 ip-172-31-8-247 web: at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:179)
Aug 21 15:05:08 ip-172-31-8-247 web: at io.vertx.core.impl.AbstractContext.emit(AbstractContext.java:181)
Aug 21 15:05:08 ip-172-31-8-247 web: at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:177)
Aug 21 15:05:08 ip-172-31-8-247 web: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Aug 21 15:05:08 ip-172-31-8-247 web: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Aug 21 15:05:08 ip-172-31-8-247 web: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Aug 21 15:05:08 ip-172-31-8-247 web: at java.lang.Thread.run(Thread.java:748)
When both environments have light-member setting set to false, the nodes only discover after several minutes.
from hazelcast-aws.
feanor07
commented on September 14, 2024
@jklingsporn I have been quite busy these couple of weeks; hope will find some time to respond back with my configuration soon.
from hazelcast-aws.
jklingsporn
commented on September 14, 2024
Just a quick update: when I disable aws-discovery for the nodes that are lite-members and enable tcp-id with a fixed ip, everything works like a charm.
from hazelcast-aws.
jklingsporn
commented on September 14, 2024
@feanor07 our configuration has some issues regarding resilience. Let's say AWS decides to replace our only know instance with another machine, our whole cluster will die. Also, when we have to reboot the known cluster-member we have to reboot all other nodes as well in order to have a working cluster.
from hazelcast-aws.
leszko
commented on September 14, 2024
@jklingsporn I think you need to change the Deployment policy to Rolling. Otherwise, you kill all Hazelcast members at once, and therefore the cluster is down and loses data.
I tested Hazelcast on AWS Elastic Beanstalk and it works correctly. I added a description in this PR: #207. It should close this issue. I suggest opening new GH issues with more precise steps to reproduce if anyone encounters any problem with the Beanstalk environment.
from hazelcast-aws.
Related Issues (20)
- AWS ECS Fargate throws "Unable to lookup role in URI" HOT 6
- Fargate discovery doesn't work properly under Quarkus and GraalVM HOT 1
- Fargate throws SocketException while starting the spring boot application instance HOT 12
- Question: Hazelcast Client on Aws ECS bridge mode HOT 3
- InvalidConfigurationException: Unknown properties: '[host-reader]' on discovery strategy HOT 4
- multiple aws tags HOT 9
- Hazelcast client auto discovery in AWS Fargate HOT 2
- hazelcast keep picking 5701 to 5708 ports HOT 2
- Change default hz-port to `5701-5703` HOT 1
- hazelcast not picking java config HOT 1
- ECS Clients tries to fetch public IPs despite AwsConfig#usePublicIp=false HOT 3
- Handle different iam roles between members HOT 1
- Standalone start warning is logged many times. HOT 1
- Auto Discovery for confluence data center not working HOT 4
- IMDSv2 support? HOT 13
- Combining ECS/Fargate clusters (for Hazelcast clients) and EC2 Clusters (for Hazelcast cluster members) HOT 2
- Question: Hazelcast Client on Aws ECS bridge mode
- Hazelcast Split-Brain Protection
- README Update for Documentation HOT 1
- Error while configuring hazelcast configuration in AWS ECS Fargate
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hazelcast-aws.