Space where we will setup DataGrid 8.4 on Azure VMs and run some performance tests

For this test run I am using 5 Azure virtual machines with 4 vCPUs and 64GB of RAM. I am also attaching 2 TB of SSD for caching evicted entries. Idea is to cache entries when we get our OFF_HEAP memory full.

Using latest RHEL 9.2 with my active subscription and java-17-openjdk-devel using yum.

This will register your VM to Red Hat Cloud access program and if Simple Cloud Access is enabled then you wont need to attach any subscription.

sudo subscription-manager register

You can verify status in two ways

sudo subscription-manager status

or

sudo subscription-manager identity

If your VMs is registered successfully then you can use yum to install Java.

sudo yum install java-17-openjdk-devel

Verify

java -version

And then add Java Home permanently

sudo vi /etc/bashrc

and add below two line at the bottom on the bashrc

export JAVA_HOME=/usr/lib/jvm/jre
export PATH=$JAVA_HOME/bin:$PATH

Source and Verify if JAVA_HOME is set

source /etc/bashrc
printenv | grep JAVA_HOME

Before proceeding make sure that you have SSDs attached to your VM instances. In my example I used 2TB Premium SSD from Azure while creating VMs.

There are many ways you can mount the disk but I followed the procedure outlined here specifically for Azure: https://katarinaslama.github.io/2020/02/26/mounting-extra-storage-on-your-VM/

Make sure you verify that correct user permissions are set before proceeding.

Here I am using DataGrid 8.4. The binaries are available on the RH downloads website.

Copy bits to your local home directory and unzip them.

unzip redhat-datagrid-8.4.0-server.zip

Go into unziped folder and create a user for DataGrid

./bin/cli.sh user create developer -p developer -g admin

Lets start the server lcoally

./bin/server.sh -b 0.0.0.0 &

Verify if server started at localhost

curl --digest http://develoepr:developer@localhost:11222/rest/v2/security/user/acl -v

If you see some content at the bottom of your screen then we are a step closer to our goal. Cheers!

As you can notice in above curl command console is accessible at 11222 port and server starts at 7800 port. So we will need to open up both ports in firewall and on Azure networking security group on Azure Portal. And these ports need to be added to both ingress and egress rules for our VMs to comminicate with their cluster members.

After you add firewall rules to allow ports 11222 and 7800 on Azure portal for ingress and egress, proceed below to open up firewall from your VM.

sudo firewall-cmd --zone=public --permanent --add-service=http && \
sudo firewall-cmd --zone=public --permanent --add-port 11222/tcp && \
sudo firewall-cmd --zone=public --permanent --add-port 7800/tcp && \
sudo firewall-cmd --reload && \
sudo firewall-cmd --list-all

Now you can verify if above settings were applied correctly by loading console on your browser using public IP and port 11222.

Now, we will need to let our DataGrid VMs notify of other VMs by modifying infinispan.xml file and adding below block.

Replace localhost to your VM's IP or hostnames. Remember hostnames and IPs should be reachable from other nodes.

Locate infinispan.xml file inside your unziped folder

vi server/conf/infinispan.xml

Update file with below block, which defined static members which we specify as initil_hosts and a cache name 'respCache'.

   <jgroups>
      <stack name="myazure" extends="tcp">
         <TCPPING initial_hosts="10.0.3.4[7800],10.0.0.8[7800],10.1.2.3[7800],10.1.2.4[7800],10.1.3.3[7800]" port_range="0" stack.combine="REPLACE" stack.position="MPING"/>
      </stack>
   </jgroups>
   <cache-container name="default" statistics="true">
      <transport cluster="${infinispan.cluster.name:cluster}" stack="${infinispan.cluster.stack:myazure}" node-name="${infinispan.node.name:}"/>
      <security>
         <authorization/>
      </security>
      <replicated-cache name="respCache" mode="ASYNC" statistics="true">
         <encoding>
            <key media-type="application/x-protostream"/>
            <value media-type="application/x-protostream"/>
         </encoding>
         <memory storage="OFF_HEAP" max-size="48GB" when-full="REMOVE"/>
         <persistence>
           <file-store shared="false" open-files-limit="10000">
              <data max-file-size="167772160"/>
           </file-store>
         </persistence>
      </replicated-cache>
      <global-state>
            <persistent-location path="/data/dg"/>
      </global-state>
   </cache-container>

And now lets start our servers again and verify if you see cluster members in the console or logs as in this screen shot

You can either user Azurel LB here or bring your own. I went through easy path of using Azure LB.

One thing to note for Azure LB is to make sure you have your VM IPs in Standard SKU and not in Basic SKU. Without having them in Standard SKU, you wont be able to add these VMs in backend pool for Load Balancer which was created in Standard SKU.

Now, lets just setup benchmark tool locally and kick off our perfromance test.

Somewhere in your local directory perfrom below operations to download, unzip and compile to generate redis-benchmark tool.

wget http://download.redis.io/releases/redis-7.0.5.tar.gz
tar xzf redis-7.0.5.tar.gz
cd src
make

Above will compile the source and generate tool under folder redis-benchmark under src folder.

We will be running this tool with about 1 million requests and perfrom both SET and GET operation using a random key for every operation out of 1m possible keys with 120kb of payload.

./redis-benchmark -t set,get -d 120000 -r 1000000 -n 1000000 -h {lbIP} -p 11222 --user developer -a developer

After few minutes you should see results on your console with summary of test run.

for example:

SET Summary:
  throughput summary: 1228.53 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
       40.616     0.376    24.415    59.519   499.455  3000.319

GET Summary:
  throughput summary: 10417.86 requests per second
  latency summary (msec):
          avg       min       p50       p95       p99       max
        1.011     0.352     1.007     1.271     1.423   211.839

Log in to your console and you can browse the performance test statistics on console.