We've just migrated our production system into HANA and are using
system replication. At first we ran with replicationmode=syncmem and
operationmode=logreplay, but hit SAP note 2329460. Since we
already had users on our system we switched to operationmode
delta_datashipping. We did (still running production on our primary):

• Cluster in maintainance mode
• unregister secondary with hdbnsutil -sr_unregister
• re-register secondary with hdbnsutil -sr_register --operationMode=delta_datashipping
• After the resync we took the cluster out of maintainance mode

After these steps we have a working system replication and Linux cluster,
but inconsistencies between the running config, global.ini,
and what the cluster thinks about our system.

Let's have a look at the global.ini first. On the primary node we still
have (-sr_unregister didn't change this):

[system_replication]
operation_mode=logreplay

On the secondary we have (changed by -sr_register):

[system_replication]
operation_mode=delta_datashipping

This configuration is reflected in the cluster attributes. Primary
node has logreplay in hana__op_mode. Secondary has delta_datashipping
in there - we had expected to find the same value in both nodes.

The reason for this is in line 2292 in SAPHana:

        # TODO: PRIO 3: check, if using getParameter.py is the best option to analyze the set operationMode
        # DONE: PRIO 3: Should we default to logreplay for SAP HANA >= SPS11 ?
        hanaOut1=$(HANA_CALL --timeout 10 --use-su --cmd "getParameter.py --key=global.ini/system_replication/operation_mode --sapcontrol=1")
        hanaFilter1=$(echo "$hanaOut1" | awk -F/ 'BEGIN {out=0} /^SAPCONTROL-OK: <begin>/ { out=1 } /^SAPCONTROL-OK: <end>/ { out=0 } /=/ {if (out==1) {print $3} }')
        hanaOM=$(echo "$hanaFilter1" | awk -F= '$1=="operation_mode" {print $2}')
        set_hana_attribute ${NODENAME} "$hanaOM" ${ATTR_NAME_HANA_OPERATION_MODE[@]}

Here we read the (stale) config from global.ini. The only command
to find the currently running configuration seems to be

hdbcons -e hdbindexserver 'replication info'

I didn't find an SQL command or any other command.
I have no idea if the output of hdbindexserver is stable
and can be used. I also agree that the situation is pretty unusual
and will be fixed by a short downtime and changes in global.ini.

Anyway, here's a preliminary and untested patch that should help:
[I'll confirm or send an updated patch after we've verified it
on a testcluster or during the next planned downtime]

--- SAPHana.orig        2016-09-06 12:27:10.000000000 +0200
+++ SAPHana     2016-09-06 12:33:20.000000000 +0200
@@ -2289,9 +2289,8 @@
             local hanaOut1=""
             # TODO: PRIO 3: check, if using getParameter.py is the best option to analyze the set OperationMode
             # DONE: PRIO 3: Should we default to logreplay for SAP HANA >= SPS11 ?
-            hanaOut1=$(HANA_CALL --timeout 10 --use-su --cmd "getParameter.py  --key=global.ini/system_replication/operation_mode --sapcontrol=1")
-            hanaFilter1=$(echo "$hanaOut1" | awk -F/ 'BEGIN {out=0} /^SAPCONTROL-OK: <begin>/ { out=1 } /^SAPCONTROL-OK: <end>/ { out=0 } /=/ {if (out==1) {print $3} }')
-            hanaOM=$(echo "$hanaFilter1" | awk -F= '$1=="operation_mode" {print $2}')
+            hanaOut1=$(HANA_CALL --timeout 10 --use-su --cmd "hdbcons -e hdbindexserver 'replication info'")
+            hanaOM=$(echo "$hanaOut1" | awk 'OperationMode { print $4 }')
             set_hana_attribute ${NODENAME} "$hanaOM" ${ATTR_NAME_HANA_OPERATION_MODE[@]}
             super_ocf_log debug "DBG: sync_attr=$sync_attr"
             case "$sync_attr" in

Any comments?

If system replication is runnig with mode "syncmem" and a switchover occurs the sync mode is reset to "sync".

In some cases like the zero-position the SAP HANA SR mapping could not be detected properly.
The "zero-position" is

• cluster base configured
• SAP HANA down on both sides
• resources for SAP HANA added

Now hdbutil does not report the mapping of host-names/sites.

According to the description the default value for PREFER_SITE_TAKEOVER should be "yes":

<parameter name="PREFER_SITE_TAKEOVER" unique="0" required="0">
    <longdesc lang="en">Should cluster/RA prefer to switchover to slave instance instead of restarting master locally? Default="yes"
    no: Do prefer restart locally
    yes: Do prefer takever to remote site
    </longdesc>
    <shortdesc lang="en">Local or site recover preferred?</shortdesc>
    <content type="boolean" default="yes" />

However looking at the code in the saphana_init function no default is specified if OCF_RESKEY_PREFER_SITE_TAKEOVER is not set:
PreferSiteTakeover="$OCF_RESKEY_PREFER_SITE_TAKEOVER"
(for other parameters like "AUTMATED_REGISTER" a default is specified:
AUTOMATED_REGISTER="${OCF_RESKEY_AUTOMATED_REGISTER:-false}")

For example on cluster setups where cman is used the nodenames will always be different from the hostnames if a separate network is used for the cluster communication:

[root@rhel6 ~]# hostname
rhel6
[root@rhel6 ~]# crm_attribute --type crm_config --name cluster-infrastructure -q
cman
[root@rhel6 ~]# crm_node -n
rhel6hb

Hi:

I have a SuSE cluster running on SLES 11 SP4 for SAP. It was built about two years ago. It has all of the latest SLES patches as of April 2017. I currently have a service request (682288682) open with Cisco at the moment. Yes, I said Cisco for they provide my SuSE support. The reason for this SR is that since our Basis team upgraded HANA to 1.00.111.00.1454183496 we are now witnessing new behavior we have not seen before and it is adding an extra step and some confusion for your junior team members during maintenance activities. Let me explain.

In the past on our two node cluster we could take the slave host out operation by running “crm node standby” This would shut the application down on the slave. After whatever work was planned we could perform a “crm node online” on the same host and the server would come back online. This included resyncing the replication. Now however we are getting a different results. After the bringing the slave online we receive startup errors out with the message below.

Failed actions: rsc-hana-TP1_start_0 on mn500d5a209 'unknown error' (1): call=69, status=complete, exit-reason='none', last-rc-change='Mon May 8 09:57:52 2017', queued=0ms, exec=19762ms

The only way to clear this is to re-register the host with the sr_register command telling where to find the master (the other side of the cluster) and then clear the failed action. At this point the cluster starts and resumes replication. I gather we can overcome this by setting the parameter AUTOMATED_REGISTER=true , however that would introduce registration on cluster failovers and we don’t want to do that. SAP also discourages this.

So crux of the issue is this never behaved like this before. Brining a slave host up and down either by putting it in standby mode or stopping and starting openais never use to force us to register the server for the replication when it was previously already a slave. Somehow it understood that this was not necessary.

So a bit more information. We are using delta data shipping between hosts. The nameserver trace does kick out these messages below.

[29160]{-1}[-1/-1] 2017-05-25 08:14:05.599552 e sr_dataaccess DisasterRecoveryProtocol.cpp(01171) : This situation can come up, if the primary has been recovered to an older point in time and a new full data shipping is needed.
[29160]{-1}[-1/-1] 2017-05-25 08:14:05.599564 e sr_dataaccess DisasterRecoveryProtocol.cpp(01172) : If this the case, please execute 'hdbnsutil -sr_register ..' on the secondary to trigger a new full data shipping.

Another thing we see from the SAPHana ofc script is this…

if version "$hdbver" ">=" "1.00.110"; then
newParameter=1

This implies the script takes different actions as before because of our upgrade. I did not dig into it since you folks are already in know. Is this normal behavior now and what can we do to bring back the old behavior? Or is something missing and is a PTF required? I will try to get the corresponding SR number for SuSE from cisco.

Tim

The line 2469 will always result in exit code 0 as the last command in any of the branches before it are assignment of variable so that returns 0 on success.

https://github.com/fmherschel/SAPHanaSR/blob/15e17acc741748dff4f8bf96b410be8e07c298b3/SAPHana/ra/SAPHana#L2469

Removing above line results in proper detection of failed promote. To test that this works properly after removing the line line 2469 it is enough to just comment the takeover command on line 2444 and test that in such case the 'promote' of SAPHana fails.

https://github.com/fmherschel/SAPHanaSR/blob/15e17acc741748dff4f8bf96b410be8e07c298b3/SAPHana/ra/SAPHana#L2444

Or is there any other intention with assignment on line 2469 that I don't see?

Hi Fabian,
I think there is a small syntax error on line 908 of the SAPHana agent.
I have observed a stderr in the /var/log/messages file (picture enclosed) and investigation shows 908 to be rc1, should this be rc=1 ?

Kind regards,
Martin.

The error is due to a "remoteHost" being used instead of "remoteNode" in an get_attributes call.

Dear Fabian, Hi !
I am currently implementing the agent for the first time on SP11. I have noticed a small issue, which may be worth considering. When the agent tries to perform an sr_register command I am seeing a failure (please see screenshot attached). I think this is due to deprecated registration parameter --mode= . See below:

As a temporary measure I had adjusted the agent register command(s) to be SP11 friendly.

However, I only "hardcoded" the additional operation mode parameter. Would it be possible it add functionality within the agent's configurable parameters, to allow someone to set the replication operation mode [delta_datashipping | logreplay ] ?

Many thanks, regards,
Martin. (Dell).

in version 0.152.17 the function contains the following commands which hardcode the SID to 'LNX':

cp /hana/shared/LNX/global/hdb/custom/config/global.ini /hana/shared/LNX/global/hdb/custom/config/global.ini.$(date +"%s")
cp /hana/shared/LNX/global/hdb/custom/config/nameserver.ini /hana/shared/LNX/global/hdb/custom/config/nameserver.ini.$(date +"%s")

'LNX' should be replaced with '$SID'

We often use "crm_mon -rRA" (--inactive --show-detail --show-node-attributes)
to use the cluster status (master/slave on which host, sync status,
HANA status). Until now we use HANA Studio to view the detailed SR status,
because SFAIL can have two meanings:

1. Sync ist still running, but will finish eventually
2. Sync has finally failed (needs full-sync etc.)

We might use systemReplicationStatus.py on the command line as well,
but will need both crm_mon and systemReplicationStatus.py for a complete
status.

I've tried to get a more detailed status in one line and think about
adding it to "crm_mon -A". Right now we have the attribute
hana_${sid}_sync_state (SOK, SFAIL, UNKNOWN) for the secondary and PRIM for
the primary HANA DB. What would be a useful, more detailed status
(attribute like hana_${sid}_sync_detail)?

• "all services in sync": When systemReplicationStatus.py reports all
  is well and sync_state will be "SOK"

• [<service>:<Replication Status>(<Replication Status Details),?]+
  For each service where the Replication Status is not "ACTIVE"
  we'll display it and the Status Details. Examples:

  nameserver:ERROR(communication channel closed)
  indexserver:SYNCING(missing log...)

  The states of the services will be concatenated with ','.

  Now we have a short summary of all "interesting" services which
  are not in sync (yet).

Right now I check the detailed state with a standalone awk-script
(around 10 lines...), but if you think that would be a useful addition
I'll try to add it to the cluster attributes and provide a patch for
SAPHana.

In the Debug log of the SAPHanaTopology Resource Agent you can se that it is unable to determine the correct value for the hanaRemoteHost parameter in environments where the nodename is not identical to the hostname:

Jun 10 17:34:06 node2 SAPHanaTopology(rsc_SAPHanaTopology_HDB_HDB00)[11188]: INFO: DEC: site=DC2, mode=primary, MAPPING=node1, hanaRemoteHost=

There is a spelling mistake for the return code "OCF_SUCCESS" in multiple places of the SAPHANA resource agent:

# grep SUCCSESS SAPHana
SAPHana:                    rc=$OCF_SUCCSESS
SAPHana:                    # TODO: PRIO3: OCF_SUCCSESS, OCF_NOT_RUNNING or OCF_ERR_xxxx ?
SAPHana:                    rc=$OCF_SUCCSESS
SAPHana:               rc=$OCF_SUCCSESS
SAPHana:       rc=$OCF_SUCCSESS

It should be OCF_SUCCESS (without the "S" before the "E") instead :)

Hi Fabian,
I have noticed that the test script show_SAPHanaSR_attributes is no longer returning any output.
Picture included. This is on SUSE 11 SP3, HANA 102 (though I have seen same behaviour on SP9).

Regards,
Martin.

We have two node cluster on RHEL7.2 pacemaker cluster. DB monitoring is failing which is leading to crash of Node B. Though backup is happening on Node A. if we run #time pcs resource debug-monitor rsc_Topology_HDB_HDB00, comes back in 48 to 120 seconds sometime.

-bash-4.2# timeout 60 su - hb1adm -c 'true; /usr/sap/HB1/HDB00/HDBSettings.sh landscapeHostConfiguration.py'
| Host | Host | Host | Failover | Remove | Storage | Storage | Failover | Failover | NameServer | NameServer | IndexServer | IndexServer | Host | Host | Worker | Worker |
| | Active | Status | Status | Status | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual | Config | Actual |

Session terminated, killing shell... ...killed.
-bash-4.2# echo $?
124
-bash-4.2#

Here is my pacemaker cluster configuration:

cat /var/lib/pacemaker/cib/cib.xml

Hello,

We have implemented 4 tier HANA system replication in HANA DB V2.03* where 2 sites are in Primary DC and we are using Pacemaker cluster tool for HA automatic Failover and rest 2 are in DR center.
We have implemented Pure chain replication -- A-->B-->C-->D and even for some DB Cluster -Multi-tier replication .
Now we are facing issue in either of replication scenario-

1. when sometimes 4 tier sites are not in sync or active , although sync_state in crm_mon is SOK - we are seeing Failover from Site-A to Site B does not happen.

2. Need to know, entire output of python systemReplication script status is important for this Failover, although logically PCS should only understand 2 sites in HA for taking decision on Failover?

-Anand

On a setup where hdbnsutil -sr_state returns the following:

# su - rstadm -c "hdbnsutil -sr_state --sapcontrol=1"
checking for active or inactive nameserver ...
SAPCONTROL-OK: <begin>
mode=primary
site id=1
site name=RST
mapping/node1=RST/node1
mapping/node1=RSTDR/node2

SAPHanaTopology is not able to determine the hanaRemoteHost, because it cant figure out the mapping of the remote HANA host.

Trying to run the command used by SAPHanaTopology manually gives the following result:

# su - rstadm -c "hdbnsutil -sr_state --sapcontrol=1"| awk -F[=/] '$1 ~ "mapping" && $3 !~ site { print $4 }' site=RST
# su - rstadm -c "hdbnsutil -sr_state --sapcontrol=1"| awk -F[=/] '$1 ~ "mapping" && $3 !~ site { print $4 }' site=RSTDR
node1

The problem seems to be that the site name of the primary site, "RST", is a substring of the sitename for the secondary site, "RSTDR", which causes the awk match to fail.

The secure store users in the SAPHana resource agent are currently hardcoded. Would be nice if this was configurable.

Hi Fabian,

I have noticed that in place below (L2160) SAPHana returns only $OCF_SUCCESS when the it is probed. This seems to not be and issue during cluster startup, but it leads to "not optimal" behaviour when the probe is needed during normal cluster operation. During normal cluster operation the probe can happen when the resource is "cleaned up" (pcs resource cleanup). In such case the node that was running the 'Master' of SAPHana will first detect $OCF_SUCCESS (which means 'Slave') and on next monitor it detects correctly 'Master'. However when this is done during the time that resource is in 'unmanaged' state, then cluster will consider the $OCF_SUCCESS return code on probe as 'Failed Master' and as soon as the resource is again managed it will do 'restart' to resolve this - which is not good. In normal scenario (as managed resource) when cleanup is done I can see that after probe the cluster will do actually 'promote' operation to get from reported 'Slave' back to 'Master' - that is pretty quick, but it seems unnecessary if the resource was detected as Master in first place.

I believe that code below can be just slightly adjusted to allow proper detection of Master state during probe. Just wanted to ask what was the motivation for returning just $OCF_SUCCESS when doing probe instead of more advanced detection of state?
Is it a concern that 'probe' will change any node attributes?

        2 | 3 | 4 ) # WARN, INFO or OK
            if ocf_is_probe; then
                rc=$OCF_SUCCESS
            else
                LPTloc=$(date '+%s')
                lpa_set_lpt $LPTloc $NODENAME
                lpa_push_lpt $LPTloc
                if [ "$promoted" -eq 1 ]; then
                    set_hana_attribute "$NODENAME" "PRIM" ${ATTR_NAME_HANA_SYNC_STATUS[@]}
                    rc=$OCF_RUNNING_MASTER
                else
                    if [ "$init_attribute" -eq 1 ]; then
                        set_hana_attribute ${NODENAME} "DEMOTED" ${ATTR_NAME_HANA_CLONE_STATE[@]}
                        rc=$OCF_RUNNING_MASTER
                    else
                        rc=$OCF_SUCCESS
                    fi
fi

Possible change that only takes care of first part

...
        2 | 3 | 4 ) # WARN, INFO or OK
                LPTloc=$(date '+%s')
                if ! ocf_is_probe; then
                    lpa_set_lpt $LPTloc $NODENAME
                    lpa_push_lpt $LPTloc
                fi
                if [ "$promoted" -eq 1 ]; then
                    if ! ocf_is_probe; then
                        set_hana_attribute "$NODENAME" "PRIM" ${ATTR_NAME_HANA_SYNC_STATUS[@]}
                    fi
                    rc=$OCF_RUNNING_MASTER
                else
                    if [ "$init_attribute" -eq 1 ]; then
                        if ! ocf_is_probe; then
                            set_hana_attribute ${NODENAME} "DEMOTED" ${ATTR_NAME_HANA_CLONE_STATE[@]}
                        fi
                        rc=$OCF_RUNNING_MASTER
                    else
                        rc=$OCF_SUCCESS
                    fi
                    if ! ocf_is_probe; then
                        ... rest of the code ...
                    fi
...

If the above makes sense I can prepare a pull request for the changes that should improved the detection of Master when doing probe in scenario where $lss (landscape status) reports 2/3/4. Please let me know what you think about this approach.

Hi Guys,

We have noticed that some of our clusters are failing over because we have 4 consecutive time out on the command hdbnsutil -sr_stateConfiguration. Do you know why that would happen? Normally this runs in about 4 seconds but once it has failed 4 times all hell breaks loose in the cluster. I am running an older version of the RAs 152.17 however, the latest code (.22) only seem to add a check for a 124 timeout on the hana call which doesn't explain why this would happen.

Thanks
Scott

crm_node -l output is different for RHEL6 (cman) and SLES11 (openais/corosync):

SLES11:
sles11n1:~ # crm_attribute --type crm_config --name cluster-infrastructure -q
classic openais (with plugin)
sles11n1:~ # crm_node -l
123456789 sles11n1 member
123456790 sles11n2 member

RHEL6:
[root@rhel6 ~]# crm_attribute --type crm_config --name cluster-infrastructure -q
cman
[root@rhel6 ~]# crm_node -l
rhel6n1 rhel6n2