implements MessageHandler handle, more specifically Indirect-Ping message handler

Subject
all members who work on this project

Proposal Detail
You may get hint about implementation details from this youtube links

Branch, Merge, Commit, PR, etc
We need some contribution guide docs.

Detail
Implement awareness functions

implementation reset function in member_map

I'm interested in suspicion mechanism so I'm now investigating more about this, so before move on to "writing codes", I think it is better to write down docs more about suspicion mech.

• seq should be unique, so every different message should have different seq, as a result register different callback function
• In the case of k indirect-ping, k different SyncSend returns k response (if all request successfully returned). so the client of MessageEndpoint.SyncSend should ignore k-1 response. because we only interested in whether final target of indirect-ping successfully responsed.

Detail
Initially Implement the highest abstraction object of swim protocol according to docs

Reference
https://github.com/DE-labtory/swim/edit/develop/docs/Docs.md

Before move on Lifeguard, why Suspicion mechanism

SWIM added this suspicion mechanism to solve flapping problem and to make protocol more robust to slow processes

Flapping problem?

• Healthy node being marked as failed
• Logs show it was never actually unhealthy but other nodes think this node as failed

However, although suspicion mechanism was shown to address slow nodes, still vulnerable to slow nodes because:

• before suspected node's alive message sent back to node which originating the probe or suspicion, alive message sent from suspicion could be timed out

Lifeguard comes to rescue

Absence of expecting message can be a signal that the local member(self) may be slow node (slow processing messages, slow network): "No messages... maybe I'm in trouble?!"

Lifeguard Components

Lifeguard plays in three situations

L1: Dynamic Fault Detector Timeouts: "Self-Awareness"

Dynamically adjust the fault detector timeouts. Starts timeouts low. and increase in response to absence of replies. If the acks are not sent back from probe or indirect probe, it judges it is local problem
So local node (node itself) vary two things:

• Probe timeout: How long long probed node has to respond
• Probe Interval: Time between successive probes

Introduce Node Self-Awareness(NSA) conter

Higher the value the worse we think we're doing.

ProbeTimeout = BaseTimeout * (NSA+1)
ProbeInterval = BaseInterval * (NSA+1)

• Node Self-Awareness
  • Failed probe (no Ack): +1
  • Probe with missed Nack: +1
  • Refute suspicion about self: +1
  • Successful probe (get Ack): -1
  • Max NSA = 8

Nack for more information

Nack is not part of SWIM protocol but since we are now interested in discovering information about absence of messages, hashicorp thought that it was good idea to add nack messages. Then what is for nack?:
Without the nack if sender doesn't hear back from an indirect ping of target, it doesn't know the cause of problem is with A, B, C or D. But for the nack we can get hint with this problem. For example, if A receive Nack from B, C we can tell that A, B, C can talk each other and probabily D isolated. On the other hand if A don't receive nack back, A maybe isolated

L2: Dynamic Suspicion Timeouts: "Dogpile"

Its principle is similar with L1

• node start with a high suspicion timeout
• node low timeout as receive more suspect messages (This means as node receive more suspect messages we can presume that it is healthy, on the other hand if we sent back less messages, this node may be slow)

L3: More Timely Refutation: "Buddy System"

The motivation for L3 is the fact that only the suspected node can refute suspicion, in other words, only suspicion node can increment incarnation number.
For example, in the network there are many "suspect" message, but to refute those suspicion, only suspect node can refute its own suspicion by increment incarnation number(because higher incarnation number message can override other messages). So if node knows about suspect, then give probe priority to suspect: "Let him(suspect) know that he is suspicion"

Detail
Implement Infection-style dissemination mechanism (ref https://github.com/DE-labtory/swim/blob/develop/docs/Docs.md)

각 그룹의 멤버 M_i는 버퍼를 가지고 있는데

• 버퍼에는 최근에 발생한 membership update 정보들을 담고 있다.
• 그리고 버퍼에 자신(M_i)이 주위 멤버들에게 piggyback한 횟수를 담고 있는 local count의 정보를 담는다. 그리고 이 local count로 자신이 다음에 누구에게 piggyback 할지를 결정할 수 있다.
• 버퍼의 사이즈가 한 멤버가 ping (또는 ack)을 piggyback 할 수 있는 최대 횟수보다 크다면(충분히 크다면) 다음으로 piggyback 할 멤버는 local count가 작은 멤버로 한다.

그리고 두 개의 그룹 멤버 리스트를 가지고 있다.

• 하나에는 아직 그룹에서 “failed” 되지 않은 멤버들을 담고 있고
• 다른 하나에는 현재 “failed” 된 멤버들을 담고 있다.

• language
• license
• coverage
• travis

going to work with @hea9549

Subject
udp

Detail
Implement packet transport for UDPTransport interface.

Reference
https://github.com/hashicorp/memberlist/blob/master/net_transport.go

Detail
implement handle ack function

Subject
conf using viper

Reference
https://github.com/it-chain/engine/tree/develop/conf

Introduce Node Self-Awareness(NSA) counter

self-awareness를 정량적으로 나타내고 그에 따라서 timeout, prove interval을 증가시키기 위해 NSA counter를 사용한다. NSA counter가 높을 수록 자기 자신 노드가 느리다고 생각하는 것이다.

ProbeTimeout = BaseTimeout * (NSA+1)
ProbeInterval = BaseInterval * (NSA+1)

위와 같이 timeout과 interval을 조정한다. 그리고 NSA counter를 바꾸는 경우는 다음과 같다.

• Failed probe (no Ack): +1
• Probe with missed Nack: +1
• Refute suspicion about self: +1
• Successful probe (get Ack): -1

그리고 최대 NSA counter 값은 hashicorp에서는 8을 사용하고 있다.

Awareness should be updated whenever there's probe or refute. So Awareness is not relevant to MessageEndpoint but to SWIM

define suspicion model. This struct manages suspect timer

• each suspect has its own timer
• timer could be accelerate according to the Lifeguard principle "Dynamic suspicion timeouts"

Where
member_map.go

Bug detail
SelectKRandomMemberID needs to return []Member instead of []MemberID.
Because we need to know address of the member to send msg.

Fix plan
Change return type to []Member

Subject
member_map

Proposal Detail
We tried to pick a K member in a round robin fashion, but we decide to modify it by simply selecting a K member from a memberMap.

Subject
write here

Detail
implement swim join function

assignee
@ChaeByunghoon

Global option

• swim --config <config> or -c <config> [subcommand]
• swim --debug or -d [subcommand]

Subcommand will be

• swim start [--address <ip:port>]
• swim add member [--address <ip:port>]

Detail
Implement func confirm in member_map.go

Detail
Implement function handling piggyback messages about me.

1. When i receive suspect message, increment incarnation and push alive message to pkbstore.

Subject
Anybody

Proposal Detail
I think travis goimports scope should restricted to source files not vendor

I think we need awareness model which is related with L1 (self-awareness).

awareness model is metric for tracking health of the local-node (myself)

reference

https://github.com/hashicorp/memberlist/blob/master/awareness.go

Detail
implement Len, Push, Get, IsEmpty function

Reference
#33

implements selectKRandomMember

reference
https://github.com/DE-labtory/swim/blob/develop/member_map.go

Where
CLA hub

Bug detail
When a PR is generated, CLA hub doesn't work although signed to CLA hub.

Fix plan
fix it plz

Reference
.

Subject
All members who code udp part

Proposal Detail
How about writing udp test using tesseract. udp transport test may needs docker test.

message_endpoint name is referenced from: https://english.stackexchange.com/questions/369828/what-is-the-generic-term-for-sender-and-receiver
select endpoint to represent both sender and receiver.

MessageEndpoint basically do receiving packet and determine which logic executed based on the packet. Additionally MessageEndpoint call MemberDelgator to update member status, and encrypt data

Subject
udp

Detail
net_transport config

Reference

• https://github.com/hashicorp/memberlist/blob/master/net_transport.go 26 line

Subject
pbkstore, pb-message

Detail
We need to pass multiple piggyback messages in a single message.

1. Converts piggyback in message to list type.
2. Send as much piggyback as you can send at once based on max udp byte size.

The swim paper needs to be documented.

• issue template
• PR template

Subject
All of members

Proposal Detail
I think in hashicorp's memberlist struct Memberlist do too much things, so I think it is better to separate responsibility. My suggestion is.. I want to separate receiving message part and sending message part. So the structure is

• Spaceship (this is Memberlist in hashicorp, and temporarily named it :), and network as galaxy..)
  • MessageTransmitter
  • MessageReceiver
  • ...

related #22

I've summarized hashicorp's memberlist. I've skipped a lot but this is basic logic and abstracted logic flow of probing node in failure detection

probeNode

probeNode handles a single round of failure checking on a node

1. Awareness score에 따라서 probe interval을 조정한다.
2. Ping 메세지와 ack handler, nack handler 준비한다
3. 현재 노드 상태가 alive라면 ping 메세지를 보낸다
4. alive가 아니라면 ping메세지와 함께 suspect 메세지를 준비한다.
5. 메세지를 보낸다. 성공적으로 돌아왔다면 함수를 종료한다.
6. 타임아웃에 걸렸을 때는 이제 k개의 랜덤노드를 뽑는다.
7. k 랜덤 노드들에 대해서 indirect-ping-req를 보낸다.
8. (config에서 DisableTcpPings 옵션이 꺼져있다면) 타임아웃에 걸렸던 노드와 tcp 연결도 시도해본다.
9. 돌아온 결과를 토대로 awareness score를 조정한다.
10. direct ping, indirect ping이 모두 실패하였으므로 해당노드를 suspect한다.
11. 만약 suspicion에 등록되어있는 suspect라면 함수 종료
12. 등록되어있지 않다면 suspicion timeout callback function과 함께 새로운 suspicion을 등록한다.

implements MessageHandler handle, more specifically Ack message handler

implements MessageHandler handle, more specifically Ping message handler

aliveNode

aliveNode is invoked by the network layer when we get a message about a live node

1. 만약에 memberMap에 해당 멤버가 없다면 추가
2. 다른 사람에 해당하는 메세지이고 메세지의 incarnation number가 작거나 같다면 리턴
3. 다른 사람에 해당하는 메세지이고 크다면 업데이트(오버라이드)
4. 자신에 해당하는 메세지이고 메세지의 incarnation number가 작으면 리턴
5. 자신에 해당하는 메세지이고 메세지의 incar가 같으면 리턴 or refute

suspectNode

suspectNode is invoke by the network layer when we get a message about a suspect node

1. incar가 작으면 리턴
2. 자기 자신에게 해당하면 refute
3. 다른 사람이면
   1. 자신의 리스트에 있으면 suspect timeout 조정
   2. 자신의 리스트에 없으면 리스트에 추가

deadNode

deadNode is invoked by the network layer when we get a message about a dead node

1. incar 넘버가 작으면 리턴
2. 자기 자신이면 refute
3. 다른 사람이면 업데이트

MemberState is used to manage our state view of another member.
I think MemberState should include:

• member info
• status
• incarnation number
• timestamp