Hi Skupper community!
Not sure that it's the correct place to open issues but you'll let me know if there's a better place...
I'm trying to setup Skupper on a multi-cluster application that is using Infinispan with cross-site replication. I basically have 2 AWS OpenShift clusters (US and EU). I have successfully deployed my app using LoadBalancer
Service types and configuring the JGroups discovery protocol (TCPPING
) accordingly. The services have been annotated with service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
.
Now I'm trying to replace this part with Services that are handled by Skupper VAN and being annotated with skupper.io/proxy: tcp
. To make sure I understand everything, I’ve deployed the tcp-echo
sample and everything is OK. However I cannot succeed making JGroups TCPPING
work with Skupper …
Looking at my configuration everything seems to work fine :
- Proxy services are created on both clusters
- Proxy pods are created on both clusters,
- Routers are reconfigured with new routes.
- No error messages in the logs
On the Infinispans pods, I can see a lot of TCP warnings like below but I used to get the same warnings when using AWS LBs.
0m�[33m10:07:00,384 WARN [org.jgroups.protocols.TCP] (TcpServer.Acceptor[7900]-2,relay-global,_infinispan-0-31537:SITE2) JGRP000006: failed accepting connection from peer: java.io.EOFException
at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202)
at org.jgroups.blocks.cs.TcpConnection.readPeerAddress(TcpConnection.java:245)
at org.jgroups.blocks.cs.TcpConnection.<init>(TcpConnection.java:53)
at org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:126)
at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:111)
at java.base/java.lang.Thread.run(Thread.java:834)
The only thing that I can see as warning is in the skipper-router
pods where there’s this Link blocked with zero credit
message :
2020-01-10 10:04:35.183580 +0000 SERVER (info) [C28] Accepted connection to 0.0.0.0:5671 from 10.128.2.108:41272
2020-01-10 10:04:35.191751 +0000 ROUTER (info) [C28] Connection Opened: dir=in host=10.128.2.108:41272 vhost= encrypted=TLSv1.2 auth=EXTERNAL user=CN=skupper-messaging container_id=infinispan-site2-skupper-proxy-547b9bd84f-pdfxm_tcp:7900=>amqp:infinispan-site2-skupper props=
2020-01-10 10:04:35.191835 +0000 ROUTER (info) [C28][L121] Link attached: dir=in source={<none> expire:sess} target={infinispan-site2-skupper expire:sess}
2020-01-10 10:04:35.191885 +0000 ROUTER (info) [C28][L122] Link attached: dir=out source={<dynamic> expire:sess} target={<none> expire:sess}
2020-01-10 10:04:41.122874 +0000 ROUTER (info) [C26][L113] Link blocked with zero credit for 16 seconds
Communication between 2 clusters using this bridge seems to be OK as I have a socket resumed
pending message into the logs of proxy on the cluster 1:
2020-01-10T10:04:35.175Z icproxy info [infinispan-site2-skupper-proxy-547b9bd84f-pdfxm_tcp:7900=>amqp:infinispan-site2-skupper] socket accepted ::ffff:10.131.0.175:60455
2020-01-10T10:04:35.177Z icproxy info client tunnel ::ffff:10.131.0.175:60455 created, sender=18d81bd6-0d27-ea49-b605-3e001dcb1407 receiver=c2652f20-5a5d-c343-b8e3-7216fd00f947
2020-01-10T10:04:35.284Z icproxy info [infinispan-site2-skupper-proxy-547b9bd84f-pdfxm_tcp:7900=>amqp:infinispan-site2-skupper] socket resumed ::ffff:10.131.0.175:60455
And some kind of corresponding message into the logs of service proxy pod on the cluster 2:
2020-01-10T10:04:35.238Z icproxy info [infinispan-site2-skupper-proxy-86f6466d48-bgzjf_amqp:infinispan-site2-skupper=>tcp://10.131.0.189:7900] receiver attached
2020-01-10T10:04:35.241Z icproxy info [infinispan-site2-skupper-proxy-86f6466d48-bgzjf_amqp:infinispan-site2-skupper=>tcp://10.131.0.189:7900] socket connected to 10.131.0.189:7900
2020-01-10T10:04:35.241Z icproxy info server tunnel created for socket 10.128.2.147:50802, receiver 18d81bd6-0d27-ea49-b605-3e001dcb1407@infinispan-site2-skupper-proxy-547b9bd84f-pdfxm_tcp:7900=>amqp:infinispan-site2-skupper
2020-01-10T10:04:35.331Z icproxy info server tunnel for socket 10.128.2.147:50802, sender 57bc0197-db84-d44e-bf30-2b467347724f
I’ve tried to setup another HTTP proxy for another service and everything works fine … Seems my problem only related to TCP communication …
As I’m new to Skupper and QPID I don’t know where to look next … Could you please provide me some hints or help on how to investigate further on this setup ? I really like to be able to run a full demo for both communication style TCP and HTTP …
Many thanks !