Comments (5)
alexellis
commented on June 1, 2024
1
Hi @danielmichaels, thanks for using inlets.
I have absolutely no issues with K3s or KinD and inlets-operator and run them on a regular basis myself.
The problem in this issue is well described and caused by IPVS. IPVS is not a default in Kubernetes, to enable it you must install a special network driver or change the kubelet flags.
If you're doing neither of these things then I'd ask you to remove your comment and raise your own issue.
inlets-operator creates VMs for tunnel servers
- Check the VM was created and that the inlets-pro service is running there?
- If you see a public IP, try connecting to it:
curl -k https://165.232.173.183:8123/.well-known/ca.crt- if that fails from your own machine, then the issue is with the VM itself.
For IPVS users use: https://inlets.dev/blog/2021/07/08/short-lived-clusters.html
Whilst I don't want to hijack this issue about IPVS, let me show you that inlets-operator is working with K3s. Please follow the steps I left for you above ( 1 and 2 )
multipass launch --cpus 2 --mem 4G -d 30G
multipass exec light-cephalopod /bin/bash
curl -sLS https://get.arkade.dev | sudo sh
arkade get k3sup
sudo mv /home/ubuntu/.arkade/bin/k3sup /usr/local/bin/
k3sup install --local --k3s-channel latest
# create do.txt
mkdir -p .inlets
# create ./inlets/LICENSE
arkade install inlets-operator \
--region lon1 \
--provider digitalocean \
--token-file ./do.txt
kubectl get tunnels -A -o wide -w
NAMESPACE NAME SERVICE TUNNEL HOSTSTATUS HOSTIP
HOSTID
kube-system traefik-tunnel traefik provisioning
322330692
kube-system traefik-tunnel traefik active
206.189.117.93 322330692
curl -i http://206.189.117.93:80
# Traefik answered
curl -i -k http://206.189.117.93:443
# Traefik answered
curl -k https://206.189.117.93:8123/.well-known/ca.crt
-----BEGIN CERTIFICATE-----
MIIDvDCCAqSgAwIBAgIRAL4BGx/MwtCUdcn4r2a8vZcwDQYJKoZIhvcNAQELBQAw
aDELMAkGA1UEBhMCR0IxFTATBgNVBAcTDFBldGVyYm9yb3VnaDEZMBcGA1UECRMQ
...
-----END CERTIFICATE-----Alex
from inlets-operator.
alexellis
commented on June 1, 2024
In addition to just using the helm chart for inlets-pro and creating the tunnel server with inlestctl (https://inlets.dev/blog/2021/07/08/short-lived-clusters.html)
...there is another workaround. If you edit the LoadBalancer service and remove the public IP address from the service using kubectl edit, IPVS won't get in the way trying to re-route the traffic to the wrong place.
You can still get the public IP with the tunnels CRD:
kubectl get tunnels -A
NAMESPACE NAME SERVICE TUNNEL HOSTSTATUS HOSTIP HOSTID
kube-system traefik-tunnel traefik active 165.22.119.191 321884435I still haven't found a way to prevent / override IPVS from doing the wrong thing.
The reason that port 8123 has connection refused is that IPVS takes over and sends traffic destined to the public IP on port 8123 to the endpoint IP in Kubernetes backed by the service within the cluster.
from inlets-operator.
danielmichaels
commented on June 1, 2024
Chiming in here,
I am experiencing this issue.
When I follow this tutorial (i.e. using Kind) the ingress-nginx-controller-tunnel-client-xxxx pod will successfully make a connection.
However, when I use k3s (single, or multi node) I get the same connection refused error as described above.
Output from ingress-nginx-controller-tunnel-client-xxxx when using Kind:
2022/10/22 00:28:28 Licensed to: [email protected] (Gumroad subscription) │
│ 2022/10/22 00:28:28 Upstream server: ingress-nginx-controller, for ports: 80, 443 │
│ inlets-pro TCP client. Copyright OpenFaaS Ltd 2022 │
│ time="2022/10/22 00:28:30" level=info msg="Connecting to proxy" url="wss://157.245.146.81:8123/connect" │
│ time="2022/10/22 00:28:30" level=info msg="Connection established" client_id=fbc1733076984a88a018ec00f1e4010b Output when using k3s:
│ 2022/10/22 00:43:05 Licensed to: [email protected] (Gumroad subscription) │
│ 2022/10/22 00:43:05 Upstream server: ingress-nginx-controller, for ports: 80, 443 │
│ Error: unable to download CA from remote inlets server for auto-tls: Get "https://165.232.173.183:8123/.well-known/ca.crt": dial tcp 165.232.173.183:8123: connect: connection refused I don't know if this makes a difference, so I'll add it anyway but when I inspect the services, kind shows this:
# external-ip only shows digitalocean ip
z ❯ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.96.26.50 157.245.146.81,157.245.146.81 80:31785/TCP,443:32284/TCP 25m
ingress-nginx-controller-admission ClusterIP 10.96.44.137 <none> 443/TCP 25m
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 28m
and k3s shows this:
# external-ip shows local ip and digitalocean ip
z ❯ k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 10m
ingress-nginx-controller-admission ClusterIP 10.43.63.142 <none> 443/TCP 9m35s
ingress-nginx-controller LoadBalancer 10.43.155.203 192.168.20.87,165.232.173.183 80:31693/TCP,443:30926/TCP 9m35s
Deployment notes:
kind - I have followed the tutorial without issue.
k3s:
k3supto provision a single node- my own terraform/ansible to build single and multinode k3s
- nginx-ingress (traefik disabled)
- using both
inlets-operatorand manually usinginletsctl(following this guide
Both kind and k3s are on digitalocean, and I've deployed across sgp1 and lon1.
I am a pro subscriber.
from inlets-operator.
danielmichaels
commented on June 1, 2024
Thank you for your prompt reply.
I can confirm that when running using Multipass or Kind everything works as expected.
It still fails to connect when I provision everything inside my proxmox cluster on my LAN. The issue must be related to how I am provisioning it or something other than inlets.
Keep up the good work! 👍
from inlets-operator.
alexellis
commented on June 1, 2024
Daniel can you try using the inlets-tcp-server chart instead of the operator? Create your tunnel server using inletsctl
from inlets-operator.
Related Issues (20)
- Add new Hetnzer provisioner to operator HOT 2
- Change to inlets-pro HOT 3
- tunnel-client pod won't start, "exec: "inlets-pro": executable file not found in $PATH" HOT 5
- GCE provisioner does not show when the VM fails to be created HOT 10
- DigitalOcean has deleted the Ubuntu 16.04 image HOT 1
- Update OS images to Ubuntu 18.04 for provisioners HOT 9
- Support custom plan for provisioner HOT 1
- Update provider pricing grid HOT 2
- Format Error: JWT Token for license-file HOT 4
- Getting 401 Unable to Authenticate from Digital Ocean HOT 2
- Linode provider with Arkade ingress-nginx - label too long HOT 2
- Update codegen and generate-crd script for Go modules
- Add plan to Inlets Operator Helm chart HOT 1
- Update build status image to use GitHub Actions HOT 2
- bugfix: Docs for installation overlook the access-key
- question: is the inletsProLicense field of the helm chart different than the lic key received in email HOT 6
- AWS security group lacks inbound rule for custom TCP port HOT 4
- digitalocean image ubuntu-18-04-x64 is no longer available HOT 5
- Is there a free version of inlets? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from inlets-operator.