patte / fly-tailscale-exit Goto Github PK

View Code? Open in Web Editor NEW

1.5K 1.5K 86.0 81 KB

Run a VPN with global exit nodes with fly.io, tailscale and github!

Home Page: https://news.ycombinator.com/item?id=36064305

Dockerfile 53.38% Shell 46.62%

exitnode flyio tailscale vpn

fly-tailscale-exit's Introduction

Hi, I'm patte ✌️

I'm a full stack software engineer with a background in networking. Mission-driven, curious and highly committed.

I'm proficient in: 📚 Typescript on NodeJS, 🌀 GraphQL (Apollo, Strawberry, Hasura), 🐘 PostgreSQL, 🚀 Redis, ⚛️ React with Next.js, 🎨 CSS, 🐳 Docker, 📈 Grafana, ☁️ Hosting on Fly.io and the big ☁️, IP networking, wireguard, linux

experienced with: 🐍 Python, 📊 Prometheus, 🔍 Meilisearch, 🐦 Swift (iOS)

hyping: 🦀 Rust

Currently looking for a job! Preferably remote or hybrid in Berlin. Contact me via email 📫.

fly-tailscale-exit's People

Stargazers

Watchers

fly-tailscale-exit's Issues

Cannot use the exit node

Is there a way to debug when after deploying everything and approving everything and then enabling:

sudo tailscale up --exit-node=fly-syd --exit-node-allow-lan-access=true

It's not possible to access the internet. All packets get dropped. ping 8.8.8.8 doesn't work.

My tailscale network is up and running and I'm able to connect to other tailscale machines, but nothing gets routed over the exit node.

Furthermore I notice that the fly-syd doesn't respond to ICMP ping. I guess you have to use tailscale ping fly-syd, which does work, and the logs on the fly dashboard seem to indicate things are fine.

$10 if running Proxy / VPN on Fly

Fly.io's free tier (160GB + 140GB in other regions) isn't meant for use by proxies.

Ref: https://community.fly.io/t/4896

Failed due to unhealthy allocations

I face this issue when deploying. How can I fix?

1 desired, 1 placed, 0 healthy, 1 unhealthy
v2 failed - Failed due to unhealthy allocations
***v2 failed - Failed due to unhealthy allocations and deploying as v3

Checking log there is an error

nrt [info] Running: `/app/start.sh` as root
nrt [info] Error: UnhandledIoError(Os { code: 13, kind: PermissionDenied, message: "Permission denied" })

Question: Deployed two machines in one app?

Thank you for your project, it has been of great use to me.
But I have a confusion, I am not sure if it is a problem with my operation or the script itself.
When I follow the process to step 10 flyctl deploy, the platform shows two machines at the same time. This is shown in the image below:

I made sure I didn't perform any additional operations. I checked the project's scripts and didn't find the problem either. Is there a way to start only one machine? Two machines in the same area seems a bit redundant.

Are direct connections working?

Thanks for this guide!

However, I'm not sure what I'm doing wrong here, but I don't seem to be able to get a direct connections to the nodes on Fly.
i.e. tailscale status and tailscale ping shows that connections are going through a DERP relay:

$ /Applications/Tailscale.app/Contents/MacOS/Tailscale status
…
100.125.56.76   fly-fra      chris@       linux   active; exit node; relay "fra", tx 3185 rx 5303

$ /Applications/Tailscale.app/Contents/MacOS/Tailscale ping fly-fra
pong from fly-fra (100.125.56.76) via DERP(fra) in 65ms
pong from fly-fra (100.125.56.76) via DERP(fra) in 57ms
…
2022/03/07 22:09:24 direct connection not established

The same happened when launching the Fly app in fra, ams, or lhr.

Which seems unexpected, as the Fly config asks for UDP port 41641 to be open, which is normally all that's needed to establish a direct connection. For example, on another exit node I have:

$ /Applications/Tailscale.app/Contents/MacOS/Tailscale status
…
100.78.27.79    xyz          chris@       linux   active; offers exit node; direct 89.14.247.123:41641, tx 9001 rx 9002

Are you folks using this setup seeing Tailscale able to make direct connections to the Fly app? Am I missing something?

Failed when running deploy

I tried with flyctl deploy after cloning to local, and it returned the error message:

==> Validating app configuration
--> Validating app configuration done
Services
UDP 41641 ⇢ 41641
Remote builder fly-builder-spring-bird-602 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon   75.2kB
[+] Building 435.9s (4/14)
 => [internal] load remote build context                                                                                                                                                                                         0.0s
 => copy /context /                                                                                                                                                                                                              0.1s
 => [internal] load metadata for docker.io/library/alpine:latest                                                                                                                                                                 0.7s
 => ERROR [tailscale 1/4] FROM docker.io/library/alpine:latest@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                                                                                         435.2s
 => => resolve docker.io/library/alpine:latest@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad                                                                                                           0.0s
 => => sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad 1.64kB / 1.64kB                                                                                                                                   0.0s
 => => sha256:1304f174557314a7ed9eddb4eab12fed12cb0cd9809e4c28f29af86979a3c870 528B / 528B                                                                                                                                       0.0s
 => => sha256:9c6f0724472873bb50a2ae67a9e7adcb57673a183cea8b06eb778dca859181b5 1.47kB / 1.47kB                                                                                                                                   0.0s
 => => sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49 0B / 2.81MB                                                                                                                                     435.2s
------
 > [tailscale 1/4] FROM docker.io/library/alpine:latest@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad:
------
Error error building: failed commit on ref "layer-sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49": "layer-sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49" failed size validation: 0 != 2806054: failed precondition

Is that a issue with the dockerfile, or with the remote worker?

Node is set up but traffic doesn't pass through the exit node

I have set up an exit node using this guide, but traffic doesn't pass through this node. If anyone else runs into this issue, I have asked about it here:

https://forum.tailscale.com/t/fly-io-exit-node-enabled-on-tailscale-but-traffic-doesnt-pass-through-the-node/1356

Failed deploy issue: context deadline exceeded

I encountered the following error while using fly deploy.

$ fly deploy
Update available 0.0.353 -> v0.0.499.
Run "fly version update" to upgrade.
==> Verifying app config
--> Verified app config
==> Building image
Error failed to fetch an image or build from source: error connecting to docker: failed building options: failed probing "personal": context deadline exceeded

After diagnosing with the fly doctor command, it was found that the problem was caused by the local network firewall, which prevented the normal use of WireGuard to connect to fly.io's servers.

$ fly doctor
Update available 0.0.353 -> v0.0.499.
Run "fly version update" to upgrade.
Testing authentication token... PASSED
Testing flyctl agent... PASSED
Testing local Docker instance... Nope
Pinging WireGuard gateway (give us a sec)... FAILED
(Error: ping gateway: no response from gateway received)

We can't establish connectivity with WireGuard for your personal organization.

WireGuard runs on 51820/udp, which your local network may block.

If this is the first time you've ever used 'flyctl' on this machine, you
can try running 'flyctl doctor' again.

Switching to another server and network without similar restriction rules, the problem was solved smoothly. I hope this can be helpful for anyone who may encounter similar problems.

-- deleted

Using wireguard iOs app

How come it seems to deploy 2 machines to tailscale?

I followed the instructions and even set flyctl scale count 1 and I see fly-syd and fly-syd-1 in my tailscale admin. Why are there 2 machines? There's even only 1 app on fly dashboard. Is there a way to know which machine is actually connected?

tailscale gets killed due to lack of memory

After using the exit node for like a 30mins-1hr it always goes offline, and the log says

Out of memory: Killed process 526 (tailscaled) total-vm:932140kB, anon-rss:185320kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:608kB oom_score_adj:0

Is there a way to prevent this?

Unable to deploy using Wget

Hey @patte,

It's been a while 😜

I have discovered an error in Wget while deploying with Fly.io's remote builder - the deployment will error out ([HOST](wget: can't connect to remote host: Host is unreachable)) while trying to download the latest package from pkgs.tailscale.com

I've created a workaround with cURL which also deploys successfully to Fly.io and will make a pull request if that's fine with you

Here are the latest deployment logs in case you can find a problem with it

*[main][~/fly-tailscale-exit]$ fly deploy
==> Verifying app config
--> Verified app config
==> Building image
Remote builder fly-builder-long-glitter-301 ready
==> Creating build context
--> Creating build context done
==> Building image with Docker
--> docker host: 20.10.12 linux x86_64
Sending build context to Docker daemon  79.54kB
[+] Building 7.1s (7/14)                                                                                              
 => [internal] load remote build context                                                                         0.0s
 => copy /context /                                                                                              0.1s
 => [internal] load metadata for docker.io/library/alpine:latest                                                 3.6s
 => [stage-1 1/8] FROM docker.io/library/alpine:latest@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b799  0.9s
 => => resolve docker.io/library/alpine:latest@sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc00  0.0s
 => => sha256:ff6bdca1701f3a8a67e328815ff2346b0e4067d32ec36b7992c1fdc001dc8517 1.64kB / 1.64kB                   0.0s
 => => sha256:e2e16842c9b54d985bf1ef9242a313f36b856181f188de21313820e177002501 528B / 528B                       0.0s
 => => sha256:b2aa39c304c27b96c1fef0c06bee651ac9241d49c4fe34381cab8453f9a89c7d 1.47kB / 1.47kB                   0.0s
 => => sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 3.37MB / 3.37MB                   0.8s
 => => extracting sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c                        0.1s
 => [tailscale 2/4] WORKDIR /app                                                                                 0.0s
 => CANCELED [stage-1 2/8] RUN apk update && apk add ca-certificates iptables ip6tables iproute2 && rm -rf /var  2.5s
 => ERROR [tailscale 3/4] RUN wget https://pkgs.tailscale.com/stable/tailscale_1.38.2_amd64.tgz &&   tar xzf ta  2.4s
------                                                                                                                
 > [tailscale 3/4] RUN wget https://pkgs.tailscale.com/stable/tailscale_1.38.2_amd64.tgz &&   tar xzf tailscale_1.38.2_amd64.tgz --strip-components=1:
#7 2.429 Connecting to pkgs.tailscale.com ([2604:a880:2:d0::61c:d001]:443)
#7 2.430 wget: can't connect to remote host: Host is unreachable
------
Error failed to fetch an image or build from source: error building: executor failed running [/bin/sh -c wget https://pkgs.tailscale.com/stable/${TSFILE} &&   tar xzf ${TSFILE} --strip-components=1]: exit code: 1

Update to new Fly template naming scheme

Hi @patte,

I've noticed that the new flyctl looks for Fly Templates now named fly.toml which causes the current fly-template.toml to not be found.

My suggestion would be to update all documentation and files to now refer to fly.toml instead of fly-template.toml

Instructions need to be updated to use V2 API (Machines)

I have been keeping my own images and noticed that now the V2 API has become mandatory. I will update my instructions according to the following issue: spotsnel#7. if you appreciate, I can port this back after I finished.

Currently dealing with some related issues with scaling, as flyctl has some unexpected behaviour:

Failed due to unhealthy allocations

please tell me how to fix this
You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment

1 desired, 1 placed, 0 healthy, 1 unhealthy
--> v0 failed - Failed due to unhealthy allocations and deploying as v1
Error logs
2022-02-10T03:39:24.854 app[f483df71] maa [info] Starting init (commit: 0c50bff)...

2022-02-10T03:39:24.871 app[f483df71] maa [info] Preparing to run: /app/start.sh as root

2022-02-10T03:39:24.874 app[f483df71] maa [info] Error: UnhandledIoError(Os { code: 2, kind: NotFound, message: "No such file or directory" })

Fly.io no longer provides dedicated IPv4 addresses for free

I encountered the following warning while using fly deploy.

$ flyctl  deploy
...
==> Creating release
Error Services defined at indexes: 0 require a dedicated IP address. You currently have no dedicated IPs allocated. Please allocate at least one dedicated IP before deploying (`fly ips allocate-v4` and/or `fly ips allocate-v6`). Affected services:
  [0] udp/41641 => 8080

$ fly ips allocate-v4
? Looks like you're accessing a paid feature. Dedicated IPv4 addresses now costs $2/mo. Are you ok with this?

After investigation, it was found that Fly.io no longer provides dedicated IPv4 addresses for free.

I wonder if there's a way to lift the restrictions mentioned above by modifying the tailscale configuration? Thank you for your help.

Faile to deploy

how i can fix this
Recent Events
TIMESTAMP TYPE MESSAGE
2023-02-04T11:49:58Z Received Task received by client
2023-02-04T11:49:59Z Task Setup Building Task Directory
2023-02-04T11:50:16Z Driver Failure rpc error: code = Unknown desc = error waiting for vsock readiness: vsock connection attempts exhausted
2023-02-04T11:50:16Z Not Restarting Error was unrecoverable
2023-02-04T11:50:16Z Alloc Unhealthy Unhealthy because of failed task
2023-02-04T11:50:16Z Killing Sent interrupt. Waiting 5s before force killing

2023-02-04T11:50:00Z [info]Configuring virtual machine
2023-02-04T11:50:00Z [info]Pulling container image
2023-02-04T11:50:06Z [info]Unpacking image
2023-02-04T11:50:08Z [info]Preparing kernel init
2023-02-04T11:50:09Z [info]Configuring firecracker
2023-02-04T11:50:09Z [info]Starting virtual machine
2023-02-04T11:50:09Z [info]Starting init (commit: e3cff9e)...
2023-02-04T11:50:09Z [info]Preparing to run: /app/start.sh as root
2023-02-04T11:50:09Z [info]Error: UnhandledIoError(Os { code: 2, kind: NotFound, message: "No such file or directory" })
2023-02-04T11:50:09Z [info][ 0.150709] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
2023-02-04T11:50:09Z [info][ 0.152751] CPU: 0 PID: 1 Comm: init Not tainted 5.12.2 #1
2023-02-04T11:50:09Z [info][ 0.152715] Call Trace:
2023-02-04T11:50:09Z [info][ 0.153150] show_stack+0x52/0x58
2023-02-04T11:50:09Z [info][ 0.153548] dump_stack+0x6b/0x86
2023-02-04T11:50:09Z [info][ 0.153997] panic+0xfb/0x2bc
2023-02-04T11:50:09Z [info][ 0.154410] do_exit.cold+0x60/0xb0
2023-02-04T11:50:09Z [info][ 0.154927] do_group_exit+0x3b/0xb0
2023-02-04T11:50:09Z [info][ 0.155760] __x64_sys_exit_group+0x18/0x20
2023-02-04T11:50:09Z [info][ 0.156397] do_syscall_64+0x38/0x50
2023-02-04T11:50:09Z [info][ 0.156867] entry_SYSCALL_64_after_hwframe+0x44/0xae
2023-02-04T11:50:09Z [info][ 0.157472] RIP: 0033:0x7f7f661f16c9
2023-02-04T11:50:09Z [info][ 0.157863] Code: eb ef 48 8b 76 28 e9 a5 03 00 00 64 48 8b 04 25 00 00 00 00 48 8b b0 b0 00 00 00 e9 af ff ff ff 48 63 ff b8 e7 00 00 00 0f 05 3c 00 00 00 48 89 d0 0f 05 eb f9 66 2e 0f 1f 84 00 00 00 00 00
2023-02-04T11:50:09Z [info][ 0.160939] RSP: 002b:00007ffc13bbb8b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
2023-02-04T11:50:09Z [info][ 0.162255] RAX: ffffffffffffffda RBX: 00007f7f65f6c340 RCX: 00007f7f661f16c9
2023-02-04T11:50:09Z [info][ 0.163163] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001
2023-02-04T11:50:09Z [info][ 0.164177] RBP: 0000000000000001 R08: 00007f7f662cba58 R09: 0000000000000000
2023-02-04T11:50:09Z [info][ 0.165102] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc13bbb918
2023-02-04T11:50:09Z [info][ 0.166010] R13: 00007ffc13bbb928 R14: 0000000000000000 R15: 0000000000000000
2023-02-04T11:50:09Z [info][ 0.166978] Kernel Offset: disabled
2023-02-04T11:50:09Z [info][ 0.167434] Rebooting in 1 seconds..
--> v7 failed - Failed due to unhealthy allocations and deploying as v8

Traffic still routed via DERP

So based on the readme I have dedicated an IPv4 on the fly.io

flyctl ips list
VERSION	IP          	TYPE  	REGION	CREATED AT
v4     	1.2.3.4 	public	global	1h44m ago

But when I tried to do ping from the flyctl vm to my machine it always routed via the DERP relay

3d8d909f754089:/# ./app/tailscale ping my-machine
pong from my-machine (100.86.107.48) via DERP(sin) in 21ms

This is my current fly.toml

# fly.toml app configuration file generated for kkk-net on 2023-10-03T14:50:04+07:00
#
# See https://fly.io/docs/reference/configuration/ for information about how to use this file.
#

app = "my-net"
primary_region = "sin"
kill_signal = "SIGINT"
kill_timeout = "5s"

[experimental]
  auto_rollback = false
  private_network = true

[build]

[env]
  PORT = "443"

[[services]]
  protocol = "udp"
  internal_port = 443
  processes = ["app"]

  [[services.ports]]
    port = 443
  [services.concurrency]
    type = "connections"
    hard_limit = 100
    soft_limit = 75

Thanks a lot in advance!

patte / fly-tailscale-exit Goto Github PK

fly-tailscale-exit's Introduction

Hi, I'm patte ✌️

fly-tailscale-exit's People

Stargazers

Watchers

Forkers

fly-tailscale-exit's Issues

Recommend Projects

Recommend Topics

Recommend Org