Comments (8)
I am guessing that it is maybe picking up the wrong interface to send things out on?
As it says even ipv4 addr is unreachable 176.58.93.154:443
, and yet:
root@udmp2333:/data/tailscale# telnet 176.58.93.154 443
Trying 176.58.93.154...
Connected to 176.58.93.154.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
from tailscale.
Also, for controlplane:
root@udmp2333:/data/tailscale# telnet 2a05:d014:386:202:f041:44b6:9559:668 443
Trying 2a05:d014:386:202:f041:44b6:9559:668...
telnet: Unable to connect to remote host: Network is unreachable
root@udmp2333:/data/tailscale# telnet controlplane.tailscale.com 443
Trying 3.125.149.81...
Connected to controlplane.tailscale.com.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
So, A
lookup and telnet to it works, but AAAA
lookup and telnet to it does not.
from tailscale.
And full tailscale logs
Apr 18 07:06:09 udmp2333 systemd[1]: Starting Tailscale node agent...
░░ Subject: A start job for unit tailscaled.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tailscaled.service has begun execution.
░░
░░ The job identifier is 3912.
Apr 18 07:06:09 udmp2333 tailscaled[645386]: logtail started
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Program starting: v1.64.0-t7e9bebdb2-gf314c5be5, Go 1.22.2: []string{"/usr/sbin/tailscaled", "--state=/var/lib/tailscale/tailscaled.state", "--socket=/run/tailscale/tailscaled.sock", "--port=41641"}
Apr 18 07:06:09 udmp2333 tailscaled[645386]: RAW-STDERR: ***
Apr 18 07:06:09 udmp2333 tailscaled[645386]: RAW-STDERR: *** Lines prefixed with RAW-STDERR below bypassed logtail and probably come from a previous run of the program
Apr 18 07:06:09 udmp2333 tailscaled[645386]: RAW-STDERR: ***
Apr 18 07:06:09 udmp2333 tailscaled[645386]: RAW-STDERR:
Apr 18 07:06:09 udmp2333 tailscaled[645386]: RAW-STDERR: network is unreachable\n"}
Apr 18 07:06:09 udmp2333 tailscaled[645386]: LogID: 4fa038657983321a890074e071fbb242554a989ae17277d4ae57eb7cca485c66
Apr 18 07:06:09 udmp2333 tailscaled[645386]: logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
Apr 18 07:06:09 udmp2333 dbus-daemon[671]: [system] Activating via systemd: service name='org.freedesktop.resolve1' unit='dbus-org.freedesktop.resolve1.service' requested by ':1.25' (uid=0 pid=645386 comm="/usr/sbin/tailscaled --state=/var/lib/tailscale/ta")
Apr 18 07:06:09 udmp2333 dbus-daemon[671]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.resolve1.service': Unit dbus-org.freedesktop.resolve1.service not found.
Apr 18 07:06:09 udmp2333 tailscaled[645386]: dns: [rc=unknown ret=direct]
Apr 18 07:06:09 udmp2333 tailscaled[645386]: dns: using "direct" mode
Apr 18 07:06:09 udmp2333 tailscaled[645386]: dns: using *dns.directManager
Apr 18 07:06:09 udmp2333 tailscaled[645386]: deleting [-j ts-input] in filter/INPUT: running [/usr/sbin/iptables -t filter -D INPUT -j ts-input --wait]: exit status 2: iptables v1.8.7 (legacy): Couldn't load target `ts-input':No such file or directory
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Try `iptables -h' or 'iptables --help' for more information.
Apr 18 07:06:09 udmp2333 tailscaled[645386]: deleting [-j ts-forward] in filter/FORWARD: running [/usr/sbin/iptables -t filter -D FORWARD -j ts-forward --wait]: exit status 2: iptables v1.8.7 (legacy): Couldn't load target `ts-forward':No such file or directory
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Try `iptables -h' or 'iptables --help' for more information.
Apr 18 07:06:09 udmp2333 tailscaled[645386]: deleting [-j ts-postrouting] in nat/POSTROUTING: running [/usr/sbin/iptables -t nat -D POSTROUTING -j ts-postrouting --wait]: exit status 2: iptables v1.8.7 (legacy): Couldn't load target `ts-postrouting':No such file or directory
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Try `iptables -h' or 'iptables --help' for more information.
Apr 18 07:06:09 udmp2333 tailscaled[645386]: deleting [-j ts-input] in filter/INPUT: running [/usr/sbin/ip6tables -t filter -D INPUT -j ts-input --wait]: exit status 2: ip6tables v1.8.7 (legacy): Couldn't load target `ts-input':No such file or directory
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Try `ip6tables -h' or 'ip6tables --help' for more information.
Apr 18 07:06:09 udmp2333 tailscaled[645386]: deleting [-j ts-forward] in filter/FORWARD: running [/usr/sbin/ip6tables -t filter -D FORWARD -j ts-forward --wait]: exit status 2: ip6tables v1.8.7 (legacy): Couldn't load target `ts-forward':No such file or directory
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Try `ip6tables -h' or 'ip6tables --help' for more information.
Apr 18 07:06:09 udmp2333 tailscaled[645386]: [RATELIMIT] format("deleting %v in %s/%s: %v")
Apr 18 07:06:09 udmp2333 tailscaled[645386]: cleanup: list tables: netlink receive: invalid argument
Apr 18 07:06:09 udmp2333 tailscaled[645386]: wgengine.NewUserspaceEngine(tun "tailscale0") ...
Apr 18 07:06:09 udmp2333 systemd[1]: Started Tailscale node agent.
░░ Subject: A start job for unit tailscaled.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit tailscaled.service has finished successfully.
░░
░░ The job identifier is 3912.
Apr 18 07:06:09 udmp2333 tailscaled[645386]: setting link attributes: setsockopt: protocol not available
Apr 18 07:06:09 udmp2333 dbus-daemon[671]: [system] Activating via systemd: service name='org.freedesktop.resolve1' unit='dbus-org.freedesktop.resolve1.service' requested by ':1.25' (uid=0 pid=645386 comm="/usr/sbin/tailscaled --state=/var/lib/tailscale/ta")
Apr 18 07:06:09 udmp2333 dbus-daemon[671]: [system] Activation via systemd failed for unit 'dbus-org.freedesktop.resolve1.service': Unit dbus-org.freedesktop.resolve1.service not found.
Apr 18 07:06:09 udmp2333 networkd-dispatcher[690]: WARNING:Unknown index 29 seen, reloading interface list
Apr 18 07:06:09 udmp2333 tailscaled[645386]: dns: [rc=unknown ret=direct]
Apr 18 07:06:09 udmp2333 tailscaled[645386]: dns: using "direct" mode
Apr 18 07:06:09 udmp2333 tailscaled[645386]: dns: using *dns.directManager
Apr 18 07:06:09 udmp2333 tailscaled[645386]: link state: interfaces.State{defaultRoute= ifs={br0:[192.168.17.70/24 llu6] eth8:[68.174.248.*/20 llu6] tun1:[192.168.30.17/32 llu6] tun2:[10.0.3.1/24 llu6]} v4=true v6=false}
Apr 18 07:06:09 udmp2333 tailscaled[645386]: onPortUpdate(port=41641, network=udp6)
Apr 18 07:06:09 udmp2333 tailscaled[645386]: router: using firewall mode pref
Apr 18 07:06:09 udmp2333 tailscaled[645386]: router: default choosing iptables
Apr 18 07:06:09 udmp2333 tailscaled[645386]: router: v6 = true, v6filter = true, v6nat = true
Apr 18 07:06:09 udmp2333 tailscaled[645386]: onPortUpdate(port=41641, network=udp4)
Apr 18 07:06:09 udmp2333 tailscaled[645386]: magicsock: disco key = d:3d8ce7f92d1c3ed6
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Creating WireGuard device...
Apr 18 07:06:09 udmp2333 tailscaled[645386]: external route: up
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Bringing WireGuard device up...
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Bringing router up...
Apr 18 07:06:09 udmp2333 systemd-networkd[403]: tailscale0: Link UP
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Clearing router settings...
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Starting network monitor...
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Engine created.
Apr 18 07:06:09 udmp2333 systemd-networkd[403]: tailscale0: Gained carrier
Apr 18 07:06:09 udmp2333 systemd-networkd[403]: tailscale0: Gained IPv6LL
Apr 18 07:06:09 udmp2333 tailscaled[645386]: pm: using backend prefs for "profile-a976": Prefs{ra=true dns=true want=false routes=[0.0.0.0/0 ::/0 192.168.17.0/24] snat=true nf=on update=on Persist{lm=, o=, n=[Aj5af] u="*@*.net"}}
Apr 18 07:06:09 udmp2333 tailscaled[645386]: envknob: PORT="41641"
Apr 18 07:06:09 udmp2333 tailscaled[645386]: logpolicy: using $STATE_DIRECTORY, "/var/lib/tailscale"
Apr 18 07:06:09 udmp2333 tailscaled[645386]: got LocalBackend in 56ms
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Start
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Backend: logs: be:4fa038657983321a890074e071fbb242554a989ae17277d4ae57eb7cca485c66 fe:
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: client.Login(false, 0)
Apr 18 07:06:09 udmp2333 tailscaled[645386]: health("overall"): error: state=Stopped, wantRunning=false
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: doLogin(regen=false, hasUrl=false)
Apr 18 07:06:09 udmp2333 tailscaled[645386]: Switching ipn state NoState -> Stopped (WantRunning=false, nm=false)
Apr 18 07:06:09 udmp2333 tailscaled[645386]: wgengine: Reconfig: configuring userspace WireGuard config (with 0/0 peers)
Apr 18 07:06:09 udmp2333 tailscaled[645386]: wgengine: Reconfig: configuring router
Apr 18 07:06:09 udmp2333 tailscaled[645386]: wgengine: Reconfig: configuring DNS
Apr 18 07:06:09 udmp2333 tailscaled[645386]: dns: Set: {DefaultResolvers:[] Routes:{} SearchDomains:[] Hosts:0}
Apr 18 07:06:09 udmp2333 tailscaled[645386]: dns: Resolvercfg: {Routes:{} Hosts:0 LocalDomains:[]}
Apr 18 07:06:09 udmp2333 tailscaled[645386]: dns: OScfg: {}
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: trying bootstrapDNS("derp10d.tailscale.com", "192.73.240.132") for "controlplane.tailscale.com" ...
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: bootstrapDNS("derp10d.tailscale.com", "192.73.240.132") for "controlplane.tailscale.com" error: Get "https://derp10d.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp 192.73.240.132:443: connect: network is unreachable
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: trying bootstrapDNS("derp4g.tailscale.com", "2a00:dd80:20::8f") for "controlplane.tailscale.com" ...
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: bootstrapDNS("derp4g.tailscale.com", "2a00:dd80:20::8f") for "controlplane.tailscale.com" error: Get "https://derp4g.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp [2a00:dd80:20::8f]:443: connect: network is unreachable
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: trying bootstrapDNS("derp5d.tailscale.com", "43.245.48.250") for "controlplane.tailscale.com" ...
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: bootstrapDNS("derp5d.tailscale.com", "43.245.48.250") for "controlplane.tailscale.com" error: Get "https://derp5d.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp 43.245.48.250:443: connect: network is unreachable
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: trying bootstrapDNS("derp21d.tailscale.com", "2607:f740:50::ca4") for "controlplane.tailscale.com" ...
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: bootstrapDNS("derp21d.tailscale.com", "2607:f740:50::ca4") for "controlplane.tailscale.com" error: Get "https://derp21d.tailscale.com/bootstrap-dns?q=controlplane.tailscale.com": dial tcp [2607:f740:50::ca4]:443: connect: network is unreachable
Apr 18 07:06:09 udmp2333 tailscaled[645386]: control: trying bootstrapDNS("derp25b.tailscale.com", "102.67.167.245") for "controlplane.tailscale.com" ...
Apr 18 07:06:09 udmp2333 tailscaled[645386]: [RATELIMIT] format("control: trying bootstrapDNS(%q, %q) for %q ...")
from tailscale.
IPV4 routing (with IP addr redacted)
ip route show table all | grep -v local
default via 68.174.240.1 dev eth8 table 201.eth8 proto dhcp
blackhole default table 251.blackhole proto PBR
10.0.3.0/24 dev tun2 proto kernel scope link src 10.0.3.1
68.174.240.0/20 dev eth8 proto kernel scope link src 68.174.248.*
192.168.17.0/24 dev br0 proto kernel scope link src 192.168.17.70
192.168.19.0/24 via 192.168.30.16 dev tun1
192.168.30.16 dev tun1 proto kernel scope link src 192.168.30.17
from tailscale.
Just to clarify- after a brief look at your logs, it looks like this device was previously successfully running at 1.64 and started failing after a restart, is that correct?
As you say, looks like it cannot reach control plane or the relay servers either on IPv4 or IPv6.
Just to double clarify, what is the output of ip route get 2a00:dd80:3c::3d5
, ip route get 176.58.93.154
on this machine?
From a brief look I don't see recent changes in the logic to figure out the default route for connections to control plane/derp. I wonder if we should log more, i.e what interface was chosen.
from tailscale.
The device failed after upgrading form 1.62.x - not just after a restart.
root@udmp2333:~# ip route get 176.58.93.154
176.58.93.154 via 68.174.240.1 dev eth8 table 201.eth8 src 68.174.248.36 uid 0
cache
root@udmp2333:~# ip route get 2a00:dd80:3c::3d5
RTNETLINK answers: Network is unreachable
Yeah - logging the interface makes sense. Or just allow a cmd-line that specifies the external interface - that can be hard to auto-detect in pathological cases anyhow..
from tailscale.
So - one more observation.
If running with --tun userspace-networking
it works just fine - but can't work as a subnet router unfortunately. So probably a tunnel selection of some sort.
from tailscale.
Upgraded to 1.66 - same issue. Any updates on triaging this?
from tailscale.
Related Issues (20)
- FR: Allow setting SSM parameter for state in Docker HOT 1
- tsnet: improve ergonomics for listening for UDP packets
- It _should_ always be defined, but in practice, we have seen various non-standard environments and I am weary of breaking folks' custom configurations (i.e non-operator kube deployments). HOT 1
- Static ts-input rule removed from INPUT when --netfilter-mode=nodivert used HOT 1
- Installer script automatically does a partial upgrade on Arch Linux. HOT 1
- FR: Wildcard on split DNS.
- FR: MagicDNS - add host 'localtailscale' that resolves to local tailscale device IP HOT 1
- MacOS VPN on demand connects VPN but not Tailscale itself HOT 2
- Impact of Enabling MagicDNS on NextDNS Configuration and Resolver Settings in Tailscale
- FR: Local Socks Proxy
- [BUG]: Auto Update not working in Debian LXC on Proxmox HOT 2
- FR: built-in uptime/performance monitor into tailscale
- gitops-pusher apply fails with INTERNAL_ERROR received from peer HOT 5
- FR: allowlist/ACL for machine access to HTTPS certificates to avoid CT log leaks HOT 3
- Don't convert localhost to 127.0.0.1 in tailscale serve HOT 1
- Novice Can't Connect Using Exit Node HOT 1
- FR: On synology "tailscale update" should maintain outbound connections configuration HOT 3
- Tailscale version 1.66.0 breaks docker swarm container connectivity to tailscale hosts HOT 3
- FR: reverse proxy in the repo
- FR: Change Mullvad authroized devices in app
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tailscale.