Comments (22)
Ah, after turning /etc/resolv.conf again into a symlink of systemd-resolved
sudo ln -rsf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
the messages disappear.
I assume it wants now to call the localhost ip of the systemd-resolved instead of the network ip of the outside resolver.
I don't know how I could let this plugin verbose debug logging into the journal, but I'm glad that it is fixed now.
from aardvark-dns.
And it's back
dns request failed: request timed out
It seems to happen if I execute nslookup
in a container.
But I get a result. Weird.
from aardvark-dns.
I detected the same problem on my raspberry pi with Fedora and rootless podman containers.
from aardvark-dns.
Having the exact same issue. DNS lookup is extremely unstable, only two thirds of lookups work.
This is on Fedora Silverblue 37 using rootless containers and the new networking stack.
from aardvark-dns.
Now using a nextcloud container which communicates with a PostgreSQL and a LDAP container.
I access the nextcloud container with a nginx reverse proxy container from the internet.
1/3 of the nextcloud web interface results into a 502 bad gateway, because of the unstable DNS.
And my logs are getting spammed with
aardvark-dns[3617]: 50310 dns request failed: request timed out
I hope the next podman release will include the new network stack which hopefully fixes this issue.
from aardvark-dns.
It is impossible to help with these issues as reporters did not provide versions for podman, netavark, and aarvark-dns. please provide as much relevant information as possible.
from aardvark-dns.
podman: 4.3.1
netavark: 1.4.0
aardvark-dns: 1.4.0
from aardvark-dns.
would you say your machine/vm is high performent or maybe has slow IO/processor/RAM limitations? I'm trying to understand if you might have a race.
from aardvark-dns.
The first bare metal server server with this issue has the HDD connected to SATA with the server.
CPU: Intel Xeon E3-1231 v3 - 3.4 GHz - 4 core(s)
RAM: 32GB - DDR3
The second bare metal server has a 870 Evo SSD connected via SATA to the server.
CPU: AMD Phenom 2 955 X4 3.2 GHz 4 cores
RAM: 6GB - DDR2
The second server spams this issue many times.
The first server spams it less often but also regularly.
from aardvark-dns.
@flouthoc wdyt?
from aardvark-dns.
Now my faster server spammed it at night while the server pods weren't used by clients.
from aardvark-dns.
We used to see similar issues in older versions of netavark and aardvark in Podman CI as well but it was fixed in newer versions with #220 but I guess there might be some issue which is not being reproduced in our CI, I'll try to reproduce this locally and see if i can reproduce this.
from aardvark-dns.
I'm seeing the same problems on a freshly installed Fedora Server 37 instance as well. The machine is basically completely idling with no load and my journal is still filled with these errors. This issue was not present before 2022, and I've ran similar setups on slower machines without any DNS lookup issues.
podman: 4.3.1
netavark: 1.4.0
aardvark-dns: 1.4.0
from aardvark-dns.
Can you test with v1.5?
from aardvark-dns.
I think the timeout is fixed.
But now sometimes I get an "empty dns response" on all machines.
from aardvark-dns.
Does this cause problems for the container or is just an error that is logged often?
from aardvark-dns.
It's logged often with long breaks between. So it appears in groups most of the time.
I think the services just repeat the DNS action again until it works.
So I would say it just consumes CPU time and maybe network speed?
Maybe I don't use it long enough to see long term errors.
from aardvark-dns.
Do you have a simple reproducer? What kind of application are you running and how many dns request does it make?
from aardvark-dns.
Tested with v1.5 and I'm getting a lot of dns request got empty response
as well. Here's a list of all the containers I'm running on my system:
eclipse-mosquitto:2
koenkk/zigbee2mqtt
homeassistant/home-assistant:stable
It's notable that none of these containers are particularly demanding on the hardware, and my system load average is generally below 0.1 at all times.
from aardvark-dns.
It happens without workload.
The server just runs
- mailrise
- swag & duckdns
- borg-backup-server
from aardvark-dns.
I'm experiencing the same issue with aardvark-dns reporting lots of dns request got empty response
errors in my logs. It seems to be causing problems for at least the containers running Uptime Kuma, Invidious and Jellyseerr. Uptime Kuma starts throwing ECONNRESET
when doing GET requests, and Invidious and Jellyseerr similarly start to have their requests fail, with external content taking a long time to load, if at all.
It happens with both 1.5.0 and the latest 1.6.0 from podman-next. For me it seems to start after around 3 days of uptime. I've tried changing machines and switching from onboard Realtek to an Intel i350-T2 controller, but both to no avail. Rebooting solves the issue, until uptime reaches 3 days again.
from aardvark-dns.
Just ran into this issue as well with Nextcloud + Nginx Proxy Manager. What's funny is that I am using the same docker-compose setup on two different servers and one works fine while the other one doesn't. The only difference is that the one that is breaking isn't publicly accessible on the Internet and is instead setup to respond over a .lan
domain which is configured on the home router. NPM has a proxy host setup that responds to mydomain.lan
and redirects it to the nextcloud container.
It will work for a bit when I up/down NPM, but then eventually fail after a few hours or even days with 502 bad gateway errors, and dns request got empty response
starts getting spammed into journalctl.
My setup
- Arch Linux linux-lts 6.6.18-1
- podman: 4.9.3-1
- podman-docker: 4.9.3-1
- docker-compose: 2.24.6-1
- netavark: 1.10.3-1
- aardvark-dns: 1.10.0-1
Here are my docker-compose files to set up each of them (rootful btw):
Nextcloud docker-compose.yml
version: '3'
services:
db:
image: mariadb
command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
restart: always
volumes:
- ./db:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=<pw here>
- MARIADB_AUTO_UPGRADE=1
- MARIADB_DISABLE_UPGRADE_BACKUP=1
env_file:
- db.env
networks:
- backend
redis:
image: redis:alpine
restart: always
networks:
- backend
nextcloud:
image: nextcloud:apache
restart: always
volumes:
- ./html:/var/www/html
environment:
- MYSQL_HOST=db
- REDIS_HOST=redis
env_file:
- db.env
depends_on:
- db
- redis
networks:
- nextcloud_frontend
- backend
cron:
image: nextcloud:apache
restart: always
volumes:
- ./html:/var/www/html
entrypoint: /cron.sh
depends_on:
- db
- redis
networks:
- backend
networks:
nextcloud_frontend:
external: true
backend:
db.env
MYSQL_PASSWORD=<pw here>
MYSQL_DATABASE=nextcloud
MYSQL_USER=nextcloud
Nginx Proxy Manager docker-compose.yml
version: '3.8'
services:
proxy:
image: 'jc21/nginx-proxy-manager:latest'
restart: always
ports:
# These ports are in format <host-port>:<container-port>
- '80:80' # Public HTTP Port
- '443:443' # Public HTTPS Port
- '81:81' # Admin Web Port
# Add any other Stream port you want to expose
# - '21:21' # FTP
# Uncomment the next line if you uncomment anything in the section
# environment:
# Uncomment this if you want to change the location of
# the SQLite DB file within the container
# DB_SQLITE_FILE: "/data/database.sqlite"
# Uncomment this if IPv6 is not enabled on your host
# DISABLE_IPV6: 'true'
healthcheck:
test: ["CMD", "/bin/check-health"]
interval: 30s
timeout: 3s
volumes:
- ./data:/data
- ./letsencrypt:/etc/letsencrypt
networks:
- nextcloud_frontend
networks:
nextcloud_frontend:
external: true
from aardvark-dns.
Related Issues (20)
- dns: inbuilt resolver should return both `IPv6` and `IPv4` records if request type is `ANY` HOT 2
- Add LICENSE file and COC to repoistory HOT 1
- Dependency Dashboard
- Disable Dependabot after renovate trial
- Need bidirectional communication channel between netavark and aardvark HOT 8
- Add host.containers.internal entry in aardvark-dns HOT 2
- [NOT UPSTREAM PROBLEM] test `packit propose-downstream` HOT 2
- [packit] Propose downstream failed for release v1.7.0
- test_backend_network_scoped_custom_dns_server fails HOT 3
- Updating trust-dns HOT 1
- DNS requests timeout HOT 24
- Is there a way to reserve or limit IP addresses when using DNS? HOT 1
- CI flake: three networks with a connect HOT 1
- When forward dns request to outside name server, `aardvark-dns` should check and ignore its own listening IPs or error out, to avoid infinite recursion. HOT 1
- Setting invalid options in /etc/resolv.conf makes dns unresponsive HOT 1
- Add response TTL settings HOT 2
- Reverse lookups in podman return `.` for domain name in answer section
- Rootful containers on debian sid host unable to resolve DNS HOT 4
- [packit] Propose downstream failed for release v1.11.0 HOT 1
- Publishing udp range larger than 16383 ending with 65535 breaks dns resolution on user defined networks with root networking. HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aardvark-dns.