Ways to reproduce the issue in linux machine
- add packet latency to simulate the host to db cross region latency
sudo tc qdisc add dev lo root netem delay 60ms
- run this code using
- once things stabilize, check connection number from postgresql side with psql
SELECT sum(numbackends) FROM pg_stat_database;
- inject packet loss between host and db to simulate network blip
sudo iptables -A OUTPUT -p tcp --dport 5432 -m statistic --mode random --probability 0.5 -j DROP
- remove packet loss event in iptables supposing added rule index was 1
sudo iptables -D OUTPUT 1
- observe that things never recover until restarted
- check connection number from db side with psql again