As this is a problem with postgres not lemmy, it would be best if you can get in contact with the postgres community and developers.

from lemmy.

Total Memory (RAM): 500 GB

You sure you did 500MB, not 500 GB?

from lemmy.

facepalm Confirmed there was user error (not only is that postgres conf wrong, but I didn't even copy it into my volume when I upgrade to 16, so I was using defaults). I have reconfigured postgres to use 1 gb (and set the container limit to 1.4 gb just for good measure). It's been running for a few hours and has not gone over 1gb.

I will update this thread in 24 hours just to confirm there is no leak (but, I expect there not to be at this point).

from lemmy.

No probs. Can re-open if necessary.

from lemmy.

I'm still experiencing what seems to be a memory leak. I have configured (to my understanding) postgres (16) to use 1 GB. I set my postgres container's memory limit to be 1400 MB. The oom killer is killing postgres every 12 to 14 hours.

When the new db (postgres 16) was created when I upgraded, I did not have my postgres.conf setup (a mistake). I have copy my postgres.conf (see setting below) into my volume and restarted postgres. Is that sufficient for changing postgres's memory settings, or do I need to modify the database directly somehow?

# DB Version: 16
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 1 GB
# Data Storage: ssd

max_connections = 200
shared_buffers = 256MB
effective_cache_size = 768MB
maintenance_work_mem = 64MB
checkpoint_completion_target = 0.9
wal_buffers = 7864kB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 655kB
huge_pages = off
min_wal_size = 1GB
max_wal_size = 4GB

from lemmy.

You need this line in your postgres docker block:

https://github.com/LemmyNet/lemmy-ansible/blob/main/templates/docker-compose.yml#L102

from lemmy.

I like to keep my config in the default place (in the volume). I don't override the command (it changes the config path). I instead copy postgresql.conf to /var/lib/postgresql/data/postgresql.conf (inside the container of course - which I have volume mounted).

$ docker-compose exec postgres psql --dbname=lemmy --username=lemmy --command="SHOW config_file"
               config_file
------------------------------------------
 /var/lib/postgresql/data/postgresql.conf
(1 row)

$  docker-compose exec postgres cat /var/lib/postgresql/data/postgresql.conf
# Listen to all addressed (importantly 0.0.0.0)
listen_addresses = '*'

# Use https://pgtune.leopard.in.ua/ to generate performance settings and insert here.
# DB Version: 16
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 1 GB
# Data Storage: ssd
max_connections = 200
shared_buffers = 256MB
effective_cache_size = 768MB
maintenance_work_mem = 64MB
checkpoint_completion_target = 0.9
wal_buffers = 7864kB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 655kB
huge_pages = off
min_wal_size = 1GB
max_wal_size = 4GB

# These settings are copied from an initialization by initdb
log_timezone = 'UTC'
default_text_search_config = 'pg_catalog.english'
datestyle = 'iso, mdy'
timezone = 'UTC'
lc_messages = 'en_US.utf8'                      # locale for system error message
lc_monetary = 'en_US.utf8'                      # locale for monetary formatting
lc_numeric = 'en_US.utf8'                       # locale for number formatting
lc_time = 'en_US.utf8'                          # locale for time formatting

from lemmy.

Did you also try setting jit = 0 as mentioned in #4406?

from lemmy.

I thought that was related to that specific postgres 15 issue. But, worth a try.

It does seem to change behavior. But, I think it's just not doing the work up front. (The processes seem to use much much less ram on startup, but they are quickly catching back up to if jit was not set to 0).

from lemmy.

So, good news. I haven't had any problems the past few days. jit = 0 seems to help.

from lemmy.