OOM death on SQL COPY about cstore_fdw HOT 7 CLOSED

Having experienced the OOM on a vanilla PostgreSQL instance, I started using this (in the init script, or a cron job), to keep Postgres out of its crosshairs:

for p in `pgrep -P 1 postmaster`; do echo "pid $p"; echo -17 > /proc/$p/oom_adj; done

from cstore_fdw.

@btubbs Thanks for reporting this. Improving the memory usage is in our road map. Giving a warning is also an interesting option, and definitely worth looking into.

Can you please send your data file + table schema to [email protected] so I can look into this in more detail?

from cstore_fdw.

@btubbs Can you also try reducing stripe_row_count to see if it helps? Currently we buffer stripe_row_count rows in memory before flushing to disk, so memory usage is "stripe_row_count * average_row_size * constant (usually 3..5)".

The default stripe_row_count is 150000. Based on the number of rows you have in your CSV file, you can adjust it to your use case, so it uses less memory. Decreasing stripe_row_count too much can cause some performance issues.

from cstore_fdw.

As an update, I have looked into the problem of memory usage, and have written a write-up with a list of our options here.

We will make some of the changes in our 1.2 release. Please see our roadmap.

from cstore_fdw.

We have been working on this issue for some time now. Write optimizations gave promising results especially in compressed case. Changes are currently in review stage, they will be here in 1.2 release which is expected to be out in mid jan 2015.

from cstore_fdw.

changes are committed into develop branch, available immediately

from cstore_fdw.

Merged into master. Closing.

from cstore_fdw.