`read_json_auto` requires higher memory limit than it uses about duckdb HOT 7 CLOSED

MacOS has compressed memory which is why the physical memory usage might be different than the memory usage reported by DuckDB. DuckDB's memory limit and memory usage is measured in uncompressed memory usage.

from duckdb.

I would recommend using ndjson as your json format - that should be more (memory) efficient

from duckdb.

I assume the profiler is incorrect, but to verify this, try setting the thread limit to 1 (pragma threads=1) and try the experiments again.
My hunch is that the memory limit can be much lower without erroring in that case.

from duckdb.

I assume the profiler is incorrect, but to verify this, try setting the thread limit to 1 (pragma threads=1) and try the experiments again.

Wouldn't that accomplish the same as the

DUCKDB_WORKER_THREADS = 1

in the reproduction script?

I tried with a

duckdb_conn.execute("SET THREADS to 1;")

I am seeing the same results. I am also checking with asitop on the OS level to view memory limit when I run it and am seeing about 380.3 MiB to match.

from duckdb.

Using DuckDB CLI with the default memory limit of 12.7GB and 10 threads doesn't seem to be using a ton of memory for the file, but inside a server it seems to be trying to need more than 26GB of memory even though it works fine on a 16 GB Mac M1. The example above is the closest example I could find to replicate the behavior.

Might have to do with having a few nested columns 🤔

from duckdb.

MacOS has compressed memory which is why the physical memory usage might be different than the memory usage reported by DuckDB. DuckDB's memory limit and memory usage is measured in uncompressed memory usage.

Thanks for your response and help. Good to know.

Are there any methods / tricks to be able to load / query JSON without so much memory?

For ex would any of the below methods help:

• loading json from pandas and querying the resulting dataframe
• providing json structure so it doesn't need to infer etc
• fetch_record_batch

from duckdb.

I am also facing the same problem. Does anyone knows how to resolve this ?

from duckdb.