C API: Writing (large) list with Appender fails about duckdb HOT 7 CLOSED

Just memory usage yes, doubling sounds like a good idea in an append scenario.

from duckdb.

Thanks for the report!

duckdb_create_data_chunk allocates vectors with as size duckdb_vector_size(). Writing more than duckdb_vector_size() entries to any of the vectors is undefined behavior. For lists, as in this example, the child vector similarly is allocated with space for duckdb_vector_size() entries.

The duckdb_list_vector_reserve can be used to resize the child vector of a list. Note that this must be done before calling methods like duckdb_vector_get_data on the list child vector because calling duckdb_list_vector_reserve can invalidate the pointer previously returned by duckdb_vector_get_data.

Below is a fix for the original example:

duckdb_list_entry entry;
auto listData = (duckdb_list_entry*)duckdb_vector_get_data(listVector);

auto listLength = vectorMaxSize + 3;

duckdb_vector listChildVector = duckdb_list_vector_get_child(listVector);
// reserve enough space for the elements
duckdb_list_vector_reserve(listVector, listLength);
auto listChildVectorData = (int32_t*)duckdb_vector_get_data(listChildVector);

from duckdb.

Thanks for the explanation @Mytherin. This will work if the caller knows the required size upfront, but I guess very frequently they won't know how many rows or how big lists they will be appending. For example, when reading data from some REST service, or sensor data, or something else. In such cases, it's not possible to call duckdb_list_vector_reserve only once.

The only workaround that I see is to keep data for all rows in memory and actually build the data chunk after every duckdb_vector_size row. Can the list child vector be resized without invalidating the data that was already written?

from duckdb.

duckdb_list_vector_reserve does not invalidate the written data, only the pointer. Think of it like realloc.

from duckdb.

Ok, I'll give it a try.

from duckdb.

What is the downside of allocating a larger vector and writing only part of the data? Is it just increased memory usage (which I hope gets freed when I reset the chunk or destroy the chunk) or is there some other hidden cost?

I'm asking because I'm thinking of doubling the allocation size when I reach the current allocation size.

from duckdb.

Thanks. Actually, I made it a little bit smarter to avoid extra allocation. As I have to call duckdb_append_data_chunk I already need to track how many rows there are. I use this row count and if I'm at the end of the chunk I will increase it only by 1.2, if in the beginning I will double it and do something in the middle in between.

from duckdb.