Out of memory with ZetaSQL about differential-privacy HOT 6 CLOSED

It seems like you're probably running into the zetasql max_intermediate_byte_size. It's set in-code here, and at the moment there's no way to configure it except through code changes. We would need to submit a feature request to zetasql to make it configurable, and then update our query execution tool to expose the option.

As a short-term fix, you could try editing the limit in code. You might also be able to revise your query to avoid the limit - what query are you trying to run?

from differential-privacy.

Thank you for your reply.

The query is quite simple, just a sum() of one column y a dataset with more than 150k-200k rows. Not much to simplfy, apart from performing multiple sums o paritions of the dataset.

Will try to modify the code until its possible to make it configurable.

Is there an official way of requesting the feature to zetasql, apart from a github issue?

from differential-privacy.

I filed an internal feature request with them on your behalf, so no need for you to do anything there :)

I'll update this issue when we have progress.

from differential-privacy.

Thank you very much!

from differential-privacy.

Thanks for your patience! We have now updated the ZetaSQL dependencies and you could now try to increase memory by setting the following flags:

• --evaluator_max_value_byte_size
• --evaluator_max_intermediate_byte_size

Defaults are:

int64_t max_value_byte_size = 1024 * 1024;
int64_t max_intermediate_byte_size = 128 * 1024 * 1024;

Let me know if this works for your use case.

from differential-privacy.

Closing this, as we have a workaround with the above flags. Documentation changes are in the pipeline to be pushed soon.

from differential-privacy.