Sampled partition stats have greatly underestimated NDVs for some columns about trino HOT 4 OPEN

It depends for what column, and what type of partitions. For daily rolling partitions NDV will be likely overlapping.

Yup. Discussion from original issue: let's say you partition by date and you have two data columns:

• userId
• eventId

Then for 1) you should rather take max (users come from the same id pool), even though userId NDV could be approximate the same as partition row count
For 2) you should sum because each eventId is unique.

from trino.

Should we add extrapolate NDVs instead?

Currently we chose the maximum NDV. If we decided to sum NDV's then we would need to extrapolate.

It seems that partitions might often be different chunks of data so that NDVs don't overlap.

It depends for what column, and what type of partitions. For daily rolling partitions NDV will be likely overlapping.

Alternatively we could store HLL state per column as an auxiliary partition property and calculate extrapolation based on merged HLLs

I think that's the only options we have. In order to do so - we need to store HLL states per columns in the partition properties, as the Mestastore API doesn't allow you to store arbitrary statistics.

from trino.

@findepi I'm not sure if we should call it a "bug". The decision of taking MAX NDV was a thoughtful decision. Let's change the label to "enhancment".

from trino.

I think that's the only options we have. In order to do so - we need to store HLL states per columns in the partition properties, as the Mestastore API doesn't allow you to store arbitrary statistics.

We could store HLL in table/partition properties

from trino.