`DataFrame.top_k` not handling nulls correctly in version 1.0.0-rc.1 about polars HOT 3 CLOSED

Thanks for the report.

The functionality was completely redone in #16804 including the new design with regards to nulls. Because the intended functionality was not present in earlier versions, this does not classify as a regression.

@orlp FYI

from polars.

Note that this is DataFrame top_k, which I believe is similar to sort().head() and Nulls have an ordering. I believe this is as expected.

from polars.

Note that this is DataFrame top_k, which I believe is similar to sort().head() and Nulls have an ordering. I believe this is as expected.

This really is a bug, at least according to the design Orson and I discussed. Null values come last, regardless of whether you're using top_k, or bottom_k, or top_k with reverse.

It's even easier to see the bug in the non-reverse case:

import polars as pl

res = pl.DataFrame(
    {
        "a": ["a", "b", "c", "d"],
        "b": [None, None, 1, 3],
    }
).top_k(k=2, by="b")
print(res)

shape: (2, 2)
┌─────┬──────┐
│ a   ┆ b    │
│ --- ┆ ---  │
│ str ┆ i64  │
╞═════╪══════╡
│ a   ┆ null │
│ b   ┆ null │
└─────┴──────┘

Clearly, values 1, 3 are the top values in b and those rows should be selected.

from polars.