User-defined functions documentation doesn't tell reader how to write fast functions about polars HOT 11 CLOSED

I will try to work on this. Depending how long it gets this might end up being a series of issues+PRs.

from polars.

There's a (somewhat) related PR open already: #13392

from polars.

Thanks! I'll take the comments and info there into account. But at this point I'm contemplating a much more significant rewrite, given these APIs are so tricky to use correctly.

from polars.

More problems: the documented behavior of map_batches() on this page doesn't match the demonstrated behavior (or perhaps the underlying behavior?). For example, it says:

Ouch.. we clearly get the wrong results here. Group "b" even got a value from group "a" 😵.

Except the actual output in the documentation isn't the wrong results, and group "b" does not in fact have values from group "a"...

from polars.

yeah it needs updating since #13181

from polars.

OK so what are the expected semantics of map_batches()? Is the batch is always the original series (in select()) or always the group (in group_by())?

from polars.

that looks right

(.venv) marcogorelli@DESKTOP-U8OKFP3:~/scratch$ cat t.py
import polars as pl

def func(x):
    print('batch is: ', x)
    return x

df = pl.DataFrame({'group': ['a', 'a', 'b'], 'value': [1, 2, 3]})

df.select(pl.col('value').map_batches(func))
df.select(pl.col('value').map_batches(func).over('group'))
(.venv) marcogorelli@DESKTOP-U8OKFP3:~/scratch$ python t.py
batch is:  shape: (3,)
Series: 'value' [i64]
[
        1
        2
        3
]
batch is:  shape: (2,)
Series: '' [i64]
[
        1
        2
]
batch is:  shape: (1,)
Series: '' [i64]
[
        3
]

from polars.

If that's the case, my first inclination is to not document map_elements() at all in the user guide, nor refer to it in API docs for map_batches()? Since the fact map_elements() sometimes takes a single element and sometimes takes a whole Series seems problematic. Or am I missing something?

from polars.

Somewhat related:

• #14521

from polars.

I guess the complexity argument in #14521 suggests one should only use map_batches() for the UDF docs, and as suggested in #14521 document the different APIs in its whole own document.

from polars.

If you exclude groups and you have func which takes a python scaler and returns a scaler then

then

df.select(pl.col('a').map_elements(func))

Is mostly there same as

df.select(pl.col('a')
.map_batches(lambda x : (
pl.Series([func(y) for y in x])
))
)

So it's just a convenience shortcut really.

from polars.