Parameter parsing fails for strings containing `=` character about papermill HOT 2 OPEN

For now it's a constraint of the implementation it seems. We could improve that '=' check to check after doing quoted string capture groups to get around it. But someone would need to write a fairly good suite of unittests to ensure it works correctly in all cases (harder problem to get right than it may seem).

from papermill.

Hello,

would it be possible to use the built-in ast package to parse the python code?
https://docs.python.org/3/library/ast.html

I saw some comments from ~2020 about this not being compatible with other languages, but I would argue it's a safer approach than the regex version and would catch corner cases that are not yet properly supported by the regex, such as the one shared here or if the type provided contains a "." (cf sample below)

Sample parameter cell

import datetime as dt

string_param: str = ""
equal_param = "./all/date=20240102/ml_flow_run_id=Y/model/"
multi_line_param: str = "test1" + "test2"
date_param: dt.datetime = dt.datetime(2024, 4, 17, 0, 0, 0)
a = b = 2
list_param = []
dict_param = {
    "key1": "value",
    "key2": [1, 2, 3],
    "key3": "=",
}

Sample parsing code:

# param_code is a variable containing the above cell's source code
tree: ast.Module = ast.parse(param_code)
for node in ast.walk(tree):
    if isinstance(ann_assign := node, ast.AnnAssign) and isinstance(
            name := ann_assign.target, ast.Name
        ):
            print(name.id)
    elif isinstance(assign := node, ast.Assign):
        for target in assign.targets:
            if isinstance(target, ast.Name):
                print(target.id)

I get the following output:

string_param
equal_param
multi_line_param
date_param
a
b
list_param
dict_param

@MSeal: if that would be of interest to you, I can propose a PR for this :)

from papermill.