Python 3.6 opcodes about python-uncompyle6 HOT 16 CLOSED

@rocky I will take a look at this

from python-uncompyle6.

@moagstar many thanks. I really appreciate this.

from python-uncompyle6.

The following links provide some more information about these opcodes and can probably be used as a starting point for writing some test cases.

BUILD_STRING
Issue 27078

STORE_ANNOTATION and SETUP_ANNOTATIONS
Issue 27985
PEP 526

CALL_FUNCTION_EX
Issue 27213

BUILD_TUPLE_UNPACK_WITH_CALL
Issue 28257

from python-uncompyle6.

Thanks for the link. xdis now has a BUILD_STRING test. I'm pretty sure you'd handle that like BUILD_LIST.

from python-uncompyle6.

@rocky I'm looking into adding support for type annoations. Reading this it seems that SETUP_ANNOTATIONS doesn't actually do anything useful in terms of decompilation (however STORE_ANNOTATION should decompile the appropriate type annotation!)

Is it possible with the spark parser to ignore an opcode? This was my first attempt but I still get a Parse error at or near `SETUP_ANNOTATIONS' exception.

from python-uncompyle6.

Note that semantic of MAKE_FUNCTION (issue27095), CALL_FUNCTION, CALL_FUNCTION_KW and BUILD_MAP_UNPACK_WITH_CALL (issue27213) was changed.

from python-uncompyle6.

@serhiy-storchaka thanks for the links. Having these make it easier to see what's going on. I think handling changed opcodes is going to be a bit of work. Perhaps you could help here?

from python-uncompyle6.

Here are differences between 3.5 and 3.6:

• BUILD_MAP_UNPACK_WITH_CALL: In 3.5 oparg is a 16-bit value consisting of the number of unpacked mappings on the stack (bits 0-7) and the location of the function (bits 8-15). In 3.6 oparg is just the number of unpacked mappings which no longer limited by 255, as in BUILD_MAP_UNPACK.The location of the function is oparg + 2.

• CALL_FUNCTION: In 3.5 oparg is a 16-bit value consisting of the number of positional arguments on the stack (bits 0-7) and the number of keyword arguments on the stack (bits 8-15). In 3.6 oparg is just the number of positional arguments on the stack which no longer limited by 255.

• CALL_FUNCTION_KW in 3.5 and 3.6 are totally different opcodes that just have the same name. In 3.5 it is like CALL_FUNCTION but with var-keyword mapping pushed on the top of the stack. In 3.6 it is like 3.6 CALL_FUNCTION but values of keyword arguments are pushed on the stack after values of positional arguments, and then a constant tuple of keyword names is pushed on the top of the stack. oparg is the sum of numbers of positional and keyword arguments. The number of keyword arguments is determined by the length of the tuple of keyword names.

• CALL_FUNCTION_EX takes 2 to 3 arguments on the stack: the function, the tuple of positional arguments, and optionally the dict of keyword arguments if bit 0 of oparg is 1.

• MAKE_FUNCTION in 3.5 and 3.6 are totally different opcodes that just have the same name. In 3.5 oparg is a 32-bit value consisting of the number of the number of default values for positional-or-keyword parameters (bits 0-7), the number of default values for keyword-only parameters (bits 8-15), and the number of annotations (bits 16-31). Values on the stack are: default values for positional-or-keyword parameters, name - default value pairs for keyword-only parameters, annotations, a tuple of names of annotations (only if the number of annotations is > 0), the code object, and the qualified name of the function. In 3.6 oparg is a set of flags. The tuple of default values for positional-or-keyword parameters, the dict of default values for keyword-only parameters, the dict of annotations and the closure are pushed on the stack if corresponding bit (0-3) is set. They are followed by the code object and the qualified name of the function.

from python-uncompyle6.

@serhiy-storchaka what simple Python code would trigger BUILD_MAP_UNPACK_WITH_CALL in Python 3.6? in Python 3.5?

from python-uncompyle6.

from python-uncompyle6.

CALL_FUNCTION_EX.zip

from python-uncompyle6.

@x0ret Are we handling STORE_ANNOTATiON? I believe this is the last of the 3.6 opcodes that needed looking at.

from python-uncompyle6.

No we don't.

$ python -V
Python 3.6.7
$ cat test14.py
x: int = 1
$ python -m compileall test14.py
Compiling 'test14.py'...
$ uncompyle6 __pycache__/test14.cpython-36.pyc
... 
Parse error...

And code inspection seems to verify that. I'll take a look.

from python-uncompyle6.

@rocky, I'm pretty sure i'm still new in case of editing parser. Anyway please take a look at following diff, for supporting STORE_ANNOTATION and SETUP_ANNOTATION. This may not so good.

diff --git a/uncompyle6/parsers/parse36.py b/uncompyle6/parsers/parse36.py
index f87da84b..2c640f93 100644
--- a/uncompyle6/parsers/parse36.py
+++ b/uncompyle6/parsers/parse36.py
@@ -143,6 +143,12 @@ class Python36Parser(Python35Parser):
                                    COME_FROM_FINALLY

         compare_chained2 ::= expr COMPARE_OP come_froms JUMP_FORWARD
+
+        stmt ::= SETUP_ANNOTATIONS
+        stmt ::= annotated_assign
+        # stmt ::= store_annotation
+        annotated_assign ::= expr store store_annotation
+        store_annotation ::= LOAD_NAME STORE_ANNOTATION
         """

     def customize_grammar_rules(self, tokens, customize):
diff --git a/uncompyle6/semantics/customize36.py b/uncompyle6/semantics/customize36.py
index 2245e648..84777f45 100644
--- a/uncompyle6/semantics/customize36.py
+++ b/uncompyle6/semantics/customize36.py
@@ -60,6 +60,13 @@ def customize_for_version36(self, version):
         'call_ex' : (
             '%c(%p)',
             (0, 'expr'), (1, 100)),
+        'store_annotation': (
+            '%|%[1]{pattr}: %c',
+            0
+            ),
+        'annotated_assign':  (
+            '%|%c = %p\n',
+             (-1, 'store_annotation'), (0, 200))

     })

test cases:

x: int = 1
#z: int

About z: int i don't know if it is rational or not! That is why i commented that grammer.

Python 3.6.7 (default, May 13 2019, 15:17:31)
>>> z: int
>>> z
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'z' is not defined

from python-uncompyle6.

@x0ret Thanks. A first read and this looks good! I would like to spend more time going over details and right now I have other work to that needs to be done in the next couple of days. (And with #254 you are making me work hard - have pity on an old guy.)

Maybe you should create a branch and do the PR thing. Thanks again for all the help.

from python-uncompyle6.

Oh, Sorry about that. I'm going to do my best to handle cases more accurate.

Maybe you should create a branch and do the PR thing.

Sure.

from python-uncompyle6.