david-furminieux / tinypy Goto Github PK

It would be great if there exist windows binaries to download. So a user
can directly start playing with tinypy...

tinypy contains several references to the CPYTHON_MOD define in the source.
 This is done to work-around several issues.  I want to investigate if this
is really needed, as I'd prefer tinypy to be just flexible enough to embed
in this way without special handling.

(That way, tinypy will be clear for embedding in other places, most likely.)

I'd like this checked out before the release if possible.

What steps will reproduce the problem?
1. Save "    \n?" to test.py
2. Run "./build/tinypy ./tinypy/py2bc.py test.py test.tpc"

What is the expected output? What do you see instead?

tinypy should output an exception regarding a tokenization error (something
like 

"""
Exception:
error: tokenize
   2: ?"
"""

Instead, tinypy crashes.

What version of the product are you using? On what operating system?

Tested on Windows XP SP2, against latest revision (r74), compiled with mingw.

Please provide any additional information below.

The crash doesn't happen when running the file directly through tinypy,
only when running it through py2bc.py. Also, it doesn't happen when there
is no indentation prior to the question mark character.

What steps will reproduce the problem?
1. Run the following code:

foo = "abc" + chr(0) + "d"
print(foo)

What is the expected output? What do you see instead?

The expected output (as well as what Python 2.5 does) is "abc d\n" (since
the NUL char has no graphic representation). Instead, tinypy prints "abc\n".

What version of the product are you using? On what operating system?

Confirmed on both Slackware Linux 12.1 and Windows XP SP2 (and really on
any platform since the problem is algorithmic) using the latest tinypy
revision (r53). 

Please provide any additional information below.

The proposed resolution is to use fwrite() instead of printf() to print
strings since the length of the string is known in advance (stored inside
tp_string_ member of tp_obj). Patch included.

What steps will reproduce the problem?

 run the following code :

  print(len(range(2**32 + 1)))

What is the expected output? What do you see instead?

  I think we should get an overflow error (list can't be longer than the
max int value), instead we get '0'

What version of the product are you using? On what operating system?

  svn version : trunk@157, on ubuntu using a 64bit computer


Please provide any additional information below.

  in the code of tp_range (builtins.c), we convert the arguments from
double to int without any check.

What steps will reproduce the problem?
print(2*3**2)

What is the expected output? What do you see instead?
18 is expected, but the result produced by tp is 36. It means that tp
interprets 2*3**2 as (2*3)**2. But it's clear that ** has higher priority
than *, so 2*3**2 should be interpreted as 2*(3**2). I think any other
expression like this will come across the same problem.

What version of the product are you using? On what operating system?
tinypy svn revision 24 and 25.

Please provide any additional information below.
NA.

What steps will reproduce the problem?
- Call range() with mischievous parameters:
- range(0,1,0)
- range(0,5,3)

What is the expected output? What do you see instead?
- Expect a valid list (maybe empty), instead vm consumes vast amounts of 
memory as very long lists are built (can be an infinite loop until RAM is 
exhausted).

What version of the product are you using? On what operating system?
- r3, Mac OS X / 10.4.11 / PPC.

Please provide any additional information below.
- Patch with unit tests and fix will be appended to this issue.

Native Visual Studio build would be nice. Attached is a patch that fixes
all Visual Studio compilation problems (tested with VS 2005) and adds
native win build support to build.py

The fixes are many but rather trivial and mostly fall into 2 categories:
* this is C code, so variables need to be defined at the beginning of the scope
* vs doesn't like tp_obj iniatilizations in the form {TP_*, ...}, so they
need to be converted to more explicit "fill the object" approach

What steps will reproduce the problem?
1. "print 'OK'"

What is the expected output? What do you see instead?
It should raise an exception since this isn't supported tinypy syntax. 
Instead it does nothing.  (It parses this as "print; 'OK'")

Here's a test case that should work:
t_render("""print 'OK'""","Exception")

I'm unsure if tp->strings still serves any purpose.  Please verify that it
does, and if not, remove it.

Also, it might be better if tp_string_slice used tp_string_sub instead of
tp_string_copy.  Please verify with memory / speed tests if this is the case.

These aren't the most important issues in the world, but they should be
checked eventually.  I've assigned this task to myself, but if someone else
has thoughts, I'd be glad to hear them.

What steps will reproduce the problem?

for example, running the following script：
-------------------------
num = 0
num = num | 1
print(num)

num = 0
num |= 1
print(num)

num = 0
num &= 1
print(num)
-------------------------

What is the expected output? What do you see instead?

The expected results should be:
1
1
0
But tinypy output exceptions for those bitwise operators. For | and |=, the
exception is:
Exception:
error: tokenize
   6: num |= 1
          ^
For &= operator, the exception is:
Exception:
tp_get: KeyError: nud


Please use labels and text to provide additional information.

The problem jump out in revision 74 of tinypy. I haven't tested it on other
versions. But I guess the problem still exists on previous versions.

Besides, My operation system is  Microsoft Windows XP [version 5.1.2600].
And I compiled tinypy under mingw.

I want setup.py to work more like this:

To build / install the Cpython API, things should work exactly how they
work for any other python module in the world:

python setup.py build
python setup.py install

To build tinypy stuff:
python setup.py tinypy (right now a platform must be specified, this should
be auto detected from sys.platform instead)
python setup.py blob (this works already)
python setup.py 64k (this works already)

Options are things like (these already work)
debug, test, boot, [optional modules to compile in]

To specify an alternate platform, the user should be able to do something
like: 
python setup.py tinypy -cmingw32

What steps will reproduce the problem?

Add this to tests.py:

    #test that the __main__ module doesn't get GC'd
    t_render("""
MODULES["__main__"] = None
for n in range(0,50000):
    x = [n]
print("OK")
""","OK")

What is the expected output? What do you see instead?

It should say "OK" instead it gives some sort of error message.  For example: 

Exception:
tp_step: invalid instruction 105'

This is because since the main reference to the __main__ module has been
removed, the module gets garbage collected while it is being used.  

What steps will reproduce the problem?
1. in the top dir, execute:
python setup.py blob

2. then enter build/ subdir, execute:
gcc -o tinypy.o -c tinypy.c

What is the expected output? What do you see instead?

It shoud have compiled out tinypy.o normally, but it didn't, meanwhile it
complains:

will complain:

tinypy.c:6378: error: expected declaration specifiers or '...' before
numeric constant
tinypy.c:6378: error: conflicting types for 'calloc'
tinypy.c: In function 'calloc':
tinypy.c:6379: error: too many arguments to function 'calloc'
tinypy.c:6382: error: 'tp' undeclared (first use in this function)
tinypy.c:6382: error: (Each undeclared identifier is reported only once
tinypy.c:6382: error: for each function it appears in.)
tinypy.c: In function 'free':
tinypy.c:6392: error: 'tp' undeclared (first use in this function)
tinypy.c: In function 'realloc':
tinypy.c:6405: error: 'tp' undeclared (first use in this function)
tinypy.c:6414: error: too many arguments to function 'calloc'



Please use labels and text to provide additional information.

blob, embedding, libtinypy.a

If one runs `print(range(10))`, tinypython will raise an:

Exception:
tp_get: KeyError: 0

What steps will reproduce the problem?
1. Run the following code:

def foo():
#    global v
    print("v=", v)
    v = 0
    print("v=", v)

v = 42
foo()
print("v=", v)

What is the expected output? What do you see instead?

Python 2.5 throws "UnboundLocalError: local variable 'v' referenced before 
assignment" due to the first line of code in foo.  Instead, tinypy finds 
the variable in the global namespace eventhough the global keyword is not 
used.

Please use labels and text to provide additional information.

What steps will reproduce the problem?

1. Embed tinypy in your application.
2. Make your tinypy code raise an uncaught exception.
3. Try to debug the problem.

What is the expected output? What do you see instead?

I would like a full traceback in gdb to the line of code which caused the
problem. Instead I get nothing at all.

The attached patch fixes it by replacing tinypy's calls to exit() with abort().

The cpython module doesn't seem to have any unittests, examples or docs.

More information here:
http://groups.google.com/group/tinypy/browse_thread/thread/67908894bff0fba9

What steps will reproduce the problem?
1. $python setup.py build

What is the expected output? What do you see instead?
OK.

What version of the product are you using? On what operating system?
SVN r143

Please provide any additional information below.
I am running osx leopard 10.5.6 with gcc 4.0, and Fedora dist linux:
$ cat /proc/version
Linux version 2.6.26cslabs (root@domain) (gcc version 4.3.0 20080428 (Red Hat 
4.3.0-8) (GCC) ) #1 SMP Tue Sep 30 17:35:22 PDT 2008

What steps will reproduce the problem?

1: def foo(x, **y):
2:    print(x, y)
3: foo(1)

What is the expected output? What do you see instead?

I would expect something like: 1 {}
Instead, I get this misleading error:

File "a.py", line 3, in ?
  foo(1)
File "a.py", line 1, in foo
  def foo(x, **y):

Exception:
tp_get: KeyError: 0

What version of the product are you using? On what operating system?

Revision 29, Debian.

Please provide any additional information below.

I think I've read somewhere that you do not want to copy the Python
behavior for * and ** arguments due to added complexity, and that makes
sense - but the above case looks like an actual bug to me.

What steps will reproduce the problem?
1. Run the following code:

x = [1, 2, 3]

for i in x:
    if i != 0 and i:
        print("abc")

What is the expected output? What do you see instead?

The expected output is "abc". Instead, tinypy prints:

Exception:
assert failed

What version of the product are you using? On what operating system?

Tested on both Slackware Linux 10 and Windows XP SP2 with the latest
revision (r74).

Please provide any additional information below.

The bug does not show up if the tests are made in different order, i.e. 

"if i and i != 0:"

Also, the erroneous behaviour shows up regardless of what logical operator
is used (as far as I could figure) and the truthfulness of the expression
(happens in cases where it is false too). It does not happen with only a
single expression (only in expressions combined with 'or', 'and', etc). It
also doesn't happen when the conditional expression is done on a 'normal'
variable outside the loop.

See the attached file for the actual problem code & comments.

This is from unmodified SVN head TinyPy.

I would suggest that an exception is made for self.*:
*.__get__(x) should never be called if the caller is a function from the
class itself (caller == self).
It should ONLY be called if caller is from another class or global, unless
it is called explocitly (as in, "self.__get__(x)" actually written in code).

The same holds for __set__(x, v), of course.

I will try to make a patch for this, but no promises... I am not very
aquainted with the TinyPy internals (yet).

I've looked at build_vs in setup.py and there's a file generated called
mymain.c, but it's not used during compilation of the tinypy.exe...

What steps will reproduce the problem?
1. Run the following code:

foo = 2 * "abc"
print(foo)

What is the expected output? What do you see instead?

Expected output:
"abcabc"

tinypy output:
"
File "baby.py", line 1, in ?
  foo = 2 * "abc"

Exception:
tp_mul(2,abc)
"

What version of the product are you using? On what operating system?

Slackware 12.1, Windows XP SP2.

Please provide any additional information below.

The issue is again in how tp_mul() checks for argument types. I would
provide a patch but I didn't want to patch both this and the issue 19 in
the same patch so I'm waiting until the patch for issue 19 gets committed.

I tried 3 ways of printing, and tinypy doesn't match Python's behavior
(only the last one works).  I didn't see anything about this in the docs. 
In practice nearly all Python programs contain print statements, so this is
likely to be one of the first things that people run into when trying tinypy.

And if you didn't know print is a special case in Python 2.x (which is a
bit unfortunate and changing in Python 3000) -- it looks like you have
implemented it as a normal function.

(Very cool project BTW)

tinypy-read-only$ ./build/tinypy ../tinypy-test/hello.py


Exception:
error: invalid statement
   1: print 'hello world'
            ^

tinypy-read-only$ ./build/tinypy ../tinypy-test/hello2.py


Exception:
error: invalid statement
   1: print "hello world"
            ^

tinypy-read-only$ ./build/tinypy ../tinypy-test/hello3.py
hello world

tinypy-read-only$ cat ../tinypy-test/hello3.py
print("hello world")

What steps will reproduce the problem?
1. checkout tinypy r50
2. $ python setup.py linux
3. $ python setup.py linux test


What is the expected output? What do you see instead?

Expect a clean compile and clean test report.

Step #2 produces this error:
/usr/include/math.h:130:17: #if with no expression
/usr/include/math.h:366:17: #if with no expression

Step #3 produces this error:
<BEGIN>
../build/tinypy tests.py -linux
# t_boot
sh: line 1: ./tinypy: No such file or directory

File "tests.py", line 905, in ?
      t_boot(["def test(): print('OK')","import tmp1; tmp1.test()"],"OK")
File "tests.py", line 872, in t_boot
      if exact: assert(res == ex)

Exception:
assert failed
exit_status 65280
<END>

Please use labels and text to provide additional information.

produced on Linux 2.4.32 (Debian)

Details of a cpython script which can crash tinypy/python.
http://groups.google.com/group/tinypy/browse_thread/thread/9d7eac41fbe6b157

What steps will reproduce the problem?
1. downloaded wicd went to install was missing python
2. downloaded python would not install downloaded tinypy
3. got error expected missing ] error 422 lines= x.strip()for x in line

What is the expected output? What do you see instead?
thought tinypy would install wicd

What version of the product are you using? On what operating system?
latest

Please provide any additional information below.
not sure if this would work but wanted to install program in lieu of what
puppy linux is using

Hello,

there has always been a need (at least for me) to redirect the output of
print() to somewhere else than stdout, especially lately, as I was
programming PSP-related things with TinyPy.

Here, I submit 3 patches and 1 new file for this - they are based off the
latest SVN revision (R148).
I did not know how the functions and types should be named, and therefore
may be currently named rather confusingly - feel free to change this if you
do decide to put this into trunk.

The patches were tested on Windows with the default TinyPy runner (just to
verify that it still works as usual) and with with PSP (on which I did use
callbacks).

At the moment, the compiler complains about tp_stderr and tp_stdin being
defined and not used - I've just added them for future use, feel free to
comment them out.

In this test case, I'd expect the result to be "OK" .. Instead it just
repeats the previous exception.

>>> t = tinypy.Tinypy()
>>> t.execute("def x():x()\nx()")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
tinypy.error: (tp_frame) RuntimeError: stack overflow
>>> t.execute("print('OK')")
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
tinypy.error: (tp_frame) RuntimeError: stack overflow

What steps will reproduce the problem?
1. Try to build on OS/X
2. python setup.py linux
3. almost works

What is the expected output? What do you see instead?
I see complaints about -Wc++compat
Removing the flag from the compile options for linux yields a usable version

What version of the product are you using? On what operating system?
1.1

Please provide any additional information below.

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


Please use labels and text to provide additional information.

What steps will reproduce the problem?
1. Compile a project which uses something like below (i.e. all of mine :P):

#define True true
#define False false
#define None NULL

2. Make use of tinypy with the tinypy.h created from the setup.py blob
3. Compilation will fail

What is the expected output? What do you see instead?

Possibility to use my own symbols named True, False and None.

What version of the product are you using? On what operating system?

SVN trunk revision 16. Linux.

Please provide any additional information below.

If this can't be easily fixed, I'd also be grateful for any possible
workarounds.. maybe some clever #define trick when including tinypy.h? Not
being able to use these three symbols in my own code would be quite bad..

Would it be possible to fully support
compiling tinypy with the tcc compiler ?

See: http://bellard.org/tcc/

What steps will reproduce the problem?
1. Run the following code:

foo = "abc" * -1

What is the expected output? What do you see instead?

There is no expected output, but the program should finish gracefully.
Instead, it crashes tinypy.

If the value of foo is subsequently printed with the print() builtin,
tinypy outputs junk (probably random memory dump) to stdout.

What version of the product are you using? On what operating system?

Tested and confirmed on Slackware 12.1 and Windows XP SP2.

Please provide any additional information below.

The problem is that negative values aren't sanitised in tp_mul() when
allocating storage for the new string. I have modified tp_mul() to return
an empty string when multiplied by a negative value (like bigpy does).  

Here the code.

1 try:
2     raise "a"
3 except e:
4     print e

What is the expected output? What do you see instead?
Should print ...

| a

Instead, this exception occured :

| (_tp_dict_get) KeyError: e

at line of 4

What version of the product are you using? On what operating system?
Version : from SVN, revision 146
OS : Linux Fedora

Please provide any additional information below.
As you understood, e does not exist in the "except" context. I cannot get
the catched exception.

(I'm using SVN trunk.)

The 'not' operator only works for numbers.  The following code works as
expected and prints "OK":

  n = 0
  if not n:
    print('OK')

However, it doesn't work for empty dicts, lists, or strings.  The following
code fails to print "OK" for any of the tests:

  d = {}
  if not d:
    print('OK')
  l = []
  if not l:
    print('OK')
  s = ''
  if not s:
    print('OK')

Internally, the bytecode that gets generated is:
  NUMBER 0
  STRING "s"
  GGET
  EQ
(The responsible code is encode.py, line 248.)

---

I've only just started looking at tinypy and the internals.  If the
internal tp_bool() function (in ops.c) were to be exposed (say, as a
builtin named 'bool'), then 'not' could be implemented as:
  NUMBER 0
  STRING "s"
  GGET
  call bool() here
  EQ
...which would behave as expected.

I'm planning on submitting a patch to do this in the next few days.  I was
surprised to see that there isn't actually a bool type in tinypy.  What are
your views on adding it?  I am rather tempted to submit a patch for this as
well...

Here's a new patch to make the API const correct - but this time I
identified the one position where it's appropriate to cast away the
constness. I still made tp_string_.val const as it seemed the right thing
to do.

What steps will reproduce the problem?
1. Run the following code:

while 1:
print("blah")

What is the expected output? What do you see instead?

tinypy should raise an IndentationError exception because there is an
indentation block missing before the print statement. Instead, tinypy
executes the code normally.

Please use labels and text to provide additional information.

What steps will reproduce the problem?
1. "test-".index("=")

What is the expected output? What do you see instead?
CPython gives "ValueError: substring not found" almost immediately.
tinypy hangs for a few seconds and says "Bus error"

It looks like the search isn't terminating at the end of the string. The
code for tp_index seems to want to return -1 for substrings that can't be
found, but for some reason it doesn't get there.

What steps will reproduce the problem?
1. first build using "python setup.py linux",
2. now try to do ./build/tinypy,
3.

What is the expected output? What do you see instead?
Exception
tp_get: KeyError: 0

What version of the product are you using? On what operating system?
1.1, Ubuntu 9.10 X86_64, Python 2.6.4, 4.4.1

Please provide any additional information below.
The same thing happens for 64k classic version of tinypy

There isn't an 'osx' target for tinypy, and building to the 'linux' target
fails with an error of 'error: unrecognized command line option
"-Wc++-compat"'.  Removing that option if sys.platform is darwin fixes
things, and "python setup.py linux boot" works fine when this is the case.

This patch implements the fix.  It's hackish, but it seems to make things
work on my system.

Using a white list instead of a blacklist for sandbox stuff.

see this thread for more info:

http://groups.google.com/group/tinypy/browse_thread/thread/5001ec89e1bbaab1/2e4e
710d95ac638f

hi,

In here, some of the function calls aren't checking for errors:
http://code.google.com/p/tinypy/source/browse/branches/sandbox/modules/cpython/c
python.c

Need to go over each of the CPython C API function calls, and check for errors.

eg, the return value of PyList_Append is not checked.



also... in:
Tinypy_init(TinypyObject *self, PyObject *args, PyObject *kwds)

Need to do input validation.  This function uses signed values for time,
and memory limits.  Is this needed?  I think checking for unsigned values
would be better.



cheers,

hi,

vsprintf doesn't check the length of the string, whereas vsnprintf does.

So it's better to use vsnprintf instead, to avoid buffer overruns.

string.c:53


cheers,

Posting the patch here where I can attach files (can't in the google groups
list) :)

I wanted to use this, but it turns out you use longjmp... It's a pity. Many
scripting languages seem perfectly capable of handling exceptions without
setjmp/longjmp. Anyway, if I may make a feature request, that'd be it. I've
been looking for a reasonably small Python implementation for a long time.

Here's a test that reproduces the issue:

t_render("def f():pass\n"+("f();"*256)+"print('OK')","OK")

There is no good reason for all the registers to run out in this example.

My earlier patch to check for duplicated prototypes had a flaw for multiple
functions starting with the same name - this patch fixes it.

Here is a patch for setup.py to build tinypy 1.1 with SDL on MacOS X using 
command

% python setup.py macosx [pygame].

Two files are attached, File setupX.py is the modified setup.py file and file  
setupX.py.diff is the 
forward diff with the original setup.py file from tinypy-1.1.

Note the comment about installing SDL for MacOS X inside the build_macosx 
function.

Disclaimer.  Tested only on MacOS X 10.4.11 (Intel) using Apple's gcc version 
4.0.1 (Apple 
Computer, Inc. build 5367) and Python 2.5.2 from ActivePython 2.5.2.2 
(ActiveState Software 
Inc.).

/Jean Brouwers