pypr / compyle Goto Github PK

Execute a subset of Python on HPC platforms

License: Other

Python 99.88% Cython 0.12%

python high-performance-computing transpile openmp opencl cuda cython

compyle's Introduction

Compyle: execute a subset of Python on HPC platforms

Compyle allows users to execute a restricted subset of Python (almost similar to C) on a variety of HPC platforms. Currently we support multi-core CPU execution using Cython, and for GPU devices we use OpenCL or CUDA.

Users start with code implemented in a very restricted Python syntax, this code is then automatically transpiled, compiled and executed to run on either one CPU core, or multiple CPU cores (via OpenMP) or on a GPU. Compyle offers source-to-source transpilation, making it a very convenient tool for writing HPC libraries.

Some simple yet powerful parallel utilities are provided which can allow you to solve a remarkably large number of interesting HPC problems. Compyle also features JIT transpilation making it easy to use.

Documentation and learning material is also available in the form of:

Documentation at: https://compyle.readthedocs.io
An introduction to compyle in the context of writing a parallel molecular dynamics simulator is in our SciPy 2020 paper.
Compyle poster presentation
You may also try Compyle online for free on a Google Colab notebook.

While Compyle seems simple it is not a toy and is used heavily by the PySPH project where Compyle has its origins.

Installation

Compyle is itself largely pure Python but depends on numpy and requires either Cython or PyOpenCL or PyCUDA along with the respective backends of a C/C++ compiler, OpenCL and CUDA. If you are only going to execute code on a CPU then all you need is Cython.

You should be able to install Compyle by doing:

$ pip install compyle

A simple example

Here is a very simple example:

from compyle.api import Elementwise, annotate, wrap, get_config
import numpy as np

@annotate
def axpb(i, x, y, a, b):
    y[i] = a*sin(x[i]) + b

x = np.linspace(0, 1, 10000)
y = np.zeros_like(x)
a, b = 2.0, 3.0

backend = 'cython'
get_config().use_openmp = True
x, y = wrap(x, y, backend=backend)
e = Elementwise(axpb, backend=backend)
e(x, y, a, b)

This will execute the elementwise operation in parallel using OpenMP with Cython. The code is auto-generated, compiled and called for you transparently. The first time this runs, it will take a bit of time to compile everything but the next time, this is cached and will run much faster.

If you just change the backend = 'opencl', the same exact code will be executed using PyOpenCL and if you change the backend to 'cuda', it will execute via CUDA without any other changes to your code. This is obviously a very trivial example, there are more complex examples available as well.

Examples

Some simple examples and benchmarks are available in the examples directory.

You may also run these examples on the Google Colab notebook

compyle's People

Contributors

Stargazers

Watchers

Forkers

adityapb abhinavmuta dineshadepu pawansnegi prajwal-prathiksh manish364824 amwi04 blq5 aerorohit harshbaldwa jiaming1999 nauaneed arunkv7 avalentino saidctb

compyle's Issues

Test failure with Python 3.9

Running the test suite og compyle 0.7 with Python 3.9 I get 6 errors.
Looking at the release notes of Python 3.9 I see that a new parser has been introduced but, as far as I can understand, the ast module should not be affected by the change.

Update: in [1] it is suggested to export PYTHONOLDPARSER=1 to force the use of the old parser.
I can confirm that anso in this case I get the same results.

[1] https://docs.python.org/3/whatsnew/3.9.html#new-parser

============================= test session starts ==============================
platform linux -- Python 3.9.0, pytest-4.6.11, py-1.9.0, pluggy-0.13.0
rootdir: /build/compyle-0.7
collected 295 items / 4 deselected / 291 selected

compyle/tests/test_array.py .ss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss.ss..ss. [ 15%]
ss.ss                                                                    [ 16%]
compyle/tests/test_ast_utils.py .......                                  [ 19%]
compyle/tests/test_capture_stream.py .....                               [ 20%]
compyle/tests/test_config.py ............                                [ 25%]
compyle/tests/test_cython_generator.py .................                 [ 30%]
compyle/tests/test_ext_module.py ......                                  [ 32%]
compyle/tests/test_gpu_struct.py s                                       [ 33%]
compyle/tests/test_jit.py .......................                        [ 41%]
compyle/tests/test_low_level.py ssss....                                 [ 43%]
compyle/tests/test_parallel.py s..ss.s.ss.s.ss.ss.ss.sss..ss..ss..ss..ss [ 58%]
..ss..ss.s.ss.s.ss.ss.ss.sss..ss..ss..ss..ss..s                          [ 74%]
compyle/tests/test_profile.py ..s                                        [ 75%]
compyle/tests/test_template.py .....                                     [ 76%]
compyle/tests/test_translator.py ...........F.....F...................F. [ 90%]
......FF.F....                                                           [ 95%]
compyle/tests/test_transpiler.py ...                                     [ 96%]
compyle/tests/test_types.py .......                                      [ 98%]
compyle/tests/test_utils.py ....                                         [100%]

=================================== FAILURES ===================================
________________________________ test_subscript ________________________________

    def test_subscript():
        # Given
        src = dedent('''
        x[1]
        ''')
    
        # When
>       code = py2c(src)

compyle/tests/test_translator.py:282: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
compyle/translator.py:65: in py2c
    result = converter.convert(src)
compyle/translator.py:252: in convert
    result = self.visit(code)
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:652: in visit_Module
    return '\n'.join(
compyle/translator.py:653: in <genexpr>
    self.visit(item) for item in node.body
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:438: in visit_Expr
    return self.visit(node.value) + ';'
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:704: in visit_Subscript
    self.visit(node.value), self.visit(node.slice.value)
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
/usr/lib/python3.9/ast.py:411: in generic_visit
    for field, value in iter_fields(node):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

node = 1

    def iter_fields(node):
        """
        Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields``
        that is present on *node*.
        """
>       for field in node._fields:
E       AttributeError: 'int' object has no attribute '_fields'

/usr/lib/python3.9/ast.py:249: AttributeError
___________________________ test_annotated_function ____________________________

    def test_annotated_function():
        # Given/When
        t = CConverter()
>       code = t.parse_function(annotated_f)

compyle/tests/test_translator.py:411: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
compyle/translator.py:307: in parse_function
    code = self.convert(src)
compyle/translator.py:252: in convert
    result = self.visit(code)
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:652: in visit_Module
    return '\n'.join(
compyle/translator.py:653: in <genexpr>
    self.visit(item) for item in node.body
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:584: in visit_FunctionDef
    body = '\n'.join(self._indent_block(self.visit(item))
compyle/translator.py:584: in <genexpr>
    body = '\n'.join(self._indent_block(self.visit(item))
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:692: in visit_Return
    return 'return %s;' % (self.visit(node.value))
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <compyle.translator.CConverter object at 0x7fece8274520>
node = <ast.Subscript object at 0x7fece8274610>

    def visit_Subscript(self, node):
        return '%s[%s]' % (
>           self.visit(node.value), self.visit(node.slice.value)
        )
E       AttributeError: 'Name' object has no attribute 'value'

compyle/translator.py:704: AttributeError
_____________________________ test_declare_matrix ______________________________

    def test_declare_matrix():
        # Given
        src = dedent('''
        x = declare('matrix((3,))')
        do(x[0])
        ''')
    
        # When
>       code = py2c(src)

compyle/tests/test_translator.py:950: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
compyle/translator.py:65: in py2c
    result = converter.convert(src)
compyle/translator.py:252: in convert
    result = self.visit(code)
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:652: in visit_Module
    return '\n'.join(
compyle/translator.py:653: in <genexpr>
    self.visit(item) for item in node.body
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:438: in visit_Expr
    return self.visit(node.value) + ';'
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:392: in visit_Call
    args=', '.join(self.visit(x) for x in node.args)
compyle/translator.py:392: in <genexpr>
    args=', '.join(self.visit(x) for x in node.args)
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:704: in visit_Subscript
    self.visit(node.value), self.visit(node.slice.value)
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
/usr/lib/python3.9/ast.py:411: in generic_visit
    for field, value in iter_fields(node):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

node = 0

    def iter_fields(node):
        """
        Yield a tuple of ``(fieldname, value)`` for each field in ``node._fields``
        that is present on *node*.
        """
>       for field in node._fields:
E       AttributeError: 'int' object has no attribute '_fields'

/usr/lib/python3.9/ast.py:249: AttributeError
_____________________________ test_cuda_conversion _____________________________

    def test_cuda_conversion():
>       check_opencl_cuda_conversion(CUDAConverter)

compyle/tests/test_translator.py:1218: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
compyle/tests/test_translator.py:1204: in check_opencl_cuda_conversion
    code = converter.convert(src)
compyle/translator.py:252: in convert
    result = self.visit(code)
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:652: in visit_Module
    return '\n'.join(
compyle/translator.py:653: in <genexpr>
    self.visit(item) for item in node.body
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:584: in visit_FunctionDef
    body = '\n'.join(self._indent_block(self.visit(item))
compyle/translator.py:584: in <genexpr>
    body = '\n'.join(self._indent_block(self.visit(item))
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:355: in visit_Assign
    return '%s = %s;' % (self.visit(left), self.visit(right))
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <compyle.translator.CUDAConverter object at 0x7fece768c850>
node = <ast.Subscript object at 0x7fece7827d90>

    def visit_Subscript(self, node):
        return '%s[%s]' % (
>           self.visit(node.value), self.visit(node.slice.value)
        )
E       AttributeError: 'Name' object has no attribute 'value'

compyle/translator.py:704: AttributeError
____________________________ test_opencl_conversion ____________________________

    def test_opencl_conversion():
>       check_opencl_cuda_conversion(OpenCLConverter)

compyle/tests/test_translator.py:1222: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
compyle/tests/test_translator.py:1204: in check_opencl_cuda_conversion
    code = converter.convert(src)
compyle/translator.py:252: in convert
    result = self.visit(code)
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:652: in visit_Module
    return '\n'.join(
compyle/translator.py:653: in <genexpr>
    self.visit(item) for item in node.body
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:584: in visit_FunctionDef
    body = '\n'.join(self._indent_block(self.visit(item))
compyle/translator.py:584: in <genexpr>
    body = '\n'.join(self._indent_block(self.visit(item))
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:355: in visit_Assign
    return '%s = %s;' % (self.visit(left), self.visit(right))
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <compyle.translator.OpenCLConverter object at 0x7fece1908310>
node = <ast.Subscript object at 0x7fece19082b0>

    def visit_Subscript(self, node):
        return '%s[%s]' % (
>           self.visit(node.value), self.visit(node.slice.value)
        )
E       AttributeError: 'Name' object has no attribute 'value'

compyle/translator.py:704: AttributeError
__________________________ test_cuda_local_conversion __________________________

    def test_cuda_local_conversion():
        @annotate(xc='ldoublep', yc='lintp')
        def knl(xc, yc):
            xc[LID_0] = 1
            yc[LID_0] = 1
    
        # When
        converter = CUDAConverter()
>       code = converter.parse(knl)

compyle/tests/test_translator.py:1254: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
compyle/translator.py:281: in parse
    code = self.parse_function(obj, declarations=declarations)
compyle/translator.py:307: in parse_function
    code = self.convert(src)
compyle/translator.py:252: in convert
    result = self.visit(code)
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:652: in visit_Module
    return '\n'.join(
compyle/translator.py:653: in <genexpr>
    self.visit(item) for item in node.body
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:584: in visit_FunctionDef
    body = '\n'.join(self._indent_block(self.visit(item))
compyle/translator.py:584: in <genexpr>
    body = '\n'.join(self._indent_block(self.visit(item))
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
compyle/translator.py:355: in visit_Assign
    return '%s = %s;' % (self.visit(left), self.visit(right))
/usr/lib/python3.9/ast.py:407: in visit
    return visitor(node)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <compyle.translator.CUDAConverter object at 0x7fece7b86640>
node = <ast.Subscript object at 0x7fece80d05b0>

    def visit_Subscript(self, node):
        return '%s[%s]' % (
>           self.visit(node.value), self.visit(node.slice.value)
        )
E       AttributeError: 'Name' object has no attribute 'value'

compyle/translator.py:704: AttributeError
=============================== warnings summary ===============================
compyle/parallel.py:928
  /build/compyle-0.7/.pybuild/cpython3_3.9_compyle/build/compyle/parallel.py:928: SyntaxWarning: "is" with a literal. Did you mean "=="?
    if func_type is 'input':

compyle/parallel.py:1035
  /build/compyle-0.7/.pybuild/cpython3_3.9_compyle/build/compyle/parallel.py:1035: SyntaxWarning: "is" with a literal. Did you mean "=="?
    if func_type is 'input':

compyle/parallel.py:1150
  /build/compyle-0.7/.pybuild/cpython3_3.9_compyle/build/compyle/parallel.py:1150: SyntaxWarning: "is" with a literal. Did you mean "=="?
    return_type = 'void' if func_type is 'output' else self.type

.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_array.py::test_radix_sort_by_keys
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_array.py::test_radix_sort_by_keys
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_array.py::test_radix_sort_by_keys
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_array.py::test_radix_sort_by_keys
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_jit.py::TestAnnotationHelper::test_const_as_call_arg
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_jit.py::TestAnnotationHelper::test_const_as_call_arg
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_jit.py::TestAnnotationHelper::test_const_as_call_arg
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_jit.py::TestAnnotationHelper::test_const_in_return
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_jit.py::TestAnnotationHelper::test_const_in_return
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_jit.py::TestAnnotationHelper::test_const_in_return
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_jit.py::TestAnnotationHelper::test_undeclared_variable_declaration_in_for
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_parallel.py::TestParallelUtilsJIT::test_reduction_works_with_external_func_cython
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_parallel.py::TestParallelUtilsJIT::test_reduction_works_with_external_func_cython
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_parallel.py::TestParallelUtilsJIT::test_reduction_works_with_map_cython
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_parallel.py::TestParallelUtilsJIT::test_reduction_works_with_map_cython
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_parallel.py::TestParallelUtilsJIT::test_reduction_works_with_map_cython
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_parallel.py::TestParallelUtilsJIT::test_unique_scan_cython
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_parallel.py::TestParallelUtilsJIT::test_unique_scan_cython
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_parallel.py::TestParallelUtilsJIT::test_unique_scan_cython_parallel
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_parallel.py::TestParallelUtilsJIT::test_unique_scan_cython_parallel
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_simple_assignment_expression
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_simple_assignment_expression
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_simple_assignment_expression
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_simple_assignment_expression
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_assignment_expressions
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_assignment_expressions
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_assignment_expressions
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_assignment_expressions
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_assignment_expressions
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_if_block
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_if_block
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_if_block
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_if_block
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_if_block
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_conditionals
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_ternary_operator
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_ternary_operator
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_ternary_operator
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_ternary_operator
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_boolops
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_boolops
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_boolops
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_boolops
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_boolops
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_bitwise_ops
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_bitwise_ops
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_multiple_bitwise_ops
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_power
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_power
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_calling_function
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_calling_function
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_calling_printf_with_string
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_calling_printf_with_string
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_simple_function_with_return
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_py3_annotations
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_calling_method_of_known_type
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_calling_method_of_known_type
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_calling_method_of_known_type_in_method
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_calling_method_of_known_type_in_method
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_while
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_decreasing_range
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_decreasing_range
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_decreasing_range
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_declare
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_two_fors
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_two_fors
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_symbols
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_symbols
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_symbols
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_nested_for_with_symbols
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_nested_for_with_symbols
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_nested_for_with_symbols
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_nested_for_with_symbols
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_nested_for_with_symbols
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_nested_for_with_symbols
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_break_continue
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_break_continue
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_break_continue
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_break_continue
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_for_with_break_continue
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_attribute_access
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_declare_call_declares_variable
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_class
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_class
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_handles_parsing_functions
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_handles_parsing_functions
  /usr/lib/python3.9/ast.py:407: DeprecationWarning: visit_Num is deprecated; add visit_Constant
    return visitor(node)

.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_jit.py::TestAnnotationHelper::test_non_jit_call_as_call_arg
  /build/compyle-0.7/.pybuild/cpython3_3.9_compyle/build/compyle/jit.py:150: UserWarning: 
  In code in line 5:
  
      return g(sin(a))
               ^
  
  
  Function called is not marked by the annotate decorator. Argument
  type defaulting to 'double'. If the type is not 'double', store
  the value in a variable of appropriate type and use the variable
  
    warnings.warn(msg)

.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_calling_printf_with_string
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_calling_printf_with_string
  /usr/lib/python3.9/ast.py:407: DeprecationWarning: visit_Str is deprecated; add visit_Constant
    return visitor(node)

.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_bool_true_false_and_none
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_bool_true_false_and_none
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_bool_true_false_and_none
.pybuild/cpython3_3.9_compyle/build/compyle/tests/test_translator.py::test_bool_true_false_and_none
  /usr/lib/python3.9/ast.py:407: DeprecationWarning: visit_NameConstant is deprecated; add visit_Constant
    return visitor(node)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
= 6 failed, 197 passed, 88 skipped, 4 deselected, 113 warnings in 147.22 seconds =

module 'inspect' has no attribute 'getargspec' in Python 3.11

Hi,

$> python3.11
Python 3.11.0+ (main, Nov  4 2022, 09:23:33) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from compyle.template import Template
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/compyle/template.py", line 10, in <module>
    inspect, 'getfullargspec', inspect.getargspec
                               ^^^^^^^^^^^^^^^^^^
AttributeError: module 'inspect' has no attribute 'getargspec'. Did you mean: 'getargs'?
>>>

"getargspec" was deprecated since Python 3.0 and removed in Python 3.11.
I think is safe to remove default value as "getfullargspec" attribute is in all
Python 3 supported versions.

Kind Regards

Test failure on i386 and armhf platforms

============================= test session starts ==============================
platform linux -- Python 3.8.1, pytest-4.6.8, py-1.8.0, pluggy-0.13.0
rootdir: /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d
collected 265 items / 1 deselected / 264 selected

compyle/tests/test_array.py ..s..s..s..s..s..s..s..s..sF.s..s..s..s..s.. [ 16%]
s                                                                        [ 17%]
compyle/tests/test_ast_utils.py .......                                  [ 19%]
compyle/tests/test_capture_stream.py .....                               [ 21%]
compyle/tests/test_config.py ............                                [ 26%]
compyle/tests/test_cython_generator.py ...............                   [ 31%]
compyle/tests/test_ext_module.py ......                                  [ 34%]
compyle/tests/test_gpu_struct.py s                                       [ 34%]
compyle/tests/test_jit.py .......................                        [ 43%]
compyle/tests/test_low_level.py s.s...                                   [ 45%]
compyle/tests/test_parallel.py s.s...s..s..s..s..s..s...s...s...s...s.s. [ 60%]
..s..s..s..s..s..s...s...s...s...                                        [ 73%]
compyle/tests/test_template.py .....                                     [ 75%]
compyle/tests/test_translator.py ....................................... [ 90%]
.............                                                            [ 95%]
compyle/tests/test_transpiler.py ..                                      [ 95%]
compyle/tests/test_types.py .......                                      [ 98%]
compyle/tests/test_utils.py ....                                         [100%]

=================================== FAILURES ===================================
_________________________ test_align_multiple[cython] __________________________

backend = 'cython'

    @test_all_backends
    def test_align_multiple(backend):
        check_import(backend)
    
        # Given
        dev_array_a = Array(np.uint32, backend=backend)
        dev_array_b = Array(np.float32, backend=backend)
        orig_array_a = array.arange(0, 1024, 1, dtype=np.uint32, backend=backend)
        orig_array_b = array.arange(
            1024, 2048, 1, dtype=np.float32, backend=backend)
        dev_array_a.set_data(orig_array_a)
        dev_array_b.set_data(orig_array_b)
    
        indices = array.arange(1023, -1, -1, dtype=np.int64, backend=backend)
    
        # When
>       dev_array_a, dev_array_b = array.align([dev_array_a, dev_array_b],
                                               indices)

compyle/tests/test_array.py:191: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
compyle/array.py:643: in align
    align_multiple_elwise(*args_list)
compyle/parallel.py:525: in __call__
    self.elementwise(*args, **kwargs)
compyle/jit.py:316: in __call__
    c_func(*c_args, **kw)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   ValueError: Buffer dtype mismatch, expected 'long' but got 'long long'

m_2a00dba787c0caa83a32476b5532225f.pyx:22: ValueError
----------------------------- Captured stdout call -----------------------------
running build_ext
cythoning /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/m_2a00dba787c0caa83a32476b5532225f.pyx to /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/m_2a00dba787c0caa83a32476b5532225f.cpp
building 'm_2a00dba787c0caa83a32476b5532225f' extension
i686-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -O2 -ffile-prefix-map=/build/1st/compyle-0.6~dev0~20190922.gitaa5a50d=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/lib/python3/dist-packages/numpy/core/include -I/usr/local/include -I/usr/include/python3.8 -c /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/m_2a00dba787c0caa83a32476b5532225f.cpp -o /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/build/temp.linux-i386-3.8/build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/m_2a00dba787c0caa83a32476b5532225f.o
i686-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-z,relro -g -O2 -ffile-prefix-map=/build/1st/compyle-0.6~dev0~20190922.gitaa5a50d=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/build/temp.linux-i386-3.8/build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/m_2a00dba787c0caa83a32476b5532225f.o -o /build/1st/compyle-0.6~dev0~20190922.gitaa5a50d/.pybuild/cpython3_3.8_compyle/.compyle/source/py3.8-linux-i386/build/lib.linux-i386-3.8/m_2a00dba787c0caa83a32476b5532225f.cpython-38-i386-linux-gnu.so

Bug: minimum and maximum are computed incorrectly for certain arrays with 'cuda' backend.

This is not an issue for most arrays.

Here is an MWE,

from compyle.array import Array, update_minmax_gpu
import pycuda.gpuarray as gpuarray
from pycuda.reduction import ReductionKernel
import numpy as np

backend = 'cuda'

# numpy
a1 = np.asarray([4.2, 2.0, 0.01, 0.08, 4.0, 29.2], dtype=np.float32)
print(f'{a1=}')
np_max_val = np.max(a1)
np_min_val = np.min(a1)

# compyle
ca1 = Array(dtype=a1.dtype, n=len(a1), backend=backend)
ca1.set(a1)
update_minmax_gpu([ca1], backend=backend)

# cuda
ga1 = gpuarray.to_gpu(a1) # or ca1.dev
max_reduce = ReductionKernel(
    np.float32, 
    neutral=str(np.finfo(a1.dtype).min),
    reduce_expr="max(a,b)",
    map_expr="x[i]",
    arguments="float *x"
)
min_reduce = ReductionKernel(
    np.float32, 
    neutral=str(np.finfo(a1.dtype).max),
    reduce_expr="min(a,b)",
    map_expr="x[i]",
    arguments="float *x"
)
cuda_max_val = max_reduce(ga1).get() 
cuda_min_val = min_reduce(ga1).get()


# checks
np.testing.assert_allclose(np_min_val, cuda_min_val)
np.testing.assert_allclose(np_max_val, cuda_max_val)

np.testing.assert_allclose(np_min_val, ca1.minimum)
np.testing.assert_allclose(np_max_val, ca1.maximum)

np.testing.assert_allclose(np_max_val, ca1.maximum) fails. Here, ca1.maximum should be 29.2. Instead, it is 4.2.

For some arrays, the minimum is incorrectly computed. I haven't noticed any patterns.

This causes further issues in PySPH's GPUNNPS.

For the same array, backend='opencl' does not give this issue. I will try to find out what's wrong.

Add non-trivial examples

Simple Laplace equation
Barnes-Hut treecode in 2D
Other miscellaneous examples.

OSError: could not get source code

The error message: Traceback (most recent call last): File "<string>", line 16, in <module> File "/home/user/miniconda3/lib/python3.11/site-packages/compyle/parallel.py", line 573, in __call__ self.elementwise(*args, **kwargs) File "/home/user/miniconda3/lib/python3.11/site-packages/compyle/profile.py", line 72, in wrapper return method(*args, **kwargs) File "/home/user/miniconda3/lib/python3.11/site-packages/compyle/jit.py", line 355, in __call__ c_func = self._generate_kernel(*args) File "/home/user/miniconda3/lib/python3.11/site-packages/compyle/jit.py", line 29, in wrapper setattr(f, 'cached_kernel', {key_val: method(*args)}) File "/home/user/miniconda3/lib/python3.11/site-packages/compyle/jit.py", line 341, in _generate_kernel declarations = helper.annotate() File "/home/user/miniconda3/lib/python3.11/site-packages/compyle/jit.py", line 160, in annotate src = dedent('\n'.join(getsourcelines(self.func)[0])) File "/home/user/miniconda3/lib/python3.11/site-packages/compyle/utils.py", line 13, in getsourcelines return inspect.getsourcelines(obj) File "/home/user/miniconda3/lib/python3.11/inspect.py", line 1244, in getsourcelines lines, lnum = findsource(object) File "/home/user/miniconda3/lib/python3.11/inspect.py", line 1081, in findsource raise OSError('could not get source code') OSError: could not get source code

Steps to reproduce: install compyle with pip (latest version from git) and run the default example.

New release might be necessary

The deprecation of numpy.bool was already addressed with #87. Starting with NumPy-v1.24.0, this depreciation has expired. Since there has been no release that incorporates #87, the package on PyPI would be giving errors with the newer NumPy.

Sorting tests fail on Debian sid with numpy 1.26

After the update of numpy to 1.26 we started observing the following failures in the compyle unittests both with Python 3.11. and 3.12.

The error is the following:

python3.12 -m pytest -k "not test_that_multiple_compiles_do_not_occur_for_same_source and not test_const_as_call_arg and not test_const_in_return" --ignore=/<<PKGBUILDDIR>>/compyle/tests/test_cuda.py --ignore=/<<PKGBUILDDIR>>/compyle/tests/test_parallel.py /<<PKGBUILDDIR>>/compyle/tests
> ============================= test session starts ==============================
> platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0
> rootdir: /<<PKGBUILDDIR>>
> configfile: pyproject.toml
> collected 327 items / 3 deselected / 324 selected
> 
> ../../../compyle/tests/test_array.py ..s..s..s..s..s..s..s..s..s..s..s.. [ 10%]
> s..s.FsF..s.sx..s..s..s..s..s..s..s..s..s..s..s..s..s..s..s..s..s..s..s. [ 33%]
> .s..s..s..s..s..s..ss..s......sss......sss......sss..s                   [ 49%]
> ../../../compyle/tests/test_ast_utils.py .......                         [ 51%]
> ../../../compyle/tests/test_capture_stream.py .....                      [ 53%]
> ../../../compyle/tests/test_config.py ............                       [ 57%]
> ../../../compyle/tests/test_cython_generator.py .................        [ 62%]
> ../../../compyle/tests/test_ext_module.py ........                       [ 64%]
> ../../../compyle/tests/test_gpu_struct.py s                              [ 65%]
> ../../../compyle/tests/test_jit.py ..........................            [ 73%]
> ../../../compyle/tests/test_low_level.py s.s.....                        [ 75%]
> ../../../compyle/tests/test_profile.py .....                             [ 77%]
> ../../../compyle/tests/test_template.py .....                            [ 78%]
> ../../../compyle/tests/test_translator.py .............................. [ 87%]
> .........................                                                [ 95%]
> ../../../compyle/tests/test_transpiler.py ...                            [ 96%]
> ../../../compyle/tests/test_types.py .......                             [ 98%]
> ../../../compyle/tests/test_utils.py ....                                [100%]
> 
> =================================== FAILURES ===================================
> __________________________ test_sort_by_keys[opencl] ___________________________
> 
> backend = 'opencl'
> 
>     @check_all_backends
>     def test_sort_by_keys(backend):
>         check_import(backend)
>     
>         # Given
>         nparr1 = np.random.randint(0, 100, 16, dtype=np.int32)
>         nparr2 = np.random.randint(0, 100, 16, dtype=np.int32)
>         dev_array1, dev_array2 = array.wrap(nparr1, nparr2, backend=backend)
>     
>         # When
>         out_array1, out_array2 = array.sort_by_keys([dev_array1, dev_array2])
>     
>         # Then
>         order = np.argsort(nparr1)
>         act_result1 = np.take(nparr1, order)
>         act_result2 = np.take(nparr2, order)
>         assert np.all(out_array1.get() == act_result1)
> >       assert np.all(out_array2.get() == act_result2)
> E       assert False
> E        +  where False = <function all at 0x7f4cb39aa9f0>(array([ 9,  3...  dtype=int32) == array([ 9,  3...  dtype=int32)
> E        +    where <function all at 0x7f4cb39aa9f0> = np.all
> E           
> E           Use -v to get more diff)
> 
> ../../../compyle/tests/test_array.py:273: AssertionError
> ------------------------------ Captured log call -------------------------------
> DEBUG    pyopencl.cache:cache.py:378 build program: binary cache miss (key: 4bf4d189ce2047b8b06d9a0a5bca2e9d)
> DEBUG    pyopencl.cache:cache.py:413 build program: start building program from source on <pyopencl.Device 'cpu-skylake-avx512-Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz' on 'Portable Computing Language' at 0x2958b20>
> DEBUG    pyopencl.cache:cache.py:419 build program: from-source build complete
> DEBUG    pytools.persistent_dict:persistent_dict.py:758 pyopencl-invoker-cache-v41: disk cache hit [key=25d010a3d8d1b7c7137e612326d8db5d04a1e8ec5bb2a48c57295ed6430bda96]
> INFO     pyopencl:__init__.py:469 build program: kernel 'get_size_and_offsets' was part of a lengthy source build resulting from a binary cache miss (0.28 s)
> DEBUG    pytools.persistent_dict:persistent_dict.py:727 pyopencl-generated-scan-kernel-cache-v1: disk cache miss [key=611aa02c9ab60fec70c25bc297d9b1653da0a6e28c0f8ca21971a6c5efbcd16e]
> DEBUG    pyopencl.scan:scan.py:1194 cache miss for generated scan kernel 'scan'
> DEBUG    pyopencl.cache:cache.py:378 build program: binary cache miss (key: 653143575044338b788fc83353afc3f7)
> DEBUG    pyopencl.cache:cache.py:413 build program: start building program from source on <pyopencl.Device 'cpu-skylake-avx512-Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz' on 'Portable Computing Language' at 0x2958b20>
> DEBUG    pyopencl.cache:cache.py:419 build program: from-source build complete
> DEBUG    pytools.persistent_dict:persistent_dict.py:727 pyopencl-invoker-cache-v41: disk cache miss [key=49efc131ac55675366d11c67a6021ffc9ab51061e766db708184340ec1e7b944]
> DEBUG    pytools.persistent_dict:persistent_dict.py:696 pyopencl-invoker-cache-v41: disk cache store [key=49efc131ac55675366d11c67a6021ffc9ab51061e766db708184340ec1e7b944]
> INFO     pyopencl:__init__.py:469 build program: kernel 'scan_lev1' was part of a lengthy source build resulting from a binary cache miss (1.37 s)
> DEBUG    pytools.persistent_dict:persistent_dict.py:727 pyopencl-invoker-cache-v41: disk cache miss [key=eca878c5a15faa094d2cd87ee42508762ce2d70071b18e16df9b2f56071b9e2a]
> DEBUG    pytools.persistent_dict:persistent_dict.py:696 pyopencl-invoker-cache-v41: disk cache store [key=eca878c5a15faa094d2cd87ee42508762ce2d70071b18e16df9b2f56071b9e2a]
> DEBUG    pytools.persistent_dict:persistent_dict.py:696 pyopencl-generated-scan-kernel-cache-v1: disk cache store [key=611aa02c9ab60fec70c25bc297d9b1653da0a6e28c0f8ca21971a6c5efbcd16e]
> DEBUG    pyopencl.cache:cache.py:384 build program: binary cache hit (key: 653143575044338b788fc83353afc3f7)
> DEBUG    pytools.persistent_dict:persistent_dict.py:758 pyopencl-invoker-cache-v41: disk cache hit [key=49efc131ac55675366d11c67a6021ffc9ab51061e766db708184340ec1e7b944]
> DEBUG    pytools.persistent_dict:persistent_dict.py:758 pyopencl-invoker-cache-v41: disk cache hit [key=eca878c5a15faa094d2cd87ee42508762ce2d70071b18e16df9b2f56071b9e2a]
> DEBUG    pyopencl.cache:cache.py:378 build program: binary cache miss (key: a7aa988ad08d4c5636b571b0cd4ae55e)
> DEBUG    pyopencl.cache:cache.py:413 build program: start building program from source on <pyopencl.Device 'cpu-skylake-avx512-Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz' on 'Portable Computing Language' at 0x2958b20>
> DEBUG    pyopencl.cache:cache.py:419 build program: from-source build complete
> DEBUG    pytools.persistent_dict:persistent_dict.py:727 pyopencl-invoker-cache-v41: disk cache miss [key=02d42ae23a40f1caecefc2770cde1b1b442262a489c1b3a002673425df9233b7]
> DEBUG    pytools.persistent_dict:persistent_dict.py:696 pyopencl-invoker-cache-v41: disk cache store [key=02d42ae23a40f1caecefc2770cde1b1b442262a489c1b3a002673425df9233b7]
> INFO     pyopencl:__init__.py:469 build program: kernel 'scan_lev2' was part of a lengthy source build resulting from a binary cache miss (1.30 s)
> DEBUG    pytools.persistent_dict:persistent_dict.py:727 pyopencl-invoker-cache-v41: disk cache miss [key=d913f3cd98e8844fab3f92bfc28d0e5dca7af2f7da3b784402ba106d08be4e68]
> DEBUG    pytools.persistent_dict:persistent_dict.py:696 pyopencl-invoker-cache-v41: disk cache store [key=d913f3cd98e8844fab3f92bfc28d0e5dca7af2f7da3b784402ba106d08be4e68]
> DEBUG    pyopencl.cache:cache.py:378 build program: binary cache miss (key: fdd1b09b4764e411254319832d21c5fb)
> DEBUG    pyopencl.cache:cache.py:413 build program: start building program from source on <pyopencl.Device 'cpu-skylake-avx512-Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz' on 'Portable Computing Language' at 0x2958b20>
> DEBUG    pyopencl.cache:cache.py:419 build program: from-source build complete
> DEBUG    pytools.persistent_dict:persistent_dict.py:727 pyopencl-invoker-cache-v41: disk cache miss [key=01951cb5ce3fe8984bb56f98fa6e9163f6396390a612759f911c18d9a47df5c2]
> DEBUG    pytools.persistent_dict:persistent_dict.py:696 pyopencl-invoker-cache-v41: disk cache store [key=01951cb5ce3fe8984bb56f98fa6e9163f6396390a612759f911c18d9a47df5c2]
> INFO     pyopencl:__init__.py:469 build program: kernel 'scan_final_update' was part of a lengthy source build resulting from a binary cache miss (0.29 s)
> DEBUG    pytools.persistent_dict:persistent_dict.py:727 pyopencl-invoker-cache-v41: disk cache miss [key=f41f4aa10a5614c18e468c40b4ca31d6c64a59c262ed220e22930319f10ea84b]
> DEBUG    pytools.persistent_dict:persistent_dict.py:696 pyopencl-invoker-cache-v41: disk cache store [key=f41f4aa10a5614c18e468c40b4ca31d6c64a59c262ed220e22930319f10ea84b]
> ___________________________ test_radix_sort_by_keys ____________________________
> 
>     def test_radix_sort_by_keys():
>         backend = 'cython'
>         for use_openmp in [True, False]:
>             get_config().use_openmp = use_openmp
>             # Given
>             nparr1 = np.random.randint(0, 100, 16, dtype=np.int32)
>             nparr2 = np.random.randint(0, 100, 16, dtype=np.int32)
>             dev_array1, dev_array2 = array.wrap(nparr1, nparr2, backend=backend)
>     
>             # When
>             out_array1, out_array2 = array.sort_by_keys([dev_array1, dev_array2],
>                                                         use_radix_sort=True)
>     
>             # Then
>             order = np.argsort(nparr1)
>             act_result1 = np.take(nparr1, order)
>             act_result2 = np.take(nparr2, order)
>             assert np.all(out_array1.get() == act_result1)
> >           assert np.all(out_array2.get() == act_result2)
> E           assert False
> E            +  where False = <function all at 0x7f4cb39aa9f0>(array([33,  7...  dtype=int32) == array([33,  7...  dtype=int32)
> E            +    where <function all at 0x7f4cb39aa9f0> = np.all
> E               
> E               Use -v to get more diff)
> 
> ../../../compyle/tests/test_array.py:294: AssertionError
> ----------------------------- Captured stdout call -----------------------------
> [1/1] Cythonizing /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_compyle/.compyle/source/py3.1-linux-x86_64/m_38c1c650fcf31d53befb93aa70eceb48.pyx
> ------------------------------ Captured log call -------------------------------
> INFO     compyle.ext_module:ext_module.py:321 Compiling code at: /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_compyle/.compyle/source/py3.1-linux-x86_64/m_38c1c650fcf31d53befb93aa70eceb48.pyx
> DEBUG    root:discovery.py:490 No `name` configuration, performing automatic discovery
> INFO     root:dist.py:985 running build_ext
> INFO     root:build_ext.py:521 building 'm_38c1c650fcf31d53befb93aa70eceb48' extension
> INFO     root:spawn.py:38 x86_64-linux-gnu-gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -O2 -Werror=implicit-function-declaration -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/lib/python3/dist-packages/numpy/core/include -I/usr/include/python3.12 -c /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_compyle/.compyle/source/py3.1-linux-x86_64/m_38c1c650fcf31d53befb93aa70eceb48.cpp -o /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_compyle/.compyle/source/py3.1-linux-x86_64/build/temp.linux-x86_64-cpython-312/<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_compyle/.compyle/source/py3.1-linux-x86_64/m_38c1c650fcf31d53befb93aa70eceb48.o -fopenmp
> INFO     root:spawn.py:38 x86_64-linux-gnu-g++ -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -O2 -Werror=implicit-function-declaration -ffile-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2 /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_compyle/.compyle/source/py3.1-linux-x86_64/build/temp.linux-x86_64-cpython-312/<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_compyle/.compyle/source/py3.1-linux-x86_64/m_38c1c650fcf31d53befb93aa70eceb48.o -L/usr/lib/x86_64-linux-gnu -o /<<PKGBUILDDIR>>/.pybuild/cpython3_3.12_compyle/.compyle/source/py3.1-linux-x86_64/build/lib.linux-x86_64-cpython-312/m_38c1c650fcf31d53befb93aa70eceb48.cpython-312-x86_64-linux-gnu.so -fopenmp
> =============================== warnings summary ===============================
> compyle/tests/test_array.py: 6 warnings
> compyle/tests/test_jit.py: 5 warnings
> compyle/tests/test_translator.py: 93 warnings
>   /usr/lib/python3.12/ast.py:407: DeprecationWarning: visit_Num is deprecated; add visit_Constant
>     return visitor(node)
> 
> compyle/tests/test_array.py: 4 warnings
> compyle/tests/test_translator.py: 93 warnings
>   /<<PKGBUILDDIR>>/compyle/translator.py:682: DeprecationWarning: Attribute n is deprecated and will be removed in Python 3.14; use value instead
>     return str(node.n)

[CUT]

> =========================== short test summary info ============================
> FAILED ../../../compyle/tests/test_array.py::test_sort_by_keys[opencl] - asse...
> FAILED ../../../compyle/tests/test_array.py::test_radix_sort_by_keys - assert...
> = 2 failed, 264 passed, 57 skipped, 3 deselected, 1 xfailed, 354 warnings in 334.14s (0:05:34) =

Simplify out-of-the-box usage

While writing compyle code, I noticed that there were a few redundancies in the syntax which made it 1) more verbose than needed, and 2) easier to make mistakes. Given below are some changes which I believe could help reduce such redundancies.

Set a global default backend: I guess cython would be a good default. I imagine most people will only use one backend at a time. Users can override the global default and users who use multiple backends can pass the backend argument explicitly or use the with_config syntax.
Remove need for annotate by default: It looks likeElementwise could wrap the function itself
Different backend for openmp: Replace parallel cython backend (backend='cython' and use_openmp=True) with possibly an 'openmp' backend (backend='openmp'). This is based on the observations that the use_openmp flag is useless by itself and that most of the algorithms compyle uses are different for cython and for openmp.
Wrapper package over numpy: This is one point I am not too sure of but it's inspired by pyopencl's array package. This makes it possible to have an interface to create arrays directly on the device if the backend is opencl or cuda. More importantly, it helps clearly differentiate between a numpy array and a "wrapped" compyle array.

If the changes above are incorporated, the simple example given for compyle would look something like this

from compyle.api import Elementwise
import compyle.array as ary

def axpb(i, x, y, a, b):
    y[i] = a*sin(x[i]) + b

x = ary.linspace(0, 1, 10000)
y = ary.zeros_like(x)
a, b = 2.0, 3.0

e = Elementwise(axpb)
e(x, y, a, b)

pip install issue

When installed from pip "template.py" is not installed (file was missing in site-packges). Using an anaconda distribution on ubuntu. This was leading to the tests to file in pysph.

I was able to solve the problem by pulling directly from git and doing python setup.py develop