diegoferigo / cmake-build-extension Goto Github PK
View Code? Open in Web Editor NEWSetuptools extension to build and package CMake projects
Home Page: https://pypi.org/project/cmake-build-extension/
License: MIT License
Setuptools extension to build and package CMake projects
Home Page: https://pypi.org/project/cmake-build-extension/
License: MIT License
Pybind11, especially for big projects that use modern C++ features, is much better than SWIG. On the other hand, if the C++ project is thought from the beginning with SWIG support in mind, using SWIG prevent manually binding each class and each class method, which is quite nice.
This being said, I see a wider and wider usage of pybind11 over SWIG in C++ / Python hybrid projects, and it would be useful extending the example to show how pybind11 can be used in this setup.
Probably the most simple implementation is:
examples/swig/
subfolder and store the example in example/
example/bindings
to example/bindings_swig/
example/bindings_pybind11/
This way, the tiny C++ mymath
library can be shared by the two systems. And, nice point, the example would provide numpy support for both, that is something that I never found in any other project and, in my experience, is quite challenging to grasp for new users.
Hello,
Ninja is currently the only generator supported by cmake-build-extension:
Unfortunately it is not installed on all platform and a great benefit of CMake is to be agnostic of the final build tool that is used.
It would be nice to add a configuration option that select a different generator in replacement of Ninja (e.g. Make)
It could be useful introducing a new env var like CMAKE_BUILD_EXTENSION_ENABLED=<0|1>
to disable the compilation of the build extensions.
As an alternative, it could be replaced by a new --no-cmake-build-extension
option of build_ext
.
This allows for instance installing only Python resources excluding C++ resources. This is already the default behavior in editable mode, but this feature would allow obtaining the same also for non-editable mode.
I need to build a Python Extension for an OS that requires compilation with Xcode (cross compile, build on MacOS).
cmake-build-extension hard-codes the use of Ninja.
No issue having Ninja installed but it can't be used to build the project.
w/o the patch, cmake is called with 2 -G flags, one for Ninja and one for XCode.
I suggest the patch below. Check if the user (in setup.py) has asked for build system.
File modified is build_extension.py, line 149.
Maybe it can be improved because it behaves differently if Ninja is explicitly requested.
ext_dir = Path(self.get_ext_fullpath(ext.name)).parent.absolute()
cmake_install_prefix = ext_dir / ext.install_prefix
##### patch start here
# CMake configure arguments
configure_args = [
f"-DCMAKE_BUILD_TYPE={ext.cmake_build_type}",
f"-DCMAKE_INSTALL_PREFIX:PATH={cmake_install_prefix}",
]
# check the user has opted-in for a build system
build_system_specified = bool([option for option in ext.cmake_configure_options if option.startswith('**-DCMAKE_GENERATOR=**')])
# Default to Ninja if the user didn't explicitly request a build system
if not build_system_specified:
configure_args += [
"-GNinja",
# Fix #26: https://github.com/diegoferigo/cmake-build-extension/issues/26
f"-DCMAKE_MAKE_PROGRAM={shutil.which('ninja')}",
]
The GitSdistFolder
class should exclude bundling files that match the gitignore.
Right now it selects all the files excluding the .git
folder.
This is a longstanding problem that basically all build-extension implementation I'm aware of suffer. Currently, all the binaries of the CMake projects are correctly installed in the package folder that is installed inside the active site-packages, typically in /path/to/site-packages/<package_name>/bin/
. However, they cannot be exposed directly to the user, i.e. there's not simple way to get them either installed or symlinked in the bin
folder that is part of the PATH
.
Setuptools supports console scripts (setuptools, python packaging, entry points), but the system works only for calling python modules or functions.
With some python magic, we can add a new option (similar to the automatic creation of the __init__.py
file) that installs somewhere in the package a __main__
magic file that acts as a Python shim around the C++ executables. In this way, the executables are exposed to the user by the package manager, that cleanly handles the installation and removal of the Python shim that will be created outside the site-packages folder.
PEP440 defines, among other things, the ~=
operator for declaring compatible releases. Here below the reported example:
# Equivalent
~= 2.2
>= 2.2, == 2.*
# Equivalent
~= 1.4.5
>= 1.4.5, == 1.4.*
This could be helpful when used together with our cmake_depends_on
option.
Example: ProjectA and ProjectB use cmake-build-extension
(or provide a similar package structure like pybind11, casadi, etc), and ProjectA depends on ProjectB. Using ~=
in the dependency specification inside ProjectA metadata could provide a more robust setting to possible ABI / API breaks of future ProjectB versions.
Note that these depends could be only build requires
(defined in the pyproject.toml
) and not install_requires
.
In fact, it is not clear what would happen in a Python application MyApplication uses both ProjectA, that requires when building a specific version of ProjectB, and a new ProjectC, that as well requires when building a specific version of ProjectB, possibly different. I suspect that if the two ProjectB versions have different ABIs, the import order of ProjectA and ProjectC matters.
After analyzing the problem, the README should be updated accordingly with this best-practice.
Here's a quote from the wrapper in bindings_swig/bingins.i
:
void normalize_numpy(double* in_1, unsigned size_in_1,
double** out_1, int* size_out_1)
{
const std::vector<double> vector(in_1, in_1 + size_in_1);
auto result = mymath::normalize(vector);
*out_1 = static_cast<double*>(malloc(result.size() * sizeof(double)));
std::copy(result.begin(), result.end(), *out_1);
*size_out_1 = result.size();
}
I can see malloc
. The question is when the vector is going to be deallocated on the Python side? Or would numpy
do it implicitly (perhaps using Python's GC)?
Due to this issue: pypa/setuptools_scm#653 and brokenness in setuptools itself it is currently not possible to turn setuptools_scm
off if it is installed inside the build environment.
That means if I depend on cmake-build-extension
for only the purpose of building CMake extensions and not using the setuptools_scm
features I have to set up a bunch of pre-emptive excludes that setuptools_scm
is so kind to add for me.
I would suggest that until upstream setuptools
and setuptools_scm
get their act together that you make setuptools_scm
and exta and opt-in via cmake-build-extension[scm]
or something similar.
Apologies, this is not an issue, but rather a question:
using cmake-build-extension, is it possible to link the generated bindings to some imported library ? (imported: downloaded *.so file with no source available).
More details :
https://stackoverflow.com/questions/73022834/cmake-setuptools-linking-against-an-imported-library-cmake-build-extension
This issue is a reminder of the failure in CI related to the Python 3.7 job on Windows. Refer to #38 for the PR that marked this job as optional in CI.
============================= test session starts =============================
platform win32 -- Python 3.7.9, pytest-7.1.3, pluggy-1.0.0 -- c:\hostedtoolcache\windows\python\3.7.9\x64\python.exe
cachedir: .pytest_cache
rootdir: D:\a\cmake-build-extension\cmake-build-extension\example, configfile: setup.cfg, testpaths: tests
plugins: icdiff-0.6
Windows fatal exception: code 0xc0000139
Thread 0x00001094 (most recent call first):
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 1043 in create_module
File "<frozen importlib._bootstrap>", line 583 in module_from_spec
File "<frozen importlib._bootstrap>", line 670 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1035 in _handle_fromlist
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\mymath_pybind11\__init__.py", line 4 in <module>
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap_external>", line 728 in exec_module
File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 953 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "D:\a\cmake-build-extension\cmake-build-extension\example\tests\test_pybind11.py", line 3 in <module>
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\assertion\rewrite.py", line 168 in exec_module
File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "<frozen importlib._bootstrap>", line 1006 in _gcd_import
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\importlib\__init__.py", line 127 in import_module
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\pathlib.py", line 533 in import_path
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 608 in _importtestmodule
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 519 in _getobj
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 301 in obj
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 536 in _inject_setup_module_fixture
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 522 in collect
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 369 in <lambda>
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 338 in from_call
File "D:\a\cmake-build-extension\cmake-build-extension\example\tests\test_swig.py", line 3 in <module>
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\assertion\rewrite.py", line 168 in exec_module
File "<frozen importlib._bootstrap>", line 677 in _load_unlocked
File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 983 in _find_and_load
File "<frozen importlib._bootstrap>", line 1006 in _gcd_import
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\importlib\__init__.py", line 127 in import_module
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\pathlib.py", line 533 in import_path
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 608 in _importtestmodule
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 519 in _getobj
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 301 in obj
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 536 in _inject_setup_module_fixture
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\python.py", line 522 in collect
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 369 in <lambda>
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 338 in from_call
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 369 in pytest_make_collect_report
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\runner.py", line 537 in collect_one_node
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 824 in genitems
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 657 in perform_collect
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 332 in pytest_collection
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 321 in _main
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 268 in wrap_session
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\main.py", line 315 in pytest_cmdline_main
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\config\__init__.py", line 165 in main
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\_pytest\config\__init__.py", line 187 in console_main
File "C:\hostedtoolcache\windows\Python\3.7.9\x64\Scripts\pytest.exe\__main__.py", line 7 in <module>
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\runpy.py", line 85 in _run_code
File "c:\hostedtoolcache\windows\python\3.7.9\x64\lib\runpy.py", line 193 in _run_module_as_main
collecting ... collected 0 items / 2 errors
=================================== ERRORS ====================================
___________________ ERROR collecting tests/test_pybind11.py ___________________
ImportError while importing test module 'D:\a\cmake-build-extension\cmake-build-extension\example\tests\test_pybind11.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\importlib\__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests\test_pybind11.py:3: in <module>
import mymath_pybind11.bindings as mymath
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\mymath_pybind11\__init__.py:4: in <module>
from . import bindings
E ImportError: DLL load failed: The specified procedure could not be found.
_____________________ ERROR collecting tests/test_swig.py _____________________
ImportError while importing test module 'D:\a\cmake-build-extension\cmake-build-extension\example\tests\test_swig.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\importlib\__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests\test_swig.py:3: in <module>
import mymath_swig.bindings as mymath
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\mymath_swig\__init__.py:4: in <module>
from . import bindings
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\site-packages\mymath_swig\bindings.py:18: in <module>
from . import _swig_bindings
E ImportError: DLL load failed: The specified procedure could not be found.
!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!
============================== 2 errors in 0.45s ==============================
Error: Process completed with exit code 1.
The extension tries to write the init.py into the wrong folder. I don't know how to change this behavior
==> Building:
$ cmake --build /home/xxx/work/cpp/build/temp.linux-x86_64-cpython-310_Blah --config Release
==> Installing:
$ cmake --install /home/xxx/work/cpp/build/temp.linux-x86_64-cpython-310_Blan
error: [Errno 2] No such file or directory: '/home/xxx/work/cpp/build/lib.linux-x86_64-cpython-310/blah/__init__.py'
[end of output]
The build folder should be temp.linux-x86_64-cpython-310_Blah
, but it tries to write the __init__.py
file into /home/xxx/work/cpp/build/lib.linux-x86_64-cpython-310/blah/
the above error is triggered by write_top_level_init=init_py,
in setup.py
Unless I'm misinterpreting something, using BuildExtension
as cmdclass['build_ext']
in setup.py
makes any non-cmake extension modules completely ineffective, because BuildExtension.run()
simply ignores them:
Is my understanding correct? How would I use this project if I also have e.g. cython
extension modules?
The following files are included in a source distribution by default:
all C source files mentioned in the ext_modules or libraries setup() arguments
However, when I use cmake-build-extension
, I have to manually add the source into sdist.
In this line, I wonder if the ninja always be checked if I already specify another generator like Unix Makefiles?
PS: no matter what generator I specify, it always throw an error:
==> Configuring:
$ cmake -S /tmp/build-via-sdist-p3l2l10i/xtensor_python_test-0.0.1 -B /tmp/build-via-sdist-p3l2l10i/xtensor_python_test-0.0.1/build/temp.linux-x86_64-cpython-310_Pybind11Bindings -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX:PATH=/tmp/build-via-sdist-p3l2l10i/xtensor_python_test-0.0.1/build/lib.linux-x86_64-cpython-310/mymath_pybind11 -DPython3_ROOT_DIR=/tmp/build-env-yxeuhf6l -DCALL_FROM_SETUP_PY:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=OFF -DEXAMPLE_WITH_SWIG:BOOL=OFF -DEXAMPLE_WITH_PYBIND11:BOOL=ON
==> Building:
$ cmake --build /tmp/build-via-sdist-p3l2l10i/xtensor_python_test-0.0.1/build/temp.linux-x86_64-cpython-310_Pybind11Bindings --config Release
==> Installing:
$ cmake --install /tmp/build-via-sdist-p3l2l10i/xtensor_python_test-0.0.1/build/temp.linux-x86_64-cpython-310_Pybind11Bindings
CMake Error: Could not create named generator "Unix Makefiles"
Thanks in advance if you can help me!
It currently uses python
.
https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html#using-a-src-layout
Some while ago, I marked the test as allowed to fail, but it would be wise to open an issue to track the problem.
The SWIG example compiles fine in all Window versions, generating the following files:
adding 'mymath/_bindings.pyd'
adding 'mymath/bindings.py'
adding 'mymath/libbindings.dll.a'
adding 'mymath/include/mymath.h'
adding 'mymath/lib/libmymath.a'
adding 'mymath-0.0.0.dist-info/METADATA'
adding 'mymath-0.0.0.dist-info/WHEEL'
adding 'mymath-0.0.0.dist-info/top_level.txt'
adding 'mymath-0.0.0.dist-info/RECORD'
However, when it is imported in the tests, it fails with the following error (more details in any CI job):
=================================== ERRORS ====================================
____________________ ERROR collecting tests/test_mymath.py ____________________
ImportError while importing test module 'D:\a\cmake-build-extension\cmake-build-extension\examples\swig\tests\test_mymath.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
c:\hostedtoolcache\windows\python\3.9.5\x64\lib\importlib\__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests\test_mymath.py:3: in <module>
import mymath.bindings
c:\hostedtoolcache\windows\python\3.9.5\x64\lib\site-packages\mymath\bindings.py:18: in <module>
from . import _bindings
E ImportError: DLL load failed while importing _bindings: The specified module could not be found.
=========================== short test summary info ===========================
ERROR tests/test_mymath.py
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
============================== 1 error in 0.37s ===============================
I have no idea what could cause this error. I marked as allowed to fail also the new Python 3.9 job.
I also caught the following in the Python 3.7 job:
C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-t1xho44w\overlay\Lib\site-packages\wheel\bdist_wheel.py:82: RuntimeWarning: Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
warn=(impl == 'cp')):
C:\Users\RUNNER~1\AppData\Local\Temp\pip-build-env-t1xho44w\overlay\Lib\site-packages\wheel\bdist_wheel.py:87: RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
sys.version_info < (3, 8))) \
The second warning disappears if Python >= 3.8.
Would it be possible to cut a new version of this module with latest commit?
thanks
Hi,
thanks for this useful project.
I was wondering if you could also publish the package to conda-forge so it is possible to use it in conda_env.yaml
files.
name: my_env
channels:
- pytorch
- conda-forge
dependencies:
- python=3.7
- pip
- cmake_build_extension
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.