Giter VIP home page Giter VIP logo

py-build-cmake's Introduction

Python Wheel Documentation PyPI - Downloads

py-build-cmake

Modern, PEP 517 compliant build backend for creating Python packages with extensions built using CMake.

Features

  • Building and packaging C, C++ or Fortran extension modules for Python using CMake
  • Declarative configuration using pyproject.toml (PEP 621), compatible with flit
  • Editable/development installations for Python modules (PEP 660)
  • Compatible with pybind11 and nanobind
  • Stub generation for type checking and autocompletion
  • Customizable CMake configuration, build and installation options
  • Support for multiple installation configurations and components
  • Cross-compilation support
  • No dependency on setuptools
  • Compatible with cibuildwheel for building Wheels

Installation

The py-build-cmake package is available on PyPI:

pip install py-build-cmake

Documentation

The documentation can be found on https://tttapa.github.io/py-build-cmake.

The format of the configuration file is explained in Config.md.

Alternatively, use the command-line interface to get the documentation for all supported options:

py-build-cmake config format

Usage

If you don't have one already, add a pyproject.toml configuration file to your project's repository. Specify the metadata required by PEP 621, and tell py-build-cmake how to build your project. For example:

[project] # Project metadata
name = "example-project"
readme = "README.md"
requires-python = ">=3.7"
license = { "file" = "LICENSE" }
authors = [{ "name" = "Pieter P", "email" = "[email protected]" }]
keywords = ["some", "keywords"]
classifiers = ["Topic :: Scientific/Engineering"]
urls = { "Documentation" = "https://tttapa.github.io/py-build-cmake" }
dependencies = ["numpy"]
dynamic = ["version", "description"]

[build-system] # How pip and other frontends should build this project
requires = ["py-build-cmake~=0.1.8"]
build-backend = "py_build_cmake.build"

[tool.py-build-cmake.module] # Where to find the Python module to package
directory = "src-python"

[tool.py-build-cmake.sdist] # What to include in source distributions
include = ["CMakeLists.txt", "src/*"]

[tool.py-build-cmake.cmake] # How to build the CMake project
build_type = "RelWithDebInfo"
source_path = "src"
build_args = ["-j"]
install_components = ["python_modules"]

[tool.py-build-cmake.stubgen] # Whether and how to generate typed stub files

The README of examples/minimal describes this configuration file in much more detail.

Then use pip, build or another PEP 517 compatible frontend to build and/or install the package.

Build sdist and wheel packages you can upload to PyPI:

python -m pip install -U build
python -m build . # find the sdist and wheel file in the 'dist' folder

Install the package in the current environment:

pip install .    # normal installation
pip install -e . # editable installation

Examples

As an introduction to py-build-cmake, see examples/minimal for a detailed overview of the configuration files and the directory structure, using a very simple Python module as an example.
For a more advanced, real-world example, see examples/pybind11-project and examples/nanobind-project.
If you are interested in packaging C/C++/Fortran programs using py-build-cmake, have a look at examples/minimal-program.

Projects using py-build-cmake

If you need more examples, you can look at the following projects using py-build-cmake as their Python build backend:

Planned features

  • macOS support
  • Entry point support
  • Namespace package support (PEP 420)
  • Doxygen and Sphinx support

py-build-cmake's People

Contributors

tttapa avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

py-build-cmake's Issues

On v0.2.0 rework branch: -A platform option passed to CMake for generators that don't support it

Noticed this trying to run a Ninja-based build on Windows with py-build-cmake.

py-build-cmake passes the -A x64 flag to cmake on Windows. This will lead to an error if using Ninja (or most other non-MSVC builders).

I found in the code that this happens in get_cmake_generator_platform. It seems indeed to be based on the OS and not on the generator.

According to the CMake docs (https://cmake.org/cmake/help/latest/variable/CMAKE_GENERATOR_PLATFORM.html), only Visual Studio generators and "Green Hills MULTI" generators support the -A flag.

Could another check be added to ensure the -A is only passed when using a VS generator?

Incorrect linking of arm64 library with x86_64 compilation

Hi Pieter. We ran into an interesting corner case running on an Apple M2 CPU running Anaconda installed for x86_64 (which I suppose macOS then emulates for automatically). The most correct solution to the problem is simply to install Anaconda for arm64, then everything works as expected. I thought we should report the experience anyway, in case you think there is something that can/should be done in py-build-cmake for it, though I'm not requesting that you do so.

In short, py-build-cmake appears to have cmake compile for x86-64 but cmake still tries to link against a dependent library (libyaml-cpp) that is built for arm64. My layman's guess is that py-build-cmake is picking up the x86_64 arch from the conda execution environment (e.g., via distlib?), but the cross-compilation configuration is somehow incomplete.

I'm not entirely sure what the correct thing to do should be, but I don't think it should get into a situation where it's trying to link libraries of the wrong architecture. Perhaps at least one of:

  1. The arm64 library path should not be searched.
  2. The arm64 library itself should not "found" by cmake even if the path is searched.
  3. The native code should be compiled as arm64 rather than x86-64.
Obtaining file:///Users/haonanwa/Projects/PipeEdge
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [200 lines of output]
      CMake Warning (dev) at /opt/homebrew/Cellar/cmake/3.27.0/share/cmake/Modules/GNUInstallDirs.cmake:243 (message):
        Unable to determine default CMAKE_INSTALL_LIBDIR directory because no
        target architecture is known.  Please enable at least one language before
        including GNUInstallDirs.
      Call Stack (most recent call first):
        CMakeLists.txt:3 (include)
      This warning is for project developers.  Use -Wno-dev to suppress it.
      
      -- Checking for py-build-cmake environment - found
      --   Using PEP 427-compatible install paths
      -- The CXX compiler identification is AppleClang 14.0.3.14030022
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Configuring done (0.4s)
      -- Generating done (0.0s)
      CMake Warning:
        Manually-specified variables were not used by the project:
      
          PY_BUILD_CMAKE_PACKAGE_NAME
          Python3_EXECUTABLE
          Python3_FIND_REGISTRY
          Python3_FIND_STRATEGY
          Python3_ROOT_DIR
      
      
      -- Build files have been written to: /Users/haonanwa/Projects/PipeEdge/.py-build-cmake_cache/cp37-cp37m-macosx_10_9_x86_64
      [ 66%] Building CXX object CMakeFiles/sched-pipeline.dir/sched-pipeline.cpp.o
      [ 66%] Building CXX object CMakeFiles/sched-pipeline.dir/schedule.cpp.o
      /Users/haonanwa/Projects/PipeEdge/src-native/schedule.cpp:200:10: warning: variable 'stage_num' set but not used [-Wunused-but-set-variable]
        size_t stage_num = 0;
               ^
      1 warning generated.
      [100%] Linking CXX executable sched-pipeline
      ld: warning: ignoring file /opt/homebrew/lib/libyaml-cpp.0.7.0.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
      Undefined symbols for architecture x86_64:
        "YAML::InvalidNode::~InvalidNode()", referenced from:
            _main in sched-pipeline.cpp.o
        NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
        "vtable for YAML::Exception", referenced from:
            YAML::Exception::Exception(YAML::Mark const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) in sched-pipeline.cpp.o
        NOTE: a missing vtable usually means the first non-inline virtual member function has no definition.
      ld: symbol(s) not found for architecture x86_64
      clang: error: linker command failed with exit code 1 (use -v to see invocation)
      make[2]: *** [sched-pipeline] Error 1
      make[1]: *** [CMakeFiles/sched-pipeline.dir/all] Error 2
      make: *** [all] Error 2
      Traceback (most recent call last):
        File "/Users/haonanwa/opt/anaconda3/envs/pipeedge/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 167, in prepare_metadata_for_build_editable
          hook = backend.prepare_metadata_for_build_editable
      AttributeError: module 'py_build_cmake.build' has no attribute 'prepare_metadata_for_build_editable'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/Users/haonanwa/opt/anaconda3/envs/pipeedge/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/haonanwa/opt/anaconda3/envs/pipeedge/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/haonanwa/opt/anaconda3/envs/pipeedge/lib/python3.7/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 176, in prepare_metadata_for_build_editable
          whl_basename = build_hook(metadata_directory, config_settings)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/build.py", line 86, in build_editable
          editable=True)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/build.py", line 248, in build_wheel_in_dir
          self.do_native_cross_cmake_build(paths, cfg, pkg_info)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/build.py", line 334, in do_native_cross_cmake_build
          cfg.cross, package_info, native_install_dir)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/build.py", line 511, in run_cmake
          cmaker.build()
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/cmake.py", line 190, in build
          self.run(cmd, cwd=cwd, check=True, env=env)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/cmake.py", line 68, in run
          return self.runner.run(*args, **kwargs)
        File "/private/var/folders/th/npccwh1j0hj106lcytk5m_4c0000gn/T/pip-build-env-zw52vokl/overlay/lib/python3.7/site-packages/py_build_cmake/cmd_runner.py", line 25, in run
          return sp_run(*args, **kwargs)
        File "/Users/haonanwa/opt/anaconda3/envs/pipeedge/lib/python3.7/subprocess.py", line 512, in run
          output=stdout, stderr=stderr)
      subprocess.CalledProcessError: Command '['cmake', '--build', '/Users/haonanwa/Projects/PipeEdge/.py-build-cmake_cache/cp37-cp37m-macosx_10_9_x86_64', '--config', 'RelWithDebInfo', '-j']' returned non-zero exit status 2.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Cheers.

Editable install doesn't work with pylint

I've created a demo project to reproduce this issue: https://github.com/cimes-isi/py-build-cmake-editable-pylint

In short, pylint produces errors when trying to import from a Python project built with py-build-cmake and installed as editable. The issue does not occur for normal installs.

I see the following differences with pip freeze. A setuptools-based project adds:

-e git+ssh://[email protected]/cimes-isi/py-build-cmake-editable-pylint.git@c0f9e2ca54368e646724ed7268488708d33d1a69#egg=foo&subdirectory=foo-setuptools
foo-setuptools==0.1.0

but a py-build-cmake project only adds:

-e git+ssh://[email protected]/cimes-isi/py-build-cmake-editable-pylint.git@c0f9e2ca54368e646724ed7268488708d33d1a69#egg=foo&subdirectory=foo-py-build-cmake

It seems like maybe the editable install is incomplete?

Thanks!

Using --prefix during install overrides CMAKE_INSTALL_PREFIX

I was exploring a project setup where we might build both a native application and a shared object library, the latter of which both the native binary and Python code can use (e.g., with ctypes or cffi). One problem was that the native binary couldn't find the library at runtime since the library installation path is not on LD_LIBRARY_PATH (or DYLD_LIBRARY_PATH on macos). I wasn't able to figure out how to configure the rpath properly for the installed binary since the actual install location was different than what I could derive from CMAKE_INSTALL_PREFIX, though admittedly my experience with this is limited.

My suspicion is that using --prefix with cmake makes this impossible. Is there a reason to not set -DCMAKE_INSTALL_PREFIX=/full/path/prefix with the initial cmake build command instead? Also note that CMake 3.21 introduced the --install-prefix option.

Cheers.

How to simultaneously maintain multiple build variants?

Hi,

Can I simultaneously maintain multiple build variants with py-build-cmake? By build variants, I mean different options passed to CMake, in addition to multiple architectures. For example, PyTorch release one variant for each CUDA version, like torch==2.0.0+cu118 or torch==2.0.0+cu117. Do I have to provide a different toml file for each variant, and override the default one by py-build-cmake --local manually?

Q: How to enable incremental builds?

I'm writing an interface to a large-ish library and my compile cycle now takes about 10 minutes.

It is frustrating that using py-build-cmake seems to do everything in a new, fresh environment every time, so that everything I didn't change gets recompiled over, every time.

How can I set things up that that only what needs to be recompiled is (ie. nothing that I haven't changed!)

I tried changing /INCREMENTAL:NO to /INCREMENTAL:YES in the example TOML - but this does seemingly nothing; a new, clean environment is used the next build, and I have to wait the full ten minutes.

Installing and locating cmake-compiled executables

As I mentioned in #3, I'm building an executable---rather than direct Python bindings---which I run as a subprocess. The question is then where and how (with py-build-cmake) to install the executable in a standard way.

I believe there are at least the following requirements:

  1. The installed executable must be discoverable by at least the Python module that created it. More broadly, it might be discoverable by any Python code in the environment, even if that requires it to know what module owns the executable.
  2. The executable path must be resolvable by the OS so that it can be executed by subprocess.run(...). This would seem to require that it must be installed on the filesystem, i.e., not in a zipped package. (For example, it may not be sufficient to treat the executable as a resource that can be located, e.g., with importlib.resources.)

The easiest thing is for the installed executable to be on PATH so that no additional searching is required. This is also perhaps the most general result we can achieve.

py-build-cmake appears to use the site-packages directory as the cmake install prefix (at least in my virtualenv), but I don't believe this or any subdirectories can be assumed (or even expected) to be on PATH. Can/should we be able to install to the bin/ directory of a virtualenv? I don't know enough about Python packaging and distribution to understand if that generalizes or if there's a standard that achieves a similar result (like an entry point, even if that requires a level of indirection?).

Thanks!

Q: Passing CMake variables on the command line

First of all: thanks for this very useful tool.

I was wondering whether it is a concsious decision (or result of a decision made in Python build flow) to only allow passing CMake overriding settings via a toml file (via -C--local=...). Oftentimes, CMake variables need to be set due to environment conditions such as the toolchain used, installation location of libraries, etc. It is a bit cumbersome to always have to create toml files for one-time passing of such settings.

It gets particularly tricky if there is a combination of different kinds of settings that we may already have separate tomls for: e.g. combining settings for finding libraries/compilers with settings such as the build type (Debug/Release/...).

Bug: MACOSX_DEPLOYMENT_TARGET is not respected

Command to build project:

MACOSX_DEPLOYMENT_TARGET=11.0 python3 -m pip wheel -v --no-deps --wheel-dir wheels .

Logs and final package:

  Running command Preparing metadata (pyproject.toml)
  loading initial cache file /Users/sandye51/Documents/Programming/git_repo/openvino_tokenizers/.py-build-cmake_cache/cp312-cp312-macosx_14_0_arm64/py-build-cmake-preload.cmake
....
  Created wheel for openvino-tokenizers: filename=openvino_tokenizers-2024.3.0.0-py3-none-macosx_14_0_arm64.whl size=44940 sha256=4ebce1f2cf2d12e880f4274c9b029004a31e71e6d9f66aec94a3addd09150032
  Stored in directory: /private/var/folders/d_/z_lpvwrx1h37bv_k0m5mf1yr0000gn/T/pip-ephem-wheel-cache-7ct8ndqt/wheels/99/f4/22/a91c0bf10aabb4e3ce24bcf9d46d57e94cf8713d37c740a312

It can be overridden via extra option --config-settings=override=cross.arch="macosx_11_0_arm64"

But some other projects like scikit-build-core respect this option

Module name cannot actually differ from project name

When trying to build a module (here called hactool) in a project with a different name (here pyhactool), it seems like this module discards the name value from tool.py-build-cmake.module:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/usr/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 314, in build_sdist
    return backend.build_sdist(sdist_directory, config_settings)
  File "/tmp/build-env-cj8x38zt/lib/python3.10/site-packages/py_build_cmake/build.py", line 83, in build_sdist
    pkg = Module(norm_name, src_dir / cfg.module.get('directory', '.'))
  File "/tmp/build-env-cj8x38zt/lib/python3.10/site-packages/flit_core/common.py", line 59, in __init__
    raise ValueError("No file/folder found for module {}".format(name))
ValueError: No file/folder found for module pyhactool

Add a swig example

It would be good to add an example using swig and the python stable abi.

cmake config and args sections (and maybe more) in py-build-cmake.local.toml does not seem to apply

I'm trying to set up a project to build a binary with py-build-cmake and need to pass some options via a config file depending on system but anything I add to args and config in the py-build-cmake.local.toml do not seem to have any effect. If I change source_path it does however so the config is clearly being parsed. It works fine if I put the same options in pyproject.toml. Same happens on both ubuntu 24.04 and windows 10 with python 3.12. I couldn't figure out if there was a way to load a different config file when running py -m build and running py-build-cmake manually just fails with Error: /home/___/___/___/minimal-program/.py-build-cmake_cache/cp312-cp312-linux_x86_64 is not a directory (or doesn't do anything meaningful and just skips the build process if there is an existing build already)

Not sure if it's in any way related but the build env also complains about /tmp/build-env-z_d23dbe/lib/python3.12/site-packages/py_build_cmake/config_options.py:510: SyntaxWarning: invalid escape sequence '\|'

Minimum cmake version

In the process of testing #5, I ran into a separate issue with cmake version requirements. It looks like py-build-cmake itself requires at least CMake >= 3.15, if not newer. I identified this lower bound is based on support for the cmake --install CLI option (see CMake 3.15 Release Notes).

Logically: let py-build-cmake's min version be A (e.g., 3.15), the user's declared minimum version be B (None --> 0.0), and the existing system cmake version be C (None --> 0.0). If D = max(A, B), then it seems you should add the cmake >= D Python package dependency if D > C.

FWIW, I use cmake --build <dir> --target install to install on older cmake versions, but I'm not sure how cleanly that maps to your approach since it forces a "build" step as part of the install process.

Compiling on `macos-14` (arm64) cp39 result in `Unable to determine extension suffix. Try manually setting PY_BUILD_EXT_SUFFIX.`

When compiling a nanobind project with py-build-cmake, I encountered Unable to determine extension suffix. Try manually setting PY_BUILD_EXT_SUFFIX.. Note that my project is using QueryPythonForNanobind.cmake as provided in the (nanobind example of this project)[https://github.com/tttapa/py-build-cmake/blob/rework-0.2.0/examples/nanobind-project/cmake/QueryPythonForNanobind.cmake].

  • This only occurs on macos-14, which is arm64 macOS github runner. This problem does not occur on macos-11 and macos-12, which are x86_64 macOS github runner.
  • This problem occurs on cp39, does not occur on cp38.
  • I am able to reproduce this bug both on 0.2.0a8 and 0.2.0a12.

Project where I discover this problem: https://github.com/laggykiller/apngasm-python/tree/stubgen-official
GitHub action log: https://github.com/laggykiller/apngasm-python/actions/runs/8062229801/job/22021550265#step:4:2210

CMake presets functionality seems to be broken

We are trying to use CMakePresets.json with py-build-cmake, but the documentation seems out of sync with what is implemented in CMake i.e. there is no concept of install preset, which is referred to in the py-build-cmake documentation. One result is that py-build-cmake generates invalid arguments to CMake resulting in errors.

Lifting and duplicating the settings out of CMakePresets.json into "old school" CMake arguments works, however this is not our preferred way of working since the Python module is just a small part of the project and leads to DRY-violations in the build system.

Are there any examples of working with CMake presets from py-build-cmake that actually work?

CMake preset ref; https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html

Namespace packages support 420

According to the README, this feature is planned. Can you say something about the status please?

What is necessary to implement this feature?

Thank you for this very useful package!

ctypes lib build with ExternalProject

Hi,

Very much like #11, I am trying to build a wheel with a pre-built library for use with ctypes, except that my external library (libjpeg-turbo, as a git submodule) does not allow inclusion by add_subdirectory. So I am trying with ExternalProject_Add.

I should add that I am not at all familiar with cmake, I'm learning as I go, and really appreciate any help.

Following #11, I haven't been able to get the install step working. After the ExternalProject_Add call it doesn't recognize the target name. Here is part of my pyproject.toml:

[tool.py-build-cmake.module]
directory = "src"


[tool.py-build-cmake.sdist] 
include = ["CMakeLists.txt", "src/*", "extern/**/*"]


[tool.py-build-cmake.cmake] # How to build the CMake project
build_type = "RelWithDebInfo"
build_path = "build"
source_path = "."  # "extern/libjpeg-turbo"
build_args = ["-j"]
install_components = ["python_module"]

and the whole CMakeLists.txt file at the root level

include(ExternalProject)


cmake_minimum_required(VERSION 3.3)
project(pylibjpeg_turbo)
find_package(Python3 REQUIRED COMPONENTS Development)

# tried as -P switch to INSTALL_COMMAND:  set(PROJECT_INSTALL_STEPS_FILE project_install_steps.cmake)
set(BUILD_SHARED_LIBS On)
set(LIB_INSTALL_DIR ${PY_BUILD_CMAKE_MODULE_NAME})

ExternalProject_Add(libjpeg-turbo
    SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/extern/libjpeg-turbo
    BUILD_COMMAND ${CMAKE_COMMAND} --build . --target turbojpeg
    INSTALL_COMMAND ""
)

# Install the module
if (WIN32)
    install(TARGETS turbojpeg
            EXCLUDE_FROM_ALL
            RUNTIME DESTINATION ${PY_BUILD_CMAKE_MODULE_NAME}
            COMPONENT python_module)
else()
    install(TARGETS turbojpeg
            EXCLUDE_FROM_ALL
            LIBRARY DESTINATION ${PY_BUILD_CMAKE_MODULE_NAME}
            COMPONENT python_module)
endif()

Which gives

 install TARGETS given target "turbojpeg" which does not exist.

I also tried a -P install_steps.cmake file for the INSTALL_COMMAND within the ExternalProject_Add, which basically just gives a subprocess error without any useful information that I can see.

I'd be happy to just copy the built libraries using INSTALL DIRECTORY FILES_MATCHING, but then I hit another problem - my built libraries remain in a tmp path even when I specify build_path. I ran with verbose to try to get more information:
python3 -m build --no-isolation . -C verbose

The first dump of the dicts gives:

cmake:
{'linux': {'args': [],
           'build_args': ['-j'],
           'build_path': '/home/dmason/git/turbo/build',
...

and similar for other platforms, but a second dump in the verbose output gives:

cmake:
{'linux': {'args': [],
           'build_args': ['-j'],
           'build_path': '/tmp/build-via-sdist-n_e1032f/pylibjpeg_turbo-0.1.0/build',

So I'm struggling with my limited knowledge and a lot of googling, how to put the pieces together which can make this happen. Really appreciate any help to get me pointed in the right direction.

A couple of BTWs:

  • on a user running the wheel install, I don't need libjpeg-turbo installed into system locations, I'd be happy to keep it as package data and give ctypes the full path.
  • I will also need to use this with cibuildwheel, if that has any bearing.

Thanks

Support abi3 wheel tag for stable ABI

It appears that the only choices for building a wheel are to use the py3 tag for a pure Python package or the native platform tag (eg. cp312). Is there a way to use the abi3 tag, for projects using the limited API / stable ABI?

That would be greatly useful to avoid having to build separate wheels for each Python version.

src vs src-python

Hello 👋

Just a question, more than an issue.

In the minimal example (as well as the others, but this is the one I read through), the src folder containing source files to be compiled is separated from a src-python folder containing the python package itself.

Intuitively, I was going to put extension files together with python files. Would this be a bad idea?

Also, assuming separation is nice, would there be a counter-point to having a structure like this?

+ package
  + src
    + py_package
        - __init__.py
        - ...
    + extension  # no CMakeLists.txt here
        - spam.c
    + include
       - spam.h
  - pyproject.toml
  - CMakeLists.txt
  - ...

Building cmake project in submodule

Hello, I am trying to create a ctypes library for rlottie. I have created this github repository: https://github.com/laggykiller/rlottie-python

I am trying to build wheel that contains rlottie.dll / librlottie.so / librlottie.dylib, which could then be used by the python module by ctypes.

I have read the documentations already, but it seems like the built files are not added to the wheel.

Part of pyproject.toml (https://github.com/laggykiller/rlottie-python/blob/master/pyproject.toml):

[tool.py-build-cmake.cmake]
build_type = "RelWithDebInfo"
source_path = "."
build_args = ["-j"]
install_components = ["python_module"]

The CMakeLists.txt in the root of my repo (https://github.com/laggykiller/rlottie-python/blob/master/CMakeLists.txt):

cmake_minimum_required(VERSION 3.3)
project(rlottie-python)
find_package(Python3 REQUIRED COMPONENTS Development)

add_subdirectory(rlottie)

# Install the module
install(TARGETS rlottie
        EXCLUDE_FROM_ALL
        LIBRARY     DESTINATION    ${PY_BUILD_CMAKE_MODULE_NAME}
        ARCHIVE     DESTINATION    ${PY_BUILD_CMAKE_MODULE_NAME}
        COMPONENT python_modules)

In theory my CmakeLists.txt should get the rlottie target from rlottie/CMakeLists.txt?

I thought of another way of doing it is by using rlottie/CMakeLists.txt directly and set LIB_INSTALL_DIR to ${PY_BUILD_CMAKE_MODULE_NAME}, but is this possible to do so in pyproject.toml? Does this actually add the built dlls to the wheel?

I am not experienced with C programming and cmake, do you have any suggestions? The documentation is a bit lacking for this kind of project (or I am just a complete fool). Thank you.

Bug: exception when try to install to subfolder with dist-info

Exception
image

Cmake code:

install(FILES "${openvino_tokenizers_SOURCE_DIR}/LICENSE"
              "${openvino_tokenizers_SOURCE_DIR}/third-party-programs.txt"
              "${openvino_tokenizers_SOURCE_DIR}/SECURITY.md"
        DESTINATION "${PY_BUILD_CMAKE_PACKAGE_NAME}-${PY_BUILD_CMAKE_PACKAGE_VERSION}.dist-info/licenses"
        COMPONENT openvino_tokenizers_licenses
        EXCLUDE_FROM_ALL)

When install to "${PY_BUILD_CMAKE_PACKAGE_NAME}-${PY_BUILD_CMAKE_PACKAGE_VERSION}.dist-info - works well.

Feature: support wheel build tag

It's convenient to name wheels like openvino_tokenizers-2024.1.0.2**-90**-py3-none-manylinux_2_17_x86_64.whl with wheel build tag to identify particular build on CI service.

Build number is part of dist-info/WHEEL file. Example:

...
Build: 90

Some solutions allows to specify it via env variable during build process on CI.

unable to build as path dependency when build tool is not globally installed

First, thanks for creating this package! It is super handy.

I would like to report a bug related to using py-build-cmake as build backend of a package a which is a local/path dependency for another package b with a build backend supporting PEP 508 path dependencies (e.g. poetry):

  • During the pre-build phase of a, b is built to generate metadata.
  • The build is done in a temp folder, and the build tool (e.g. ninja) is downloaded to there.
  • For this for-metadata-build, the cmake cache is persistet in .py-build-cmake_cache in the package source folder of b.
  • During the actually build process of a, the build of b fails because the cache references the build tool in the non-existent temp folder.

Proposed solutions:

  • Create cmake cache not relative to source, but in package build folder.
  • As workaround: Add a flag to disable the cmake cache.

Pure C Modules and Packages which import from C modules

Right now there is no way (unless I missed something) to build and distribute a pure C module or a package which imports directly from a C module in __init__.py because py-build-cmake tries to read metadata from the designated python source folder before building the CMake targets.

Similarly, this prevents stubgen from generating stubs for the C modules.

I haven't experimented yet, but I don't see anything preventing us from doing things in the opposite order. Build and place everything in staging, generating metadata from staging, and then using that metadata for source distributions (which would still come from the source folders, obviously) and wheels.

Motivation: For many C modules there's no need to ship a __init__.py or any python source files at all. All you want to ship in the wheel is the final binary and some type stubs. If an __init__.py were necessary for some reason it would be something trivial like:

import sys
sys.modules[__name__] = __import__('CModule.CModule', fromlist=(None,))

In more complex scenarios, packages may want to expose some exports from a CModule in the root package:

"""I am a package with a CModule inside me"""
__version__ = "1.0.1"

from CModule import my_very_fast_function
from OtherPythonStuffThatMightAlsoImportFromCModules import other, things

Which again, isn't supported right now due to the import failing, because we haven't built CModule yet or placed everything in staging.

Given that both pure C modules and packages that want to import from them are common scenarios, py-build-cmake should support them

Minimum Python version

Hello again,

My software runs on an older system that has Python 3.7, but py-build-cmake declares a minimum version of 3.8, so installation fails. Is Python>=3.8 a hard constraint, or is it straightforward to support older versions?

Thanks.

Q: how to automatically detect platform_tag?

Question 1:
Currently, when I built wheel on Linux, platform tag is something like linux_x86_64
How to ensure that it detect current glibc version and set platform tag in manylinix_x_y_arch format?

I don't want to use auditwheel for it, because it's an extra step and not user friendly.

Question 2:
how to specify that my project does not contain Python extension and ensure py3-none is used instead of cp310-cp310

Question 3:
How to install several license files / custom license to the folder dist-info?

nanobind project unable to static link python library on Windows

Original discussion at wjakob/nanobind#262

As adviced, find_python = true was set in py-build-cmake config to help find python in venv, which did help find python in venv (Not sure why this works though).

However...

LINK : fatal error LNK1104: cannot open file 'python38.lib' [D:\a\apngasm-python\apngasm-python\.py-build-cmake_cache\cp38-cp38-win_amd64\_apngasm_python.vcxproj]

See: https://github.com/laggykiller/apngasm-python/actions/runs/5855715610/job/15874048205

Seems like even setting find_python = true does not help find pythonXY.lib? Or venv does not contain pythonXY.lib? Or am I still missing something?

As adviced, I have enabled more verbose debug message in my project. By pure luck, I decided to copy your QueryPythonForNanobind.cmake from develop branch (https://github.com/tttapa/py-build-cmake/blob/develop/examples/nanobind-project/cmake/QueryPythonForNanobind.cmake). Same problem occured on Windows x86 and x64 (Surprisingly not on arm64):

         "D:\a\apngasm-python\apngasm-python\.py-build-cmake_cache\cp38-cp38-win_amd64\ALL_BUILD.vcxproj" (default target) (1) ->
         "D:\a\apngasm-python\apngasm-python\.py-build-cmake_cache\cp38-cp38-win_amd64\_apngasm_python.vcxproj" (default target) (3) ->
         (Link target) -> 
           LINK : fatal error LNK1104: cannot open file 'python38.lib' [D:\a\apngasm-python\apngasm-python\.py-build-cmake_cache\cp38-cp38-win_amd64\_apngasm_python.vcxproj]

See: https://github.com/laggykiller/apngasm-python/actions/runs/5856676134

macOS: client project builds but fails to install

Thanks for working on this project. It seems to be what I'm looking for so I'm trying it out to build a C++ executable in my project which I use as a subprocess. This should be an even simpler use case than the current examples in that I don't actually need to generate Python bindings.

I'm using the latest version 0.0.7.

I can build:

$ python -m build .
* Creating venv isolated environment...
* Installing packages in isolated environment... (py-build-cmake)
* Getting dependencies for sdist...
* Building sdist...
* Building wheel from sdist
* Creating venv isolated environment...
* Installing packages in isolated environment... (py-build-cmake)
* Getting dependencies for wheel...
* Building wheel...
-- The CXX compiler identification is AppleClang 13.1.6.13160021
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    Python3_EXECUTABLE
    Python3_FIND_REGISTRY
    Python3_FIND_STRATEGY
    Python3_ROOT_DIR
    VERIFY_VERSION


-- Build files have been written to: /private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/build-via-sdist-28_rq46r/edgepipe-0.1.0/.py-build-cmake_cache/cp38-cp38-macosx_10_14_arm64
[ 50%] Building CXX object CMakeFiles/sched-pipeline.dir/sched-pipeline.cpp.o
[100%] Linking CXX executable sched-pipeline
[100%] Built target sched-pipeline
Installing component scheduler_applications
-- Installing: /var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/tmpnmmn4n8e/staging/sched/sched-pipeline
Successfully built edgepipe-0.1.0.tar.gz and edgepipe-0.1.0-cp38-cp38-macosx_10_14_arm64.whl

But installation fails (both normal and editable):

$ pip install .
Processing /Users/cimes/local/fogsys/EdgePipe
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [31 lines of output]
      Traceback (most recent call last):
        File "/Users/cimes/local/fogsys/EdgePipe/venv/bin/cmake", line 5, in <module>
          from cmake import cmake
      ModuleNotFoundError: No module named 'cmake'
      Traceback (most recent call last):
        File "/Users/cimes/local/fogsys/EdgePipe/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel
          hook = backend.prepare_metadata_for_build_wheel
      AttributeError: module 'py_build_cmake.build' has no attribute 'prepare_metadata_for_build_wheel'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/Users/cimes/local/fogsys/EdgePipe/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/Users/cimes/local/fogsys/EdgePipe/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/Users/cimes/local/fogsys/EdgePipe/venv/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
        File "/private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/pip-build-env-617ki2fv/overlay/lib/python3.8/site-packages/py_build_cmake/build.py", line 47, in build_wheel
          whl_name = self.build_wheel_in_dir(wheel_directory, tmp_build_dir)
        File "/private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/pip-build-env-617ki2fv/overlay/lib/python3.8/site-packages/py_build_cmake/build.py", line 183, in build_wheel_in_dir
          self.do_native_cross_cmake_build(tmp_build_dir, staging_dir,
        File "/private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/pip-build-env-617ki2fv/overlay/lib/python3.8/site-packages/py_build_cmake/build.py", line 211, in do_native_cross_cmake_build
          self.run_cmake(src_dir, staging_dir, metadata, cmake_cfg, cfg.cross,
        File "/private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/pip-build-env-617ki2fv/overlay/lib/python3.8/site-packages/py_build_cmake/build.py", line 377, in run_cmake
          self.run(configure_cmd, check=True, env=cmake_env)
        File "/private/var/folders/dd/kxwnp72j2q9byml8x9y8k3kr0000gn/T/pip-build-env-617ki2fv/overlay/lib/python3.8/site-packages/py_build_cmake/build.py", line 298, in run
          return sp_run(*args, **kwargs)
        File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 516, in run
          raise CalledProcessError(retcode, process.args,
      subprocess.CalledProcessError: Command '['cmake', '-B', '/Users/cimes/local/fogsys/EdgePipe/.py-build-cmake_cache/cp38-cp38-macosx_10_14_arm64', '-S', '/Users/cimes/local/fogsys/EdgePipe/partition', '-D', 'VERIFY_VERSION=0.1.0', '-D', 'Python3_EXECUTABLE:FILEPATH=/Users/cimes/local/fogsys/EdgePipe/venv/bin/python3', '-D', 'Python3_ROOT_DIR:PATH=/Users/cimes/local/fogsys/EdgePipe/venv', '-D', 'Python3_FIND_REGISTRY=NEVER', '-D', 'Python3_FIND_STRATEGY=LOCATION', '-D', 'CMAKE_BUILD_TYPE=RelWithDebInfo']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

It doesn't show the stdout/stderr for the cmake command that failed. My cmake project depends on another package (located with find_package) and links the executable against it, so perhaps the pip install environment is somehow different than the build environment? It's difficult to tell without additional output.

CMakeLists.txt:

cmake_minimum_required(VERSION 3.8)

project(edgepipe VERSION 0.1.0
                 LANGUAGES CXX)

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)
add_compile_options(-Wall -Wextra)

find_package(yaml-cpp REQUIRED)

add_executable(sched-pipeline sched-pipeline.cpp)
target_link_libraries(sched-pipeline PRIVATE yaml-cpp)

install(TARGETS sched-pipeline
        EXCLUDE_FROM_ALL
        COMPONENT scheduler_applications
        DESTINATION sched)

Snippet from pyproject.toml:

[build-system]
requires = ["py-build-cmake"]
build-backend = "py_build_cmake.build"

[tool.py-build-cmake.module]
directory = "."

[tool.py-build-cmake.sdist]
include = [
    "partition/CMakeLists.txt",
    "partition/sched-pipeline.cpp",
    "partition/yaml_types.h",
    "partition/README.md",
]

[tool.py-build-cmake.cmake]
build_type = "RelWithDebInfo"
source_path = "partition"
build_args = ["-j"]
install_components = ["scheduler_applications"]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.