Giter VIP home page Giter VIP logo

depend_on_what_you_use's Introduction

License: MIT Ruff pre-commit

Depend on what you use (DWYU)

DWYU is a Bazel aspect for C++ projects making sure the headers your Bazel targets are using are aligned with their dependency lists.

DWYUs enforces the design principle:
A cc_* target X shall depend directly on the targets providing the header files which are included in the source code of X.

The main features are:

  • Finding include statements which are not available through a direct dependency, aka preventing to rely on transitive dependencies for includes.
  • Finding unused dependencies.
  • Given one uses implementation_deps, making sure one distinguishes properly between public and private dependencies for cc_library targets. For more details see features chapter Implementation_deps.

More information about the idea behind DWYU and the implementation of the project is available in the docs.

Getting started

Get a release

Choose a release from the release page and follow the instructions.

Get a specific commit

bzlmod (recommended)

Put the following into your MODULE.bazel file

bazel_dep(name = "depend_on_what_you_use", version = "0.0.0")
git_override(
    module_name = "depend_on_what_you_use",
    commit = <commit_you_are_interested_in>,
    remote = "https://github.com/martis42/depend_on_what_you_use",
)

legacy approach

Put the following into your WORKSPACE file to use a specific DWYU commit

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
    name = "depend_on_what_you_use",
    commit = <commit_you_are_interested_in>,
    remote = "https://github.com/martis42/depend_on_what_you_use",
)

load("@depend_on_what_you_use//:setup_step_1.bzl", dwyu_setup_step_1 = "setup_step_1")
dwyu_setup_step_1()

load("@depend_on_what_you_use//:setup_step_2.bzl", dwyu_setup_step_2 = "setup_step_2")
dwyu_setup_step_2()

Use DWYU

Configure the aspect

The features which can be configured through the aspect factory attributes are documented at Configuring DWYU. Put the following inside a .bzl file:

load("@depend_on_what_you_use//:defs.bzl", "dwyu_aspect_factory")

# Provide no arguments for the default behavior
# Or set a custom value for the various attributes
your_dwyu_aspect = dwyu_aspect_factory()

Use the aspect

Invoke the aspect through the command line on a target:
bazel build <target_pattern> --aspects=//:aspect.bzl%your_dwyu_aspect --output_groups=dwyu

If no problem is found, the command will exit with INFO: Build completed successfully.
If a problem is detected, the build command will fail with an error and a description of the problem will be printed in the terminal. For example:

================================================================================
DWYU analyzing: '<analyzed_target>'

Result: Failure

Unused dependencies in 'deps' (none of their headers are referenced):
  Dependency='//some/target/with_an_unused:dependency'
===============================================================================

Create a rule invoking the aspect

You can invoke the aspect from within a rule. This can be useful to make the execution part of a bazel build without having to manually execute the longish aspect build command.

The Bazel documentation for invoking an aspect from within a rule can be found here.

This is demonstrated in the rule_using_dwyu example.

Configuring DWYU

Custom header ignore list

By default, DWYU ignores all header from the standard library when comparing include statements to the dependencies. This list of headers can be seen in std_header.py.

You can exclude a custom set of header files by providing a config file in json format to the aspect:

your_aspect = dwyu_aspect_factory(ignored_includes = "//<your_config_file>.json")

The config file can contain these fields which should be lists of strings. All fields are optional:

Field Description
ignore_include_paths List of include paths which are ignored by the analysis. Setting this disables ignoring the standard library include paths.
extra_ignore_include_paths List of include paths which are ignored by the analysis. If ignore_include_paths is specified as well, both list are combined. If ignore_include_paths is not set, the default list of standard library headers is extended.
ignore_include_patterns List of patterns for include paths which are ignored by the analysis. Patterns have to be compatible to Python regex syntax. The match function is used to process the patterns.

This is demonstrated in the ignoring_includes example.

Skipping targets

If you want the DWYU aspect to skip certain targets and negative target patterns are not an option you can do so by setting the no-dwyu tag on those. You can also configure the aspect to skip targets based on a custom list of tags:

your_aspect = dwyu_aspect_factory(skipped_tags = ["tag1_marking_skipping", "tag2_marking_skipping"])

Another possibility is skipping all targets from external workspaces. Often external dependencies are not our under control and thus analyzing them is of little value. This option is mostly useful in combination with the recursive analysis. You configure it like this:

your_aspect = dwyu_aspect_factory(skip_external_targets = True)

Both options are demonstrated in the skipping_targets example.

Recursion

By default, DWYU analyzes only the target it is being applied to.

You can also activate recursive analysis. Meaning the aspect analyzes recursively all dependencies of the target it is being applied to:

your_aspect = dwyu_aspect_factory(recursive = True)

This is demonstrated in the recursion example.

Implementation_deps

Bazel offers the experimental feature implementation_deps to distinguish between public (aka interface) and private (aka implementation) dependencies for cc_library. Headers from the private dependencies are not made available to users of the library.

DWYU analyzes the usage of headers from the dependencies and can raise an error if a dependency is used only in private files, but not put into the private dependency attribute. Meaning, it can find dependencies which should be moved from deps to implementation_deps.

Activate this behavior via:

your_aspect = dwyu_aspect_factory(use_implementation_deps = True)

Usage of this can be seen in the basic example.

Target mapping

Sometimes users don't want to follow the DWYU rules for all targets or have to work with external dependencies not following the DWYU principles. For such cases DWYU allows creating a mapping which for a chosen target makes more headers available as the target actually provides. In other words, one can combine virtually multiple targets for the DWYU analysis. Doing so allows using headers from transitive dependencies without DWYU raising an error for select cases.

Such a mapping is created with the dwyu_make_cc_info_mapping rule. This offers multiple ways of mapping targets:

  1. Explicitly providing a list of targets which are mapped into a single target.
  2. Specifying that all direct dependencies of a given target are mapped into the target.
  3. Specifying that all transitive dependencies of a given target are mapped into the target.

Activate this behavior via:

your_aspect = dwyu_aspect_factory(target_mapping = "<mapping_target_created_by_the_user>")

This is demonstrated in the target_mapping example.

Verbosity

One can configure the DWYU aspect to print debugging information.

Activate this behavior via:

your_aspect = dwyu_aspect_factory(verbose = True)

Applying automatic fixes

Warning

Please note that the tool cannot guarantee that your build is not being broken by the changes. Always make sure your project is still valid after the changes and review the performed changes.

DWYU offers a tool to automatically fix some detected problems. The workflow is the following:

  1. Execute DWYU on your workspace. DWYU will create report files containing information about discovered problems in the Bazel output directory for each analyzed target.
  2. Execute bazel run @depend_on_what_you_use//:apply_fixes -- <your_options>. The tool discovers the report files generated in the previous step and gathers the problems for which a fix is available. Then, buildozer is utilized to adapt the BUILD files in your workspace.

The apply_fixes tool requires you to explicitly choose which kind or errors you want to be fixed. You can see the full command line interface and more information about the script behavior and limitations by executing:
bazel run @depend_on_what_you_use//:apply_fixes -- --help

If the apply_fixes tool is not able to discover the report files, this can be caused by the bazel-bin convenience symlink at the workspace root not existing or not pointing to the output directory which was used by to generate the report files. The tool offers options to control how the output directory is discovered.

Discovering the DWYU report files automatically can take a large amount of time if the bazel-bin directory is too large. In such cases you can pipe the command line output of executing the DWYU aspect into a file and forward this file to the apply_fixes script via the --dwyu-log-file option. The apply_fixes script will then deduce the DWYU report file locations without crawling though thw whole bazel-bin directory.

Unfortunately, the tool cannot promise perfect results due to various constraints:

  • If alias targets are involved, this cannot be processed properly. Alias targets are resolved to their actual target before the DWYU aspect is running. Thus, the DWYU report file contains the actual targets in its report and buildozer is not able to modify the BUILD files containing the alias name.
  • Buildozer is working on the plain BUILD files as a user would see them in the editor. Meaning without alias resolution or macro expansion. Consequently, buildozer cannot work on targets which are generated inside a macro or whose name is constructed.
  • Adding missing direct dependencies is based on a heuristic and not guaranteed to find the correct dependency.
  • If you execute DWYU only on some targets and not the complete build tree, this can break the overall build. For example dependency X in library A is unused and would be removed. But a downstream user of library A might transitively depend on X. Removing the unused dependency will break the build as the downstream dependency no longer finds dependency X.

Assumptions of use

The code has to be compilable

DWYU is not performing a compilation. It works by statically analyzing the source code and build tree. However, non compiling code can contain errors infringing the assumptions DWYU is based on. For example, including header files which do not exist at the expected path.

Include paths have to be unambiguous

There shall not be multiple header files in the dependency tree of a target matching an include statement. Even if analysing the code works initially, it might break at any time if the ordering of paths in the analysis changes.

Supported Platforms

Aspect

Platform Constraints
Operating system Integration tests check [Ubuntu 22.04, Macos 12, Windows 2022].
Python Minimum version is 3.8. Integration tests check [3.8, 3.9, 3.10, 3.11, 3.12].
Bazel Minimum version is 5.4.0. Integration tests check [5.x, 6.x, 7.x, 8.0.0-pre.x].

Applying fixes

Platform Constraints
Operating system Integration tests check [Ubuntu 22.04, Macos 12, Windows 2022].
Python Minimum version is 3.8. Integration tests check 3.8.
Bazel No known constraint. Integration tests check 7.0.0.
Buildozer No known constraint. Integration tests check 6.4.0.

Alternatives to DWYU

Layering check

To make sure no headers from transitive dependencies or private headers from dependencies are used you can use Layering check with Clang which is natively supported by Bazel. The main benefit of this approach is it being directly integrated into Bazel without need of further tooling like DWYU.

Still, there are reasons to use DWYU instead of or in addition to layering_check:

  • DWYU does not require a compiler, it works purely by text parsing. The only requirement towards your platform is the availability of a Python interpreter.
  • DWYU is able to analyze header only libraries.
  • DWYU detects unused dependencies.
  • DWYU allows optimizing implementation_deps.
  • DWYU offers automatic fixes for detected issues.

Gazelle

Gazelle is a tool automatically creating BUILD files for your code. It seems there is no public and established C++ extension for gazelle.

Still, if one agrees with the best practices enforced by DWYU but cannot use it, investing time into a gazelle C++ extension might be worth it. Automatically generating correct BUILD files based on your source code is a more efficient approach compared to having to manually execute DWYU regularly to make sure no error was introduced.

Versioning

This project uses semantic versioning. Please be aware that the project is still in an early phase and until version 1.0.0 has been reached all releases can contain breaking changes.

The following things can always break and do not promise stability with respect to the semantic versioning:

  • The report files DWYU generates to facilitate running automatic fixes are considered an implementation detail. Changing their content is not considered a breaking change.
  • How to include DWYU in your project might change at any time.

Contributing

See Contributing.

License

Copyright © 2022-present, Martin Medler.
This project licensed under the MIT license.

This projects references several other projects which each have individual licenses. See the content of third_party for all references to other projects.

depend_on_what_you_use's People

Contributors

bayerc avatar cmuck avatar dependabot[bot] avatar freeformstu avatar jsharpe avatar martis42 avatar renovate[bot] avatar storypku avatar vertexwahn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

depend_on_what_you_use's Issues

Automatic Fixes: Be more robust against alias targets

Due to the aspect working on the resolved alias targets but buildozer working on the plain BUILD files alias targets are a problem.
buildozer can't work directly with the targets from the report files, since they don't match the content of the BUILD in case of aliased targets.

However, buildozer can query the content of BUILD files. Maybe we are able to find the targets corresponding to the content of the report file.

DWYU and gRPC

Problem Description

Running DWYU on cc_grpc_library targets will fail with errmsg like below:

DWYU analyzing: '//proto:hello_world_cc_grpc'
Result: FAILURE
Includes which are not available from the direct dependencies:
  File='bazel-out/k8-fastbuild/bin/proto/hello_world.grpc.pb.h', include='grpcpp/impl/codegen/client_callback.h'
  File='bazel-out/k8-fastbuild/bin/proto/hello_world.grpc.pb.h', include='grpcpp/impl/codegen/message_allocator.h'
  File='bazel-out/k8-fastbuild/bin/proto/hello_world.grpc.pb.h', include='grpcpp/impl/codegen/server_context.h'
  File='bazel-out/k8-fastbuild/bin/proto/hello_world.grpc.pb.h', include='grpcpp/impl/codegen/method_handler.h'
  File='bazel-out/k8-fastbuild/bin/proto/hello_world.grpc.pb.h', include='grpcpp/impl/codegen/status.h'

See https://github.com/storypku/dwyu_grpc_demo.git, which was created to illestrate this issue.

Analysis

From DWYU's perspective, this issue was due to the missing direct dependency of @com_github_grpc_grpc//:grpc++_codegen_base. In fact, if @com_github_grpc_grpc//:grpc++_codegen_base was added to deps of //proto:hello_world_cc_proto, running DWYU will succeed.

However, since the cc_grpc_library rule was provided by gRPC, making changes to cc_grpc_library rule implementation seems a better solution for issues like this.

BTW, I have tried either adding grpcpp/impl/codegen stuff to DWYU's ignore headers list, or simply tagged cc_grpc_library targets as "no-dwyu", neither of them seems a good workaround.

Discussion Time

My question is, should we workaround this within DWYU or file an PR for upstream gRPC?

Action failed, file name to long

We are bumping to 0.0.11 and getting failures due to too long file names on Linux. The following is the file that is causing the issue:

src/aspect/dwyu.bzl:
output_path = "{}_processed_dep_{}.json".format(_label_to_name(target.label), _label_to_name(dep.label)),

Action <redacted>.json failed: failed to delete output files before executing action: <redacted>.json 

would it be possible to find a unique filename that is not as long?

Automatic Fixes - Make public dependencies private

Continuation for the automatic fixes feature from #10

Offer a feature to move public dependencies which are used solely privately to the private dependency attribute for cc_library, given one uses the corresponding experimental Bazel feature.

DWYU Doesn't take into account `includes` dirs listed in cc_* rules when resolving headers

Super excited to stumble onto your project :) I wrote a very similar tool for a hackathon! Happy to see your version integrated as a bazel aspect vs a tool built exclusively on top of bazel queries :P

https://github.com/martis42/depend_on_what_you_use/blob/main/src/analyze_includes/evaluate_includes.py#L98
I believe this line only allows for resolving headers that are full paths to the root of a workspace, however, if I have

- a/BUILD
cc_binary(
   name = "a",
   deps = "//b:headers"
)
- a/foo.cc
#include <b.h>
- b/BUILD
cc_library(
   name = "headers",
   hdrs = ["include/b.h"],
   includes = ["include"],
)
- b/include/b.h

This fails to resolve //a:a as using b.h from //b:b

Unable to execute the aspect

Looks like fetch of one of pip packages fails, even though I am able to locate it in /var/cache/pip folder.

ERROR: An error occurred during the fetch of repository 'dwyu_py_deps_pcpp':
   Traceback (most recent call last):
        File "/bazel_cache/output_user_root/61da6a75bcd751b8ee01d9c498363d41/external/rules_python/python/pip_install/pip_repository.bzl", line 438, column 13, in _whl_library_impl
                fail("whl_library %s failed: %s (%s)" % (rctx.attr.name, result.stdout, result.stderr))
Error in fail: whl_library dwyu_py_deps_pcpp failed: Looking in links: /var/cache/pip
Collecting pcpp==1.30
  File was already downloaded /var/cache/pip/pcpp-1.30-py2.py3-none-any.whl
 (Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/bazel_cache/output_user_root/61da6a75bcd751b8ee01d9c498363d41/external/rules_python/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__main__.py", line 4, in <module>
    main()
  File "/bazel_cache/output_user_root/61da6a75bcd751b8ee01d9c498363d41/external/rules_python/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__init__.py", line 68, in main
    whl = next(iter(glob.glob("*.whl")))
StopIteration
)
ERROR:<STRIPPED>/WORKSPACE:1474:18: fetching whl_library rule //external:dwyu_py_deps_pcpp: Traceback (most recent call last):
        File "/bazel_cache/output_user_root/61da6a75bcd751b8ee01d9c498363d41/external/rules_python/python/pip_install/pip_repository.bzl", line 438, column 13, in _whl_library_impl
                fail("whl_library %s failed: %s (%s)" % (rctx.attr.name, result.stdout, result.stderr))
Error in fail: whl_library dwyu_py_deps_pcpp failed: Looking in links: /var/cache/pip
Collecting pcpp==1.30
  File was already downloaded /var/cache/pip/pcpp-1.30-py2.py3-none-any.whl
 (Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/bazel_cache/output_user_root/61da6a75bcd751b8ee01d9c498363d41/external/rules_python/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__main__.py", line 4, in <module>
    main()
  File "/bazel_cache/output_user_root/61da6a75bcd751b8ee01d9c498363d41/external/rules_python/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__init__.py", line 68, in main
    whl = next(iter(glob.glob("*.whl")))
StopIteration
)
ERROR: /bazel_cache/output_user_root/61da6a75bcd751b8ee01d9c498363d41/external/depend_on_what_you_use/src/analyze_includes/BUILD:4:11: @depend_on_what_you_use//src/analyze_includes:lib depends on @dwyu_py_deps_pcpp//:pkg in repository @dwyu_py_deps_pcpp which failed to fetch. no such package '@dwyu_py_deps_pcpp//': whl_library dwyu_py_deps_pcpp failed: Looking in links: /var/cache/pip
Collecting pcpp==1.30
  File was already downloaded /var/cache/pip/pcpp-1.30-py2.py3-none-any.whl
 (Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/bazel_cache/output_user_root/61da6a75bcd751b8ee01d9c498363d41/external/rules_python/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__main__.py", line 4, in <module>
    main()
  File "/bazel_cache/output_user_root/61da6a75bcd751b8ee01d9c498363d41/external/rules_python/python/pip_install/parse_requirements_to_bzl/extract_single_wheel/__init__.py", line 68, in main
    whl = next(iter(glob.glob("*.whl")))
StopIteration
)
ERROR: Analysis of aspects '[//:aspect.bzl%your_dwyu_aspect] with parameters {} on <TARGET>' failed; build aborted: 

bazel version: 6.3.2
dowyu version: 0.13

ignore_include_paths hits include libraries that contain the substring

I'm trying to add "array" to the ignore_include_paths; however, this then causes issues with a custom include that also contains array in the name. I think it would be a simple fix to instead check the strings for equality, but this might break peoples current work flow. Wanted to open a discussion on what the best way to fix this would be!

implementation_deps and cc_test

While testing 0.0.5 on our repo I hit upon a suggestion from DWYU to use implementation_deps on my cc_test target. That attribute seems to only be available for cc_library. I couldn't find a test for this specific test. It should be fairly easy to recreate with use_implementation_deps = True and a cc_test, if not let me know and I will create a MWE, thanks.

Evaluate Dependency Utilization

Is the feature to analyze the dependency utilization indeed useful?

It is not part of the core DWYU concept to make sure only header from direct dependencies are used and no superfluous dependencies exist.
While the implementation of the feature is not complex it is still code which has to be tested, documented and maintained.
On top, the feature is not reliable when virtual includes are involved.
Furthermore, most likely an extension will be required to allow users to specify an allow list of dependencies where low utilizations are accepted for whatever reason.

Then again, having small targets allows building slim and efficient dependency trees and this feature can help with that.
Moving this into an own project seems wasteful, since the implementation would be mostly identical to DWYU.

Current ideas:

  • Remove it, if user feedback for this is not positive. It is not part to invest much here, since it is not a core DWYU feature.
  • If user feedback is positive, move it into an own tool/aspect but reuse the DWYU implementation.

Document design philosophy

Extend the documentation to explicitly explain why one should only use headers from direct dependencies.

Automatic Fixes: bazel compilation-mode was not respected.

If we first run bazel build -c opt --config=dwyu //somepath/... then followed by a bazel run -c opt @depend_on_what_you_use//:apply_fixes -- --workspace=/path/to/my/project, apply fixes won't try to glob xxx_dwyu_report.json files under the actual bazel-bin directory:

ls -ld bazel-bin
lrwxrwxrwx 1 story story 93 May  8 17:30 bazel-bin -> /home/story/.cache/bazel/5f6cfd791ed6436cd7d0a9ad06515f99/execroot/com_qcraft/bazel-out/k8-opt/bin

shows that it was symlinked to some xxx/k8-opt/bin directory

However, the directory from running bazel info bazel-bin was xxx/k8-fastbuild/bin (See below).

$ bazel info bazel-bin
/home/story/.cache/bazel/5f6cfd791ed6436cd7d0a9ad06515f99/execroot/com_qcraft/bazel-out/k8-fastbuild/bin

Shall we use readlink bazel-bin rather than bazel info bazel-bin to locate the actual bazel-bin directory ?

Use the tag `no-dwyu` to opt out of DWYU on specific targets

Hi, I'd like to propose using the tag no-dwyu to allow opting out of DWYU on specific targets.

We use some external dependencies which do not work with dwyu.

It would be extremely useful to be able to opt out of these checks for a few targets while still getting the benefits for all other targets.

seeing false reports of non unique headers.

seeing errors of the form:
Found multiple targets which potentially can provide include 'rt/libs/libspatial/mechanisms/ids.h' of target '@//rt/libs/libtrackingutils/state_trajectory_tracking:kcmpc'.
Please fix this manually. Candidates which have been discovered:

  • //rt/libs/libkalman2:libkalman2
  • //rt/libs/libspatial/mechanisms:ids
    It seems like the code might be confused, there are two header files named ids.h but they are each included with fully qualified paths, seems like the code might be missing that fact.

apply fixes cannot resolve unused deps from implementation or interface deps

The aspect correctly detects unused implementation or interface deps (although tests are missing). But the report file does not contain the information which attribute holds the unused dependency and the apply_fixes script always assumes it s working with the deps attribute.

  • Add tests for unused implementation_deps and interface_deps
  • Add the dependency attribute containing the unused dependency to the report file
  • Adapt the apply fixes script be able to remove dependencies from any dependency attribute
  • Addd tests for automatically removing unused implementation_deps and interface_deps

Conflict between implementation_deps and deps when using target mapping

I'm not sure if the issue I'm encountering depends on some miscofiguration on my part or it is a bug in the tool, but I'll report it here.
There are some configurations, even very simple ones, where using the use_implementation_deps=True flag creates unsatisfaiable requirements for dwyu if a target map is specified, even when the map itself is completely empty.

What follows is a simple example i was able to prepare

# MODULE.bazel
module(name = "example")

bazel_dep(name = "rules_cc", version = "0.0.9")
bazel_dep(name = "fmt", version = "10.2.1")
bazel_dep(name = "spdlog", version = "1.12.0")
bazel_dep(name = "depend_on_what_you_use", version = "0.0.0")
git_override(
    module_name = "depend_on_what_you_use",
    commit = "b817c225d79c492c25b28872445e34be4a36d5aa",
    remote = "https://github.com/martis42/depend_on_what_you_use",
)
# BUILD.bazel
load("@depend_on_what_you_use//:defs.bzl", "MAP_TRANSITIVE_DEPS", "dwyu_make_cc_info_mapping")

package(default_visibility = ["//visibility:public"])

exports_files([
    "ignore_includes.json",
])

# DWYU mappings
dwyu_make_cc_info_mapping(
    name = "dwyu_mapping",
    mapping = {
# Empty mapping
    },
)
# dwyu.bzl
load("@depend_on_what_you_use//:defs.bzl", "dwyu_aspect_factory")

dwyu = dwyu_aspect_factory(use_implementation_deps = True, recursive = True, skip_external_targets = True, target_mapping = Label("//:dwyu_mapping"))
# main/BUILD.bazel
load("@rules_cc//cc:defs.bzl", "cc_binary", "cc_library")

cc_library(
    name = "logging",
    hdrs = ["logging.h"],
    deps = ["@fmt", "@spdlog"],
)

cc_library(
    name = "lib",
    srcs = ["lib.cpp"],
    hdrs = ["lib.h"],
    # Commenting one and uncommenting the other does not satisfy dwyu
    # implementation_deps = [":logging"], 
    deps = [":logging"],
)

cc_binary(
    name = "main",
    srcs = ["main.cpp"],
    deps = [":lib"],
)
//----------------------------
// In main/lib.h

// Empty

//----------------------------
// In main/lib.cpp

#include "lib.h"
#include "logging.h"

//----------------------------
// In main/logging.h

#ifndef NLOG
#include <fmt/core.h>
#include <spdlog/spdlog.h>
#else
#endif

//----------------------------
// In main/main.cpp

#include "lib.h"

Running

bazel build --aspects=//:dwyu.bzl%dwyu --output_groups=dwyu //main

Output if using deps

================================================================================
DWYU analyzing: '@@//main:lib'

Result: FAILURE

Public dependencies which are used only in private code:
  Dependency='@@//main:logging'
================================================================================

Output if using implementation_deps

================================================================================
DWYU analyzing: '@@//main:lib'

Result: FAILURE

Includes which are not available from the direct dependencies:
  File='main/lib.cpp', include='logging.h'
================================================================================
Aspect //:dwyu.bzl%dwyu of //main:main failed to build

EDIT:

bazel version 7.1.1

Support more include patterns

The known limitations are caused by the complexity of all possible ways to include a header in C++.
The complexity is caused mainly by the preprocessor which allows dynamically adding include statements through macros and to enable/disable includes based on defines.

We could reimplement the preprocessor logic inside DWYU to support more complex include patterns.
The benefit would be that DYWU remains self-contained without external dependencies.
However, this would also be reinventing the wheel.
The C preprocessor is an old problem which has been solved many times until now.
Trying to implement and test a C preprocessor solely for the sake of prevent third party dependencies is wasteful.
The world does not need yet another C Preprocessor.

No design how to proceed has been made up yet.

The following tools could be part of the solution if we want to keep Python as implementation language:

We could also move the implementation to C++.
The LLVM toolchain is a quite powerful toolset and allows interaction with the preprocessor.
See for example https://clang.llvm.org/doxygen/classclang_1_1Preprocessor.html#details.
LLVM is however not a lightweight dependency and having to compile DWYU will reduce its portability.

Yet another possibility is telling the compiler to generate a .d file, which lists the includes required for a given source file.
This provides an easy to parse output listing all include statements which are required after resolving macros and defines.
However, a first test shows all transitive headers are resolved.
This is a problem, since we need to know the include statements for the file under inspection, but not more.
If all transitive includes are listed as well, we cannot compare this list to the direct dependencies.
It is not yet known if this behavior can be modified.

6fb152f

Offer function to load required dependencies

Offer a function which allows loading the dependencies (e.g. rules_python). Should utilize maybe to allow others overwriting the version by loading the dependency upfront in a specific version.

automatic fixes fail for aliases

Automatic fixes for alias targets do not work. DWYU reports the resolved alias. However, buildozer will fail to remove this resolved target from a dependency list. After all, the BUILD file has the alias in the dependency list and not the resolved target.

Announcement: Raising Minimum Python Version to 3.8

In the near future we will raise the minimum required Python version from 3.6 to to 3.8.
There are multiple reasons for this:

  • Python 3.7 is reaching soonish the end-of-life at 2023-06-27, see https://devguide.python.org/versions/
  • One of the benefits when working with Bazel is the sandboxed execution and thus the high reproducability across different machines, given one uses a hermetic toolchains. rules_python offers ready to use hermetic Python toolchains. They are however only available for Python 3.8 and later.
  • We can use more modern Python features. Some examples are dataclass and the leaner subprocess.run API.

If anybody has concerns regarding this, please comment here.

Report potential regression in Bazel 7

Invoking DWYU with Bazel 7 and implementation_deps activated on a cc_test target fails, which is why we have to disable the corresponding test.

Error:

ERROR: .../e663c61e10a8ed8d3fb71670c233b37b/external/local_jdk/BUILD.bazel:2:10: in fail_rule rule @local_jdk//:jdk: 
Traceback (most recent call last):
	File ".../e663c61e10a8ed8d3fb71670c233b37b/external/bazel_tools/tools/jdk/fail_rule.bzl", line 19, column 13, in _fail_rule_impl
		fail("%s %s" % (ctx.attr.header, ctx.attr.message))
Error in fail: Auto-Configuration Error: Cannot find Java binary bin/java in .../e96122088fa3cf13e8284bd38097b13e/embedded_tools/tools/jdk/nosystemjdk; either correct your JAVA_HOME, PATH or specify Java from remote repository (e.g. --java_runtime_version=remotejdk_11)
ERROR: .../e663c61e10a8ed8d3fb71670c233b37b/external/local_jdk/BUILD.bazel:2:10: Analysis of target '@local_jdk//:jdk' failed
ERROR: .../e663c61e10a8ed8d3fb71670c233b37b/external/remote_coverage_tools/BUILD:10:12: errors encountered resolving toolchains for @remote_coverage_tools//:Main
ERROR: Analysis of target '//test/aspect/implementation_deps:test_using_foo' failed; build aborted: 
INFO: Elapsed time: 0.154s
INFO: 0 processes.

TODO

Create a minimal example for reproducing and report upstream as regression for Bazel 7.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Awaiting Schedule

These updates are awaiting their schedule. Click on a checkbox to get an update now.

  • chore(deps): update dependency typing-extensions to v4.12.1

Detected dependencies

bazel
third_party/dependencies.bzl
  • bazel_skylib 1.7.0
bazel-module
MODULE.bazel
  • bazel_skylib 1.7.0
github-actions
.github/workflows/ci.yaml
  • actions/checkout v4
  • snok/install-poetry v1
  • actions/setup-python v5
  • pre-commit/action v3.0.1
  • actions/checkout v4
  • actions/checkout v4
  • actions/checkout v4
  • actions/checkout v4
  • ubuntu 22.04
  • ubuntu 22.04
.github/workflows/release.yaml
  • actions/checkout v4
  • softprops/action-gh-release v2
pip_requirements
third_party/mypy_requirements.in
  • mypy ==1.10.0
third_party/mypy_requirements.txt
  • mypy ==1.10.0
  • mypy-extensions ==1.0.0
  • tomli ==2.0.1
  • typing-extensions ==4.12.0
poetry
pyproject.toml
  • python ^3.9.0
  • pre-commit 3.7.1
pre-commit
.pre-commit-config.yaml
  • pre-commit/pre-commit-hooks v4.6.0
  • executablebooks/mdformat 0.7.17
  • keith/pre-commit-buildifier 6.4.0
  • astral-sh/ruff-pre-commit v0.4.7

  • Check this box to trigger a request for Renovate to run again on this repository

Setup Tests for various Python versions

DWYU is mainly developed based on Python 3.8. However, some users use older Python versions. We should setup test execution with multiple Python versions to make sure support for old Python versions does not break.

Quote include of file in same directory failing to be found

This time, repro'ed within DWYU test case :)

looks like if you have a single build rule with two headers in sub directories, one including the other, DWYU doesn't find the locally sourced header.

Tested with:
bazel build --aspects=//my_awesome:aspect.bzl%your_dwyu_aspect --output_groups=cc_dwyu_output --override_repository=depend_on_what_you_use=/home/michael.johnson/code/depend_on_what_you_use @depend_on_what_you_use//test/aspect/includes:includes2

0001-Add-relative-include.patch

Support Tree Artifacts

Code generators often use tree artifacts to model their outputs.
DWYU should support those as well.

Problem with private includes

I think a regression was introduced in a31db72 and is not working on latest.

We have the following module:

$ tree foo/
foo/
├── BUILD.bazel
├── foo.h
└── src
    ├── foobar.h
    └── foo.cpp
$ cat foo/BUILD.bazel 
cc_library(
    name = "foo",
    srcs = [
        "src/foo.cpp",
        "src/foobar.h",
    ],
    hdrs = [
        "foo.h",
    ],
    deps = [],
)
$ cat foo/src/foo.cpp 
#include "foo/foo.h"

#include "foobar.h"
// #include "foo/src/foobar.h"  // changing to this makes it work again.

int foobar()
{
    return 42;
}
================================================================================
DWYU analyzing: '@//foo:foo'

Result: FAILURE

Includes which are not available from the direct dependencies:
  File='foo/src/foo.cpp', include='foobar.h'
================================================================================

This worked fine in f2ecd05. Was this change intentional or unintentional?

Automatically deduce globally available headers

Right now we maintain an allow list of headers which one can use without dedicated dependency.
If the aspect would depend on the active CC toolchain and query the headers which are available through the toolchain, we might be able to remove this manually maintained list of standard headers.

apply_fixes extremely slow

Hi,
I'm experiencing incredibly long run times when I run the script that applies the fixes.

I'm doing this on a single target, the actual dwyu run takes just a handful of seconds, and then I run bazel run @depend_on_what_you_use//:apply_fixes -- --fix-all and it runs for hours. I kid you not. Hours.

Am I doing something wrong? Is it stuck on something but not telling me ?
Do you have any tips for me so that I can provide more useful information without posting here my company's code ?

Thanks!

Project Status

Currently I cannot invest much time into this project.

You can expect me to answers questions, but I won't be able to perform much coding.
I am however still very much interested in this project. I intend to invest more time into this again in autumn and to reach version 1.0.0 before Bazel 6.0.0 is released.

apply_fixes missing_deps discovery respects visibility

Followup to #18

Currently the feature to automatically add missing direct dependencies does not consider if a discovered dependency is visible to the target under inspection.

Assuming we make no error in matching include statement to target, there are 2 cases we can resolve by implementing this:

  • If multiple targets in the dependency tree provide a given header we skip this right now with a warning. With visibility we could potentially decide which target to choose.
  • Even if we find only a single dependency, it might be invalid. If aliases are involved we might add a target which breaks the build, although the added dependency provides the required header.

Announcement: Raising Minimum Bazel Version to 5.0.0

In the near future we will raise the minimum required Bazel version from 4.0.0 to to 5.0.0.
There are multiple reasons for this:

  • Bazel is a rapidly developing tool. We aim to be up to date with the latest developments. Version 4.0.0 will reach end of life in 2023 either way.
  • Many of the established rule sets already require a Bazel. version > 5.0.0.
  • Several integration tests don't work with Bazel < 5.0.0.
  • One of the main features of DWYU is enforcing implementation_deps which is only available starting with Bazel 5.0.0.
  • We want to support the new dependency management concept bzlmod eventually, which is however only available starting with Bazel 5.0.0.

If anybody has concerns regarding this, please comment here.

Make aspect impl targets private

When the minimum Bazel version is new enough that we can expect all users to have set --incompatible_visibility_private_attributes_at_definition or it being enforced by Bazel, we should restrict the visibility of our implementation details.

Feature request: "export" functionality

Hi!

I was wondering if there are plans for adding some sort of "export" functionality to DWYU, similar to the equivalent functionality in IWYU. We currently have issues on googletests, because DWYU forces us to duplicate the deps:

        "@googletest//:gtest",
        "@googletest//:gtest_main",

It would be great if we could tell DWYU that gtest_main exports gtest, so it's fine to include only gtest_main.

Thanks!

Respect defines from included headers

Right now we only process defines which are set inside a file under inspection or are injected through Bazel (rule attributes, toolchain, ...).
If a define X is set in a header A which is included in file B, parsing file B is treating X as undefined.

By improving how we manage include paths and make them available to the code parser and pcpp preprocessor, we should be able to resolve this.

Consecutive apply_fixes invocations might fail

Follow up on an issue raised in #39

If one performs the apply_fixes script multiple times, consecutive runs will fail due to the report files being outdated and the issues already being resolved.

A solution proposal does not yet exist. It would be trivial to consume the report files by deleting them after using them as input. However, this would meddle with bazel-out, which seems like a hack.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.