lifting-bits / cxx-common Goto Github PK
View Code? Open in Web Editor NEWCommon dependency management for various Trail of Bits C++ codebases
License: Apache License 2.0
Common dependency management for various Trail of Bits C++ codebases
License: Apache License 2.0
The python-pip
and python-lzma
packages no longer install via apt-get
. Possibly more issues as well.
I think building LLVM and Clang should be required, not optional, and it should happen first. The built LLVM/Clang should then be used to build all other things. The key is to get the C++ ABI versioning tag right, e.g. when building a llvm 3.6-based toolchain on ubuntu 16.04.
The LLVM installer is currently ignoring the version specified in the command line, and it is always defaulting to 4.0.
Additionally, the 3.5 version can't be obtained from the GitHub mirror, because it's missing a component (libc++abi).
This blog looks very interesting and something we should keep track of. Currently it's only aimed at support embedded developers, but eventually support for desktop development would be nice.
Hi, I encountered the following compilation error caused by what seems to be an old bug. Here are some related threads and the upstream patch about the error.
My environment was using Clang 12.0.1. The output is for llvm-10, but the same happened for llvm 9 and 11.
I'm guessing that the error only manifests under certain compiler versions or flag combinations?
FAILED: utils/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o
/usr/bin/clang++ -DHAVE_POSIX_REGEX -DHAVE_STD_REGEX -DHAVE_STEADY_CLOCK -D_DEBUG -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/x64-linux-rel-rel/utils/benchmark/src -I/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src -I/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/x64-linux-rel-rel/include -I/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/include -I/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/include -I/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/../include -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fPIC -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -std=c++11 -Wall -Wextra -Wshadow -pedantic -pedantic-errors -Wshorten-64-to-32 -Wfloat-equal -fstrict-aliasing -fno-exceptions -Wstrict-aliasing -Wthread-safety -O3 -UNDEBUG -std=c++14 -MD -MT utils/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o -MF utils/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o.d -o utils/benchmark/src/CMakeFiles/benchmark.dir/benchmark_register.cc.o -c /home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.cc
In file included from /home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.cc:15:
/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.h:17:30: error: no member named 'numeric_limits' in namespace 'std'
static const T kmax = std::numeric_limits<T>::max();
~~~~~^
/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.h:17:45: error: 'T' does not refer to a value
static const T kmax = std::numeric_limits<T>::max();
^
/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.h:8:20: note: declared here
template <typename T>
^
/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.h:17:47: error: no member named 'max' in the global namespace; did you mean 'kmax'?
static const T kmax = std::numeric_limits<T>::max();
^~~~~
kmax
/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.h:17:18: note: 'kmax' declared here
static const T kmax = std::numeric_limits<T>::max();
^
/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.h:17:18: error: default initialization of an object of const type 'const long'
static const T kmax = std::numeric_limits<T>::max();
^
= 0
/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.cc:270:3: note: in instantiation of function template specialization 'AddRange<long>' requested here
AddRange(&arglist, start, limit, range_multiplier_);
^
In file included from /home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.cc:15:
/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.h:17:18: error: default initialization of an object of const type 'const int'
static const T kmax = std::numeric_limits<T>::max();
^
= 0
/home/kyc/cxx-common-0.1.4/vcpkg/buildtrees/llvm-10/src/org-10.0.1-1ebdccd6cb.clean/llvm/utils/benchmark/src/benchmark_register.cc:417:3: note: in instantiation of function template specialization 'AddRange<int>' requested here
AddRange(&thread_counts_, min_threads, max_threads, 2);
^
5 errors generated.
Low priority.
Make our builds use -fuse-ld=lld
. It should link considerably faster and we already depend on LLVM.
I'm unable to compile McSema and remill on windows.
Any chance the workflow can be modified to add these artifacts?
A major release of vcpkg has just been finalized that moves some important features from experimental to fully supported https://devblogs.microsoft.com/cppblog/all-vcpkg-enterprise-features-now-generally-available-versioning-binary-caching-manifests-and-registries/
For this repo, the registries feature makes the most sense https://learn.microsoft.com/en-us/vcpkg/maintainers/registries
One weird thing that we need is support for patching older versions of LLVM. I haven't yet looked too hard at how support for patching old LLVM versions would be or what it would look like if we followed vcpkg's versioning scheme where only a single version of a package/library exists.
We have more and more need to use arm64 versions of cxx-common. Lets ensure we have a working version for Linux/arm64 and Mac/arm64 building in CI and saving to releases.
We can use AWS for these tasks, although kind of awkwardly, since AWS has both MacOS/arm64 and Linux/arm64 machines available.
I think this is related to 63b879a.
The vcpkg_windows-vs2019_llvm-11_amd64.7z file on v0.1.4 cannot be opened with 7zip or winrar, both complaining that the file is corrupted. Additionally the llvm-12 release is missing for v0.1.4.
Both windows 7z's for v0.1.3 open without issue.
With the recent addition of LLVM 14, there have been a few bugs that prevent expected behavior during our tool builds. These bugs could have been prevented with a simple LLVM project that can be used as a smoke-test to make sure certain features of the LLVM package(s) are behaving as expected.
The simple LLVM project should have the following features (feel free to request more):
LLVM
librariesClang
librariesClang
C and C++ compiler to compile sample source filesMLIR
librariesOn my machine I had xcode installed, but still had the command line tools selected. This causes the build to fail when configuring LLVM. If it is a requirement to have xcode installed and selected it would be good to detect that up front and inform the user how to fix the problem.
Vanilla LLVM is building for me without issues with just the command line tool though, so not sure why there is an error in your port.
#959 added a patch that introduced a new feature in LLVM.
Upstream vcpkg does not allow patches that add new features, but since this is a repo dedicated to supporting the Lifting Bits organization tools, I allowed it.
However, there is valid concern that this patch could introduce bugs.
The proposed solution is to gate the patch behind a feature flag pasta
and only include the patch if that feature is selected. Furthermore, the list of downloads for pre-built cxx-common will now have both the vanilla LLVM and patched LLVM per machine configuration.
It looks like build-dependencies.sh
currently only works on Linux (on macos 12 I get errors related to https://github.com/lifting-bits/cxx-common/blob/master/build_dependencies.sh#L95).
Would be nice if this script was rewritten in python or cmake so Windows can also be supported.
Our self hosted runner is up! This will let us test arm64 builds.
Matrix in CI tests for:
runs-on: self-hosted
This has the tags Linux
and ARM64
We would like Windows support for the GHA AWS runner. To do that:
... whereas trying to pull them from a server based in NY was much faster.
(DigitalOcean SF = ~40m download time, DigitalOcean NY = ~1m download time).
Hello, there!
As part of the university research we are currently doing regarding the security of Github Actions, we noticed that one or many of the workflows that are part of this repository are referencing vulnerable versions of the third-party actions. As part of a disclosure process, we decided to open issues to notify GitHub Community.
Please note that there are could be some false positives in our methodology, thus not all of the open issues could be valid. If that is the case, please let us know, so that we can improve on our approach. You can contact me directly using an email: ikoishy [at] ncsu.edu
Thanks in advance
The vulnerability fix that is missing by actions' versions could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider updating the reference to the action.
If you end up updating the reference, please let us know. We need the stats for the paper :-)
Modules were initially installed in the /repository/path/cmake folder. We then added the CMake to the list of packages that can be built, and we accidentally merged the installation prefix with the directory that contained are .cmake files.
Package: llvm-15[clang,compiler-rt,core,cxx-common-targets,default-options,enable-assertions,enable-bindings,enable-eh,enable-rtti,enable-threads,enable-z3,enable-zlib,mlir,target-aarch64,target-arm,target-nvptx,target-sparc,target-webassembly,target-x86,tools,utils]:arm64-osx -> 15.0.1
Host Environment
To Reproduce
vcpkg install xed z3 glog gtest gflags protobuf abseil grpc[absl-sync,codegen] flatbuffers roaring rocksdb[zstd] capnproto cpp-httplib re2 zstd doctest llvm-15
Failure logs
-- Using cached llvm-llvm-project-llvmorg-15.0.1.tar.gz.
-- Cleaning sources at /Users/tetsuo/Code/cxx-common/vcpkg/buildtrees/llvm-15/src/org-15.0.1-7e77ecfa07.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source /Users/tetsuo/Code/cxx-common/vcpkg/downloads/llvm-llvm-project-llvmorg-15.0.1.tar.gz
-- Applying patch 0001-Fix-install-paths.patch
-- Applying patch 0002-Fix-DR-1734.patch
-- Applying patch 0003-Fix-tools-path.patch
-- Applying patch 0004-Fix-compiler-rt-install-path.patch
-- Applying patch 0005-Fix-tools-install-path.patch
-- Applying patch 0006-Fix-libffi.patch
-- Applying patch 0007-Fix-install-bolt.patch
-- Applying patch 0020-fix-FindZ3.cmake.patch
-- Applying patch 0021-fix-find_dependency.patch
-- Applying patch 0023-clang-sys-include-dir-path.patch
-- Applying patch 0024-remove-elf_relocation-checks.patch
-- Applying patch 0026-fix-prefix-path-calc.patch
-- Using source at /Users/tetsuo/Code/cxx-common/vcpkg/buildtrees/llvm-15/src/org-15.0.1-7e77ecfa07.clean
-- Found external ninja('1.11.0').
-- Getting CMake variables for arm64-osx
-- Configuring arm64-osx
-- Building arm64-osx-dbg
-- Building arm64-osx-rel
CMake Error at /Users/tetsuo/Code/cxx-common/vcpkg-llvm-15-install/installed/arm64-osx/share/vcpkg-cmake-config/vcpkg_cmake_config_fixup.cmake:36 (message):
'/Users/tetsuo/Code/cxx-common/vcpkg/packages/llvm-15_arm64-osx/debug/lib/cmake/clang'
does not exist.
Call Stack (most recent call first):
/Users/tetsuo/Code/cxx-common/ports/llvm-15/portfile.cmake:334 (vcpkg_cmake_config_fixup)
/Users/tetsuo/Code/cxx-common/ports/llvm-15/portfile.cmake:348 (llvm_cmake_package_config_fixup)
scripts/ports.cmake:147 (include)
Additional context
I tried building LLVM 15 on my M1 Mac. Not sure what happened but it seems to be looking for Debug Clang artifacts during the Release build?
Configuring the cache may solve the build timeout when Travis is running.
Hi, team
We are so happy to see there were multi-arch support in cxx-common, but we could only see x86 tar.xz releases in [1].
Do we have any plan to add the Aarch64 release in there? Thanks!
There is a lot of duplicated code for setting up and tearing down the AWS runners, in addition to any code that is duplicated for the actual building of the dependencies if we ever want to support multiple LLVM versions.
I think composite GitHub Actions could help to reduce duplication while still providing a flexible way to build what/how we want.
After #961 lands, we will be duplicating much of the same binary artifacts. While GitHub public repos technically have unlimited storage space, this still feels wasteful and is not good for scaling if any other features come up.
vcpkg isn't built like Homebrew to favor a pre-built cache of artifacts. This makes it difficult to distribute anything but a complete installation directory of vcpkg artifacts.
MLIRConfig.cmake contains a broken mlir-tblgen
path on llvm-11 / Ubuntu 20.04 / amd64
set(MLIR_TABLEGEN_EXE "/__w/cxx-common/cxx-common/vcpkg/buildtrees/llvm-11/x64-linux-rel-rel/NATIVE/bin/mlir-tblgen")
should be something like
set(MLIR_TABLEGEN_EXE "${MLIR_INSTALL_PREFIX}/tools/llvm/mlir-tblgen")
Ubuntu 16.04
After a (long) time it will error out.
./travish.sh linux build
...
Installing the CMake modules...
===
> Erasing the LLVM and CMake build folders...
> Re-launching the build script using the newly built clang...
===
Build type: Release
Platform type: linux
Updating the PATH environment...
Setting the C compiler: /opt/trailofbits/bootstrap/llvm/bin/clang
Setting the C++ compiler: /opt/trailofbits/bootstrap/llvm/bin/clang++
Invalid package: protobuf
This issue is to keep track of frequently asked questions (FAQ), complaints, and suggestions for this repo. It is primarily for indexing and will link to other issues or discussions, if they are relevant.
Feel free to comment with something if you don't feel like it deserves its own issue. We can create an issue or discussion if a comment or topic deserves a longer conversation.
There are currently 6 ARM CI workflows (3 latest LLVM versions by the 2 latest LTS Ubuntu distributions), but since we only have 1 available worker, I think we should reduce it to at most 2:
What do you think of this @artemdinaburg
It should be possible to cross compile to arm64 for M1 using the Github MacOS runner
This will be more convenient for M1 users so they can grab a prebuilt artifact from Github instead of compiling it from source
This might only be an issue on macOS, I'm not sure. The net result is that, without this, glog cannot be configured on the command line.
We should be better about removing old packages that are no longer used.
We could use this GitHub Action https://github.com/actions/delete-package-versions
A good start might be to delete all but 5 versions of a package (to account for multiple PRs that could be open).
Packages and their info can be viewed here https://github.com/orgs/lifting-bits/packages?repo_name=cxx-common
Related to #961.
As the title says, this should be ported to Windows
vcpkg is nice when everything is packaged correctly, however it is very difficult to share the result of a single pre-compiled package with someone else to install into their existing set of packages. This design is nice to ensure maximum compatibility for a single machine by hashing almost every component that goes into building the package (compiler, source code, patches, cmake scripts, dependencies, etc.), but it is very difficult to share pre-compiled packages by themselves.
On the other hand, package managers like Homebrew get away with compiling a package once (and any dependencies) and distributing the result to all macOS users on the same OS version. Mac is relatively easy to do this because there is really only a single target operating system. Linux is more difficult if you aren't careful about libc and lib(std)c++ compatibilities, but Homebrew seems to support Linux, and we could only officially target Ubuntu 20.04.
The nice thing about vcpkg is that you only need to specify the CMAKE_TOOLCHAIN_FILE
to choose which dependency set you'd like to compile against. With Homebrew, it's more of a global installation (kind of), which would require manually specifying package directories like -DLLVM_DIR=/usr/local/opt/llvm...
Known issue on LLVM 11 when trying to compile with gcc sanitizers https://reviews.llvm.org/D93316?id=312385
Probably an issue on previous versions as well.
Should be a small patch that can easily be included in the llvm ports.
I am a little confused about how to build it on windows 7 because I only see the dependent software and I don’t know how run the .sh file and what the parameters should I add. Thanks a lot.
Something like
./build_dependencies.sh --asan --export-dir vcpkg-llvm-12-install llvm-12
Or after installing everything but llvm-12:
./vcpkg/vcpkg install --triplet=arm64-osx-asan '@overlays.txt' llvm-12
Results in a failure to configure zlib:
CMake Error at cmake/config-ix.cmake:134 (message):
Failed to configure zlib
Call Stack (most recent call first):
CMakeLists.txt:670 (include)
Still investigating
If we want to use the clang
compiler built with vcpkg as a self-contained toolchain, we should also have the option to build the target project with the equivalent sanitizer support.
I think this might be enabled by building the compiler-rt
project https://compiler-rt.llvm.org/
Initially, this should be limited to the Address and Undefined sanitizers at first, both to reduce release artifact blowup and because other sanitizers like Memory sanitizer require instrumented standard c++ library.
The following command should work in order to close this issue:
CXXFLAGS="-fsanitize=address -fno-omit-frame-pointer -fno-optimize-sibling-calls -ffunction-sections -fdata-sections -Wl,-undefined,dynamic_lookup" \
CFLAGS="-fsanitize=address -fno-omit-frame-pointer -fno-optimize-sibling-calls -ffunction-sections -fdata-sections -Wl,-undefined,dynamic_lookup" \
LDFLAGS="-fsanitize=address -Wl,-undefined,dynamic_lookup" \
cmake -G Ninja -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_INSTALL_PREFIX=$(pwd)/install -DVCPKG_ROOT="/Users/ekilmer/src/cxx-common/vcpkg" ..
Currently, it errors with something like:
/usr/bin/ld: cannot find ....
<vcpkg_root>/installed/x64-linux-rel/lib/clang/11.0.0/lib/linux/libclang_rt.asan-x86_64.a: No such file or directory
We have a user who would benefit from this lifting-bits/remill#641
The implementation in cxx-common will need to use cross-compilation on the x86 GitHub Action runners, possibly similar to how we generate ARM binaries for Mac.
The if
statement on line 89 of build_dependencies.sh uses if [[ ... || ... ]]
but this doesn't work on macOS on an M1; I think if [ ... || ... ]
is needed. However, when I make this change, I get a syntax error about a missing ]
, and an error about CC
being unbound. I figured the former was due to the latter, and defining CC
and CXX
prior to invoking the script worked to fix the problem.
If someone uses the build_dependencies.sh
script to build everything from source, there should be an option (or another script) to run that will export the installation directory (similar to CI) to a directory of the user's choice.
We need this because we cannot have multiple LLVM versions installed in the same installation PREFIX.
Even though vcpkg caches the individual built libraries to its own cache directory (which makes re-running the script fast, as along as all hashes match up), it would be nice to have more persistent directories for each LLVM versions to exist simultaneously.
A workflow might look like:
./build_dependencies.sh --release --export-dir /opt/trailofbits/vcpkg-llvm_11-lifting-bits llvm-11
Internally, this would use vcpkg's official export
command, like CI and do a mv
.
So then you can more easily switch between LLVM versions:
$ ./build_dependencies.sh --release --export-dir /opt/trailofbits/vcpkg-llvm_11-lifting-bits llvm-11
$ cmake -G Ninja -DCMAKE_INSTALL_PREFIX=$(pwd)/install -DVCPKG_ROOT=/opt/trailofbits/vcpkg-llvm_11-lifting-bits ..
...
$ ./build_dependencies.sh --release --export-dir /opt/trailofbits/vcpkg-llvm_10-lifting-bits llvm-10
$ cmake -G Ninja -DCMAKE_INSTALL_PREFIX=$(pwd)/install -DVCPKG_ROOT=/opt/trailofbits/vcpkg-llvm_10-lifting-bits ..
...
Ubuntu 18.04 uses an old version of CMake which makes writing/refactoring CMake files more difficult.
With recent updates, the lifting-bits tools are now bumped to CMake minimum version 3.14, which is a reasonable minimum version in my mind, but it is not available by default on Ubuntu 18.04...
Rebuilding LLVM when vcpkg misses a cache is very annoying, especially when compiling both Debug and Release builds.
We should have a guide on how to build LLVM outside of vcpkg (while mirroring what vcpkg is doing) and use that build with a tool by specifying LLVM_DIR
.
Something like the following should be added somewhere in the README
If you want to cut down on unneeded LLVM components or specify new ones that aren't present by default, you can specify all of the features you want by doing something like the following:
./build_dependencies.sh --export-dir llvm-12-custom --release llvm-12[core,clang,tools,enable-rtti,enable-z3,<more_features>]
Which will build llvm according to your features and then install it into $(pwd)/llvm-12-custom directory
Should also talk about where to find that feature list and what is installed by default (in the CONTROL
file)
Should also link to the upstream vcpkg README for installation
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.