fanquake / core-review Goto Github PK

View Code? Open in Web Editor NEW

106.0 20.0 33.0 2.29 MB

Scripts, tools & info for doing Bitcoin Core development and code review

Home Page: https://github.com/bitcoin/bitcoin

C++ 10.85% Shell 7.23% Dockerfile 54.13% AppleScript 3.20% Python 18.85% HTML 5.73%

bitcoin gitian code-review bitcoin-core clang lldb

core-review's Introduction

Core Review

Scripts, tools & info for doing Bitcoin Core development and code review. Based on using a macOS 10.14 system.

AppleScript - Scripts for UI & test automation
assumevalid Updates - How to review pull requests that update assumevalid.
big-wallet.py - Create a wallet with lots of transactions.
Binary Compare - Notes on doing binary comparisons of Bitcoin Core
Clang Build Analyser - Clangs -ftime-trace & ClangBuildAnalyzer
Clang Tools - Setting up & using Clang Tools
Compiler Bugs
Compiler Defaults
Compiler Explorer - Example Compiler Explorer use case.
CoreRPC - Swift RPC wrapper.
Coz - Profiling bitcoind with Coz
depends - notes on using the depends system
Determinism - Notes on determinism.
diffoscope - A tool for generating diffs.
Flame Graphs - Notes on producing Flame Graphs.
fortify.py - Calculate _chk % replacement when using FORTIFY_SOURCE
Fuzzing - Fuzzing using Clang & libFuzzer.
github-merge.py - Script for merging PRs from GitHub.
gnuplot - gnuplot usage for benchmarking and analysis.
Guix - Notes on setting up / using Guix.
Hardening - Hardening in Bitcoin Core.
IRC - Where to find Bitcoin Core related discussion on IRC.
Link Time Optimization - Notes on using LTO.
Links - Useful links that I would otherwise forget about.
LLDB - Notes on using LLDB.
Locale Dependence - Locale dependence examples.
Mesh - Running bitcoind with Mesh
NixOS - Building on NixOS
Operating Systems - OS's that Core should support.
Profile Guided Optimization - Notes on using PGO.
reprotest - Using reprotest.
rtld-audit - Dynamic Linker Auditing - linux dynamic linker auditing
Scripted Diff
Subtree Merges - How to verify subtree merges are done correctly.
.tbd stubs - Using .tbd stubs.
URIs - URIs for testing bitcoin: uri handling.
Vagrant - Box definitions for PR review/testing.
Valgrind - Installing Valgrind on macOS.
Windows - Windows 10 VM setup for native builds.
Windows Cross Compile - Building Windows binaries on macOS

TODO

Add some notes LD notes for why_live, dead_strip_dyibs, DYLD_ etc.
glibc function usage
CVE history

core-review's People

Contributors

Stargazers

Watchers

core-review's Issues

Dockerfile compile of bitcoin

Thanks for your great work on bitcoin. I'm trying to use the Dockerfile to generate a Mac client. I now have copied the SDK to the SDKs folder, however, when I start I see errors like this:

bash-5.1# HOSTS="x86_64-apple-darwin18" ./contrib/guix/guix-build
/bin/sh: gcc: not found
/bin/sh: gcc: not found
/bin/sh: g++: not found
/bin/sh: g++: not found
env: can't execute '/bitcoin/depends/x86_64-apple-darwin18/native/bin/clang': No such file or directory
env: can't execute '/bitcoin/depends/x86_64-apple-darwin18/native/bin/clang': No such file or directory
env: can't execute '/bitcoin/depends/x86_64-apple-darwin18/native/bin/clang++': No such file or directory
env: can't execute '/bitcoin/depends/x86_64-apple-darwin18/native/bin/clang++': No such file or directory
Found macOS SDK at '/bitcoin/depends/SDKs/Xcode-11.3.1-11C505-extracted-SDK-with-libcxx-headers', using...
Checking that we can connect to the guix-daemon...

Hint: If this hangs, you may want to try turning your guix-daemon off and on
      again.

/bin/sh: gcc: not found
/bin/sh: gcc: not found
/bin/sh: g++: not found
/bin/sh: g++: not found
/bin/sh: gcc: not found
/bin/sh: gcc: not found
/bin/sh: g++: not found
/bin/sh: g++: not found
make: Entering directory '/bitcoin/depends'
/bin/sh: gcc: not found
/bin/sh: gcc: not found
/bin/sh: g++: not found
/bin/sh: g++: not found
/bin/sh: gcc: not found
/bin/sh: gcc: not found
/bin/sh: g++: not found
/bin/sh: g++: not found
make[1]: Entering directory '/bitcoin/depends'
/bin/sh: gcc: not found
/bin/sh: gcc: not found
/bin/sh: g++: not found
/bin/sh: g++: not found
env: can't execute '/bitcoin/depends/x86_64-apple-darwin/native/bin/clang': No such file or directory
env: can't execute '/bitcoin/depends/x86_64-apple-darwin/native/bin/clang': No such file or directory
env: can't execute '/bitcoin/depends/x86_64-apple-darwin/native/bin/clang++': No such file or directory
env: can't execute '/bitcoin/depends/x86_64-apple-darwin/native/bin/clang++': No such file or directory
Fetching boost_1_71_0.tar.bz2 from https://dl.bintray.com/boostorg/release/1.71.0/source/

Is this normal?

It does seem to be compiling now though...

It ultimately finished. I guess that's good!

Writing XML data...
Generating UDIF metadata...
Master checksum: 6e4436be
Writing out UDIF resource file...
Cleaning up...
Done
rm /outdir-base/x86_64-apple-darwin18/bitcoin-e1e1e708fa0f-osx-unsigned.temp.iso

how to run bitcoin unittests + functional tests with sanitizers

Hi,
a small guide from ground up on how to run sanitizers(asan/ubsan/msan) on core would be awesome :)
@practicalswift ;)

Consider moving to a distroless image

https://github.com/GoogleContainerTools/distroless#why-should-i-use-distroless-images

Restricting what's in your runtime container to precisely what's necessary for your app is a best practice employed by Google and other tech giants that have used containers in production for many years. It improves the signal to noise of scanners (e.g. CVE) and reduces the burden of establishing provenance to just what you need.

Distroless images are very small. The smallest distroless image, gcr.io/distroless/static-debian11, is around 2 MiB. That's about 50% of the size of alpine (~5 MiB), and less than 2% of the size of debian (124 MiB).

core-review/guix/debian.Dockerfile

Line 1 in 741d402

FROM debian:bullseye-slim

These images are also signed https://github.com/GoogleContainerTools/distroless#how-do-i-verify-distroless-images

Check guix.carldong.io

Error from docker using the --buil-args instructions:

7.403 wget: can't connect to remote host (4.53.92.115): Connection refused

That's Carl's IP:

--> dig guix.carldong.io +short
4.53.92.115

Simpler docker build command seems to work OK

unable to download Mac xip file

When I go to the link provided (https://download.developer.apple.com/Developer_Tools/Xcode_11.3.1/Xcode_11.3.1.xip), I get the following. I am an Apple Developer, signed in, and everything should be up to date (expires July 2021).

Sorry, you cannot view this page.

The Apple ID you signed in with does not have permission to view this page.
If you’re currently a member of the Apple Developer Program, you or your Account Holder may need to update your account by agreeing to the latest license agreement in order to access this page. To view your current membership status and benefits, visit your account.

Maybe it is no longer available? I have Xcode 12.4 installed on my Mac and I am trying to setup your docker.

Update. I had to go to https://developer.apple.com/download/more/ and then scroll in Developer Tools to Xcode 11.3.1. It offered up the same link https://download.developer.apple.com/Developer_Tools/Xcode_11.3.1/Xcode_11.3.1.xip but this time gave me a prompt.

I recently switched Macs and probably accepted this prompt years ago. Works after that.

Symbols from libQMIParserDynamic.tbd in libconsensus stub

When generating a .tbd stub for libconsensus, the weakly defined c++ symbols come from libQMIParserDynamic.tbd. How does this work, why is it the case?

LLVM 8 non-determinism

bitcoin/bitcoin#20447

--privileged required to run guix docker container

Running any docker image with --privileged is far from ideal.

When the operator executes docker run --privileged, Docker will enable access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host.

I started looking at running the guix image without --privileged:

guix environment: error: build failed: cloning builder process: Operation not permitted

Adding --cap-add=SYS_ADMIN, which is also far from ideal, see "CAP_SYS_ADMIN: the new root", solved that, but lead to:

guix environment: error: build failed: while setting up the build environment: cannot set loopback interface flags: Operation not permitted

So I added --cap-add=NET_ADMIN. I'm now at:

guix environment: error: build failed: while setting up the build environment: cannot pivot old root directory onto '/gnu/store/380c843hbacsr4dvj9dpqb5dw11x60dd-ca-certificate-bundle.drv.chroot/real-root': Operation not permitted

Windows compile using Dockerfile ends with SEH error

In running "HOSTS="x86_64-w64-mingw32" ./contrib/guix/guix-build" in the Dockerfile, the compile terminated with below. I am running the alpine image in Docker on a Mac.

compiling ../windows/qwindowsinputcontext.cpp
{standard input}: Assembler messages:
{standard input}:8688: Warning: end of file not at end of a line; newline inserted
{standard input}: Error: open SEH entry at end of file (missing .seh_endproc)
x86_64-w64-mingw32-g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[5]: *** [Makefile.Release:5873: .obj/release/qwindowsmime.o] Error 1
make[5]: *** Waiting for unfinished jobs....
make[5]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms/direct2d'
make[4]: *** [Makefile:38: release] Error 2
make[4]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms/direct2d'
make[3]: *** [Makefile:95: sub-direct2d-make_first] Error 2
make[3]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms'
make[2]: *** [Makefile:48: sub-platforms-make_first] Error 2
make[2]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins'
make[1]: *** [Makefile:582: sub-plugins] Error 2
make[1]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src'
make: *** [funcs.mk:281: /bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/./.stamp_built] Error 2
make: Leaving directory '/bitcoin/depends'

Looks like the issue first arises earlier:

compiling ../windows/qwindowswindow.cpp
x86_64-w64-mingw32-g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[5]: *** [Makefile.Release:2757: .obj/release/qwindowswindow.o] Error 1
make[5]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms/windows'
make[4]: *** [Makefile:38: release] Error 2
make[4]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms/windows'
make[3]: *** [Makefile:70: sub-windows-make_first] Error 2
make[3]: *** Waiting for unfinished jobs....

If you can shed some light on what is going on, I would really appreciate it. In case it helps, here is where I'm at:

commit e1e1e708fa0fbc0c51460305da5d401ed8f218f3 (HEAD -> master, origin/master, origin/HEAD)
Merge: 298fd02e4 415fb2e1a
Author: Hennadii Stepanov <[email protected]>
Date:   Thu Apr 29 23:38:46 2021 +0300

I also tried rerunning, but it eventually also terminated:

compiling qwindowsscreen.cpp
make[5]: Entering directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms/direct2d'
compiling ../windows/qwindowsmime.cpp
x86_64-w64-mingw32-g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[5]: *** [Makefile.Release:3664: .obj/release/qwindowscontext.o] Error 1
make[5]: *** Waiting for unfinished jobs....
compiling ../windows/qwindowstheme.cpp
x86_64-w64-mingw32-g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[5]: *** [Makefile.Release:4043: .obj/release/qwindowsscreen.o] Error 1
compiling ../windows/qwindowsmenu.cpp
make[5]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms/windows'
make[4]: *** [Makefile:38: release] Error 2
make[4]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms/windows'
make[3]: *** [Makefile:70: sub-windows-make_first] Error 2
make[3]: *** Waiting for unfinished jobs....
generating .moc/release/moc_predefs.h
compiling ../windows/qwindowsservices.cpp
compiling ../windows/qwindowsnativeinterface.cpp
compiling ../windows/qwindowsopengltester.cpp
compiling ../windows/qwin10helpers.cpp
compiling ../windows/qwindowssystemtrayicon.cpp
compiling ../windows/qwindowsclipboard.cpp
compiling ../windows/qwindowsdrag.cpp
compiling ../windows/qwindowstabletsupport.cpp
compiling ../windows/uiautomation/qwindowsuiaaccessibility.cpp
compiling ../windows/uiautomation/qwindowsuiaprovidercache.cpp
x86_64-w64-mingw32-g++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[5]: *** [Makefile.Release:9812: .obj/release/qwindowstabletsupport.o] Error 1
make[5]: *** Waiting for unfinished jobs....
make[5]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms/direct2d'
make[4]: *** [Makefile:38: release] Error 2
make[4]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms/direct2d'
make[3]: *** [Makefile:95: sub-direct2d-make_first] Error 2
make[3]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins/platforms'
make[2]: *** [Makefile:48: sub-platforms-make_first] Error 2
make[2]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src/plugins'
make[1]: *** [Makefile:582: sub-plugins] Error 2
make[1]: Leaving directory '/bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/qtbase/src'
make: *** [funcs.mk:281: /bitcoin/depends/work/build/x86_64-w64-mingw32/qt/5.12.10-260e83bf8d1/./.stamp_built] Error 2
make: Leaving directory '/bitcoin/depends'