tencent / yadcc Goto Github PK
View Code? Open in Web Editor NEWYet Another Distributed C++ Compiler. yadcc是一套腾讯广告自研的分布式编译系统,用于支撑腾讯广告的日常开发及流水线。相对于已有的同类解决方案,我们针对实际的工业生产环境做了性能、可靠性、易用性等方面优化。
License: Other
Yet Another Distributed C++ Compiler. yadcc是一套腾讯广告自研的分布式编译系统,用于支撑腾讯广告的日常开发及流水线。相对于已有的同类解决方案,我们针对实际的工业生产环境做了性能、可靠性、易用性等方面优化。
License: Other
after install gcc by apt install in ubuntu 20.04
/usr/bin/gcc is a symlink file, the realpath of gcc is /usr/bin/x86_64-linux-gnu-gcc-9
the daemon can not find any compiler, because the bellow code skip symlink
if (S_ISLNK(buf.st_mode)) { // Symbolic link, ignore it.
continue;
}
https://github.com/Tencent/yadcc/blob/master/yadcc/doc/client/cxx.md
YADCC_IGNORE_TIMESTAMP_MACROS这个参数该如何使用,在哪里配置使用呢?
1、error occurred under gcc8 when building python2.7. compiled successfully under gcc7
2、python: can't decompress data; zlib not available when building python2.7.
solution
3、ubuntu 18.04.5(g++ 8.5)、ubuntu20.04(g++ 9.3)
The same compilation error occurred when i tryed to build yadcc on my PC which installed with ubuntu 18.04.5(g++ 8.5
) or ubuntu20.04(g++ 9.3
), but pthread.a(/usr/lib/x86_64-linux-gnu/libpthread.a
) and pthread.so(/usr/lib/x86_64-linux-gnu/libpthread.so
) exist. Strangely enough, it can be compiled successfully in the company environment which installed with ubuntu 18.04.
[3/75] LINK SHARED build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.so
FAILED: build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.so
g++ -shared -o build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.so -Lthirdparty/jdk/lib -Xlinker --no-undefined build64_release/flare/rpc/protocol/http/builtin/rpc_form_handler.objs/rpc_form_handler.cc.o build64_release/flare/rpc/protocol/http/builtin/libstatic_resources.so build64_release/flare/rpc/protocol/http/libhttp.so build64_release/flare/rpc/protocol/http/libbinlog_proto.so build64_release/flare/rpc/protocol/http/libbuffer_io.so build64_release/flare/rpc/protocol/http/libhttp_handler.so build64_release/flare/rpc/protocol/http/libmessage.so build64_release/flare/base/libcompression.so build64_release/flare/base/compression/libgzip.so build64_release/flare/base/compression/liblz4.so build64_release/thirdparty/lz4/lib64/liblz4.so build64_release/flare/base/compression/libsnappy.so build64_release/thirdparty/snappy/lib64/libsnappy.so build64_release/flare/base/compression/libzstd.so build64_release/thirdparty/zstd/lib64/libzstd.so build64_release/flare/base/buffer/libcompression_output_stream.so build64_release/flare/base/compression/libutil.so build64_release/flare/base/compression/libcompression.so build64_release/flare/net/http/libpacket_desc.so build64_release/flare/net/http/libhttp_request.so build64_release/flare/net/http/libhttp_response.so build64_release/flare/net/http/libhttp_message.so build64_release/flare/net/http/libhttp_headers.so build64_release/flare/net/http/libtypes.so build64_release/flare/rpc/internal/libsession_context.so build64_release/flare/rpc/binlog/liblog_reader.so build64_release/flare/rpc/binlog/libdry_runner.so build64_release/flare/base/libstatus.so build64_release/flare/fiber/liblogging.so build64_release/flare/fiber/libtimer.so build64_release/flare/fiber/libwork_queue.so build64_release/flare/rpc/binlog/liblog_writer.so build64_release/flare/rpc/binlog/libdumper.so build64_release/flare/rpc/binlog/libpacket_desc.so build64_release/flare/base/buffer/libzero_copy_stream.so build64_release/flare/rpc/tracing/libtracing_ops.so build64_release/flare/rpc/tracing/libtracing_ops_provider.so build64_release/thirdparty/opentracing-cpp/lib/libopentracing.so build64_release/flare/rpc/protocol/libstream_protocol.so build64_release/flare/base/libbuffer.so build64_release/flare/base/buffer/libbuiltin_buffer_block.so build64_release/flare/init/libon_init.so build64_release/flare/rpc/protocol/libcontroller.so build64_release/flare/rpc/protocol/libmessage.so build64_release/flare/base/experimental/libuuid.so build64_release/flare/base/net/libendpoint.so build64_release/flare/fiber/libfiber_basic.so build64_release/flare/base/internal/libindex_alloc.so build64_release/flare/fiber/detail/libscheduling_parameters.so build64_release/flare/fiber/detail/libfiber_impl.so build64_release/flare/fiber/detail/libcontext.so build64_release/flare/fiber/detail/librun_queue.so build64_release/flare/fiber/detail/libstack_allocator.so build64_release/flare/base/libmonitoring.so build64_release/flare/base/internal/libdpc.so build64_release/flare/base/monitoring/libinit.so build64_release/flare/base/monitoring/libdispatcher.so build64_release/flare/base/monitoring/libmonitoring_system.so build64_release/thirdparty/yaml-cpp/lib/libyaml-cpp.a build64_release/flare/base/libobject_pool.so build64_release/flare/base/object_pool/libdisabled.so build64_release/flare/base/object_pool/libglobal.so build64_release/flare/base/object_pool/libmemory_node_shared.so build64_release/flare/base/libexposed_var.so build64_release/flare/base/libtsc.so build64_release/flare/base/write_mostly/libmetrics.so build64_release/thirdparty/jsoncpp/lib/libjsoncpp.so build64_release/flare/base/internal/libtime_keeper.so build64_release/flare/base/internal/libbackground_task_host.so build64_release/flare/base/internal/libthread_pool.so build64_release/flare/base/internal/libcpu.so build64_release/flare/base/libstring.so build64_release/flare/base/object_pool/libthread_local.so build64_release/flare/base/libdemangle.so build64_release/flare/base/liberased_ptr.so build64_release/flare/base/thread/libattribute.so build64_release/flare/base/thread/liblatch.so build64_release/flare/base/thread/libout_of_duty_callback.so build64_release/flare/base/liblogging.so build64_release/flare/base/libchrono.so build64_release/flare/base/internal/libmemory_barrier.so build64_release/flare/base/internal/liblogging.so build64_release/thirdparty/fmt/lib64/libfmt.so build64_release/flare/base/thread/libspinlock.so build64_release/thirdparty/protobuf-3.4.1/libprotobuf.so build64_release/thirdparty/protobuf-3.4.1/libprotobuf_lite.so build64_release/thirdparty/glog/lib/libglog.so build64_release/thirdparty/gflags/lib/libgflags.so build64_release/thirdparty/zlib/lib/libz.so build64_release/thirdparty/ctemplate/lib/libctemplate.so -ldl -lrt -lz -lpthread -lm
/usr/bin/ld: build64_release/flare/rpc/protocol/http/builtin/rpc_form_handler.objs/rpc_form_handler.cc.o: in function `flare::object_pool::detail::memory_node_shared::GlobalPoolDescriptor* flare::object_pool::detail::memory_node_shared::GetGlobalPoolDescriptor<flare::PolymorphicBuffer>()':
/usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:700: undefined reference to `pthread_once'
collect2: error: ld returned 1 exit status
yadcc能不能用于安卓系统AOSP构建加速
build linux kernel, get some error
Failed on the cloud with [-126]. Failing back to local machine.
按照安装步骤执行
git clone https://github.com/Tencent/yadcc
cd yadcc
git submodule init
git submodule update .
编译时:./blade build yadcc/...
报错:
checking whether to build with libidn2... no
checking for pkg-config... no
configure: error: --with-nghttp2 was specified but could not find libnghttp2 pkg-config file.
[284/2107] CONFIGURE //thirdparty/glog:glog_configure
ninja: build stopped: subcommand failed.
Blade(error): Build failure.
Blade(info): Cost time 1m57.3s
Blade(error): Failure
As official doc describes, PCH can be used in preprocess, and insert "#pragma GCC pch_preprocess "filename"" into the output. Furthermore, the path can be absolute or relative to the working dir. That's to say, it is possible to use PCH to achieve distribute build, which could improve the compilation efficiency. And i have skipped your code and found during the preparation, yadcc skips the PCH and just use -E, -fno-working-directory and -fdirectives-only to produce preprocess output. Now, i have two questions and wish your reply.
1、As i knew, -E option just preprocess the included headers directly without PCH, and should takes more time not only during the preprocess but also during the compilation. So, have u attemp to use PCH to achieve distributely build in yadcc, and if so, why not use it?
2、As gcc doc describes, -fworking-directory is default setting. However, yadcc turns it off and user -fno-working-directory, and some debug information like file path will not be inserted into the preprocess output. As i think, the purpose is to avoid absoulte path in the output which may cause missing cache in spite of the same source file. I have tried to handle the preprocess output of msvc compiler, and found without absolute path, cache will be hit if the source file is same in different computer. It helps much to save compilation time, but i can use the results to start executable files but can not use the results obj files or exp files or pdb files to debug if turning off -fworking-directory . Thus, i wonder that have yadcc found similar problems after turning path off.
Wish your reply, thank you very much.
My machine: 16u32g
Is there a simple relationship?
flare_rpc_client_max_connections_per_server,flare_concurrency_hint,flare_rpc_server_stream_concurrency,flare_rpc_client_stream_concurrency, etc
Is it possible to provide a configuration guide document based on the configuration of personal computers?
能否给daemon加个功能,当scheduler重启后自动重链
FAILED: lib/Support/BLAKE3/CMakeFiles/LLVMSupportBlake3.dir/blake3_sse2_x86-64_unix.S.o
/opt/buildtools/yadcc/client/cxx/yadcc-cxx -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/workspace/llvm-project/tmp/lib/Support/BLAKE3 -I/home/workspace/llvm-project/llvm/lib/Support/BLAKE3 -I/home/workspace/llvm-project/tmp/include -I/home/workspace/llvm-project/llvm/include -fPIC -o lib/Support/BLAKE3/CMakeFiles/LLVMSupportBlake3.dir/blake3_sse2_x86-64_unix.S.o -c /home/workspace/llvm-project/llvm/lib/Support/BLAKE3/blake3_sse2_x86-64_unix.S
buld llvm-project-11.0.0.tar.xz will pass
but build llvm main branch will fail
[2023-03-22 21:19:47.961755] [FATAL] [yadcc/client/common/utility.cc:157] Failed to find executable [-DGTEST_HAS_RTTI=0] in path.
yadcc/build64_release/thirdparty/opentracing-cpp/opentracing-cpp-1.5.1/3rd_party/include/opentracing/catch2/catch.hpp:6631:45: error: size of array ‘altStackMem’ is not an integral constant-expression
6631 | char FatalConditionHandler::altStackMem[SIGSTKSZ] = {};
glibc changed SIGSTKSZ to be a syscall instead
of being constant. #catchorg/Catch2#2178
Is there any plan to add support for heterogeneous build environments?
Like icecream did, pack the buildenv to tarball then send to remote peer....
即 ./yadcc-scheduler --acceptable_tokens=some_fancy_token
这个命令的some_fancy_token应该如何生成?急,求指导 @0x804d8000 @tencent-adm @hzlzh @chen3feng @darkinsky
compile linux kernel use cross_compile like this
export ARCH=arm64
export CROSS_COMPILE=aarch64-none-linux-gnu-
make menuconfig ARCH=arm64 CROSS_COMPILE=$CROSS_COMPILE
make ARCH=arm64 CROSS_COMPILE=$CROSS_COMPILE -j$(nproc) zImage
yadcc can not support it
[45/1637] MAKE //thirdparty/nghttp2:nghttp2_build
......
libtool: install: /usr/bin/install -c .libs/libnghttp2.so.14.20.0 /home/name/workspace/yadcc/build64_release/thirdparty/nghttp2/lib/libnghttp2.so.14.20.0
/usr/bin/install: cannot stat '.libs/libnghttp2.so.14.20.0': No such file or directory
ll build64_release/thirdparty/nghttp2/nghttp2-1.41.0/lib/.libs
-rw-rw-r-- 1 wuminghui03 wuminghui03 819848 Oct 13 17:46 libnghttp2.a
lrwxrwxrwx 1 wuminghui03 wuminghui03 16 Oct 13 17:46 libnghttp2.la -> ../libnghttp2.la
-rw-rw-r-- 1 wuminghui03 wuminghui03 1023 Oct 13 17:46 libnghttp2.lai
lrwxrwxrwx 1 wuminghui03 wuminghui03 21 Oct 13 17:46 libnghttp2.so -> libnghttp2.so.14.20.0
lrwxrwxrwx 1 wuminghui03 wuminghui03 21 Oct 13 17:46 libnghttp2.so.14 -> libnghttp2.so.14.20.0
-rw-rw-r-- 1 wuminghui03 wuminghui03 27264 Oct 13 17:46 nghttp2_buf.o
安装报错 .libs/libnghttp2.so.14.20.0 不存在,查看目录 build64_release/thirdparty/nghttp2/nghttp2-1.41.0/lib/.libs
, 有指向 libnghttp2.so.14.20.0
但是这个库是不存在的,这个库应该是新构建的才对,但是并未出现这个动态库链接报错。
已经按照 REAMD 更新了submodule, 并且在 build/external/flare 指向过 git lfs install/pull
, 是有步骤不对少拉了文件吗
机器 CentOS Linux release 7.4.1708 (Core)。
Code in master
has long been superseded internally, using code in next
brings us the following benefits:
--coverage
(i.e., gcov);Yet, IIRC, it does regress in performance when compiling C++. This is mostly due to switch of protocol between client and daemon, from a dirty-and-quick one to a JSON based one. JsonCpp costs us quite some cpu cycles here.
That regression shouldn't hurt much, it's ~2%, and should be unnoticeable most of time.
./blade build yadcc/...
./blade: line 7: /root/yadcc/thirdparty/blade/blade: No such file or directory
查了一下,这个路径里面写的blade/blade并不存在,去掉这个blade也不行
W1022 14:18:14.378872 14776 distributed_cache_reader.cc:152] Failed to load compilation cache bloom filter from cache server: [1003] , last_bf_update: 355156, last_bf_full_update: 355158
last_bf_update < last_bf_full_update, but cache server requires last_bf_update >= last_bf_full_update
No logic error is found when working through distributed_cache_reader.cpp code.
机器1:client(user,且不不接受任务):ubuntu1804,gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
机器2:servant(专用编译机):ubuntu2004,gcc-7 (Ubuntu 7.5.0-6ubuntu2) 7.5.0。
servant上支持的gcc-7的哈希值与user需要的不一致,对于这种情况有什么建议吗?ubuntu2004能编译出来和ubuntu1804哈希值一样的gcc吗?你们对于这样的场景,你们是怎么做的?
{
flare::NoncontiguousBuffer standard_output;
......
auto result = flare::FlattenSlow(output->standard_output);
auto deps_list = flare::Split(result, '\n');
}
a 400 MB file is cached in standard_output.
After I use it, I find that the memory is still occupied and cannot be released.
当import module后,yadcc能否工作。
When I do secondary development on yadcc daemon, the newly developed module causes segment errors. I feel confused as to what causes the problem.
the following cases:
The newly developed module:
Flare::Fiber used
after installing git-lfs,
./blade build yadcc/...
the "Blade(error): This repository need GIT LFS, see README.md" still occerd
Is any guide or experience to compiling the AOSP code?
Android AOSP has several compiling systems per building target and android version.
Most community contributors rarely know about blade, when we want to add address sanitizer option for this project, we get confused.
在12c的机器上测试 make -j60 ,限制本地轻任务并发为12,并在 RewriteFile
前面加日志作为预处理时间。发现正常情况下预处理在1秒左右的文件,预处理变成了12秒。
尝试从 RewriteFile
往下找,最后在似乎是由客户端调用的syscall
。本地守护进程,设置了YADCC_LOG_LEVEL=0,也没打印预处理的日志。所以预处理是由客户端自己完成的?如果不是的话,能否告知在哪里加日志,可以获取预处理任务等待的时间?
client/common/command.cc:] Check failed: [WIFEXITED(status)]. Child process exited unexpectedly with status [13].
错误日志如下 (Ubuntu22.10 + gcc 12.2 + cgroup-v2-workaround)
execution_engine.cc:82] Check failed: parts.size() >= 3 (2 vs. 3) Unexpected cgroup setting: 0::/user.slice/user-1000.slice/[email protected]/app.slice/app-org.gnome.Terminal.slice/vte-spawn-b6c5fa9c-c6b5-4316-b3d5-97fa888e8ef4.scope *** Check failure stack trace: *** @ 0x55fbd5c17233 google::LogMessage::Fail() @ 0x55fbd5c1997e google::LogMessage::SendToLog() @ 0x55fbd5c16dde google::LogMessage::Flush() @ 0x55fbd5c19f0f google::LogMessageFatal::~LogMessageFatal() @ 0x55fbd54b1d71 _ZZZN5yadcc6daemon5cloud12_GLOBAL__N_115IsCGroupPresentEvENKUlOT_OT0_E0_clImiEEDaS4_S6_ENKUlvE_clEv @ 0x55fbd54b273c _ZN5yadcc6daemon5cloud15ExecutionEngineC2Ev.cold @ 0x55fbd56fd8fa yadcc::daemon::cloud::ExecutionEngine::Instance() @ 0x55fbd56e358b yadcc::daemon::cloud::DaemonServiceImpl::Heartbeat() @ 0x55fbd5b4b791 _ZZN5flare8FunctionIFvvEE15ErasedCopySmallIZZNS_5fiber8SetTimerENSt6chrono10time_pointINS5_3_V212steady_clockENS5_8durationIlSt5ratioILl1ELl1000000000EEEEEESC_ONS0_IFvmEEEENUlT_E_clImEEDaSH_EUlvE_EEPKNS2_7TypeOpsEPvOSH_ENUlSO_E_4_FUNESO_ @ 0x55fbd5b55cda flare::fiber::detail::FiberProc() Aborted (core dumped)
能否提供一个docker镜像
为了方便回溯,建议发布一个 v1.0 之类
我按技术文档配置,调度机和客户机都各自启动了scheduler和daemon,调度机检测到新的servant。
I1110 14:19:54.431716 1780984 init.cc:114] Flare started.
I1110 14:19:54.432881 1780984 runtime.cc:425] Using fiber scheduling profile [neutral].
I1110 14:19:54.432904 1780984 runtime.cc:220] Starting 8 worker threads per group, for a total of 1 groups. The system is treated as UMA.
I1110 14:19:54.436939 1780984 init.cc:122] Flare runtime initialized.
I1110 14:19:54.702870 1780997 task_dispatcher.cc:205] [192.168.33.5:46518] Discovered new servant at [192.168.33.5:8335]
但是运行YADCC_LOG_LEVEL=0 CXX='/home/ubuntu/.yadcc/symlinks/g++' LD='/home/ubuntu/.yadcc/symlinks/g++' CC='/home/ubuntu/.yadcc/symlinks/gcc' make -j8
提示
[2022-11-10 14:20:33.068854] [TRACE] [yadcc/client/yadcc.cc:261] Started
[2022-11-10 14:20:33.068886] [DEBUG] [yadcc/client/utility.cc:92] Looking up for [gcc] in [/home/ubuntu/.yadcc/symlinks].
[2022-11-10 14:20:33.068912] [DEBUG] [yadcc/client/utility.cc:92] Looking up for [gcc] in [/usr/local/sbin].
[2022-11-10 14:20:33.068919] [DEBUG] [yadcc/client/utility.cc:92] Looking up for [gcc] in [/usr/local/bin].
[2022-11-10 14:20:33.068926] [DEBUG] [yadcc/client/utility.cc:92] Looking up for [gcc] in [/usr/sbin].
[2022-11-10 14:20:33.068932] [DEBUG] [yadcc/client/utility.cc:92] Looking up for [gcc] in [/usr/bin].
[2022-11-10 14:20:33.068942] [TRACE] [yadcc/client/utility.cc:99] Found [gcc] at [/usr/bin].
[2022-11-10 14:20:33.068947] [TRACE] [yadcc/client/yadcc.cc:133] Using compiler: /usr/bin/gcc
[2022-11-10 14:20:33.068980] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
[2022-11-10 14:20:34.064286] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
[2022-11-10 14:20:34.065163] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
[2022-11-10 14:20:34.068817] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
[2022-11-10 14:20:34.069060] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
[2022-11-10 14:20:35.064416] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
[2022-11-10 14:20:35.065259] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
[2022-11-10 14:20:35.068913] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
[2022-11-10 14:20:35.069152] [ERROR] [yadcc/client/task_quota.cc:61] Cannot contact delegate daemon. Daemon died?
请问可能是什么问题?
Great repo! And any develop roadmap on this ? I would a contributor on yadcc.
I have build yadcc suceessfully on a server but failed on another one. The error is at tho bottom.
on the suceess server, curl's configure is without idn2:
checking for libidn2 options with pkg-config... no
configure: IDN_LIBS: "-lidn2"
configure: IDN_LDFLAGS: ""
configure: IDN_CPPFLAGS: ""
configure: IDN_DIR: ""
checking if idn2_lookup_ul can be linked... no
checking idn2.h usability... no
checking idn2.h presence... no
checking for idn2.h... no
configure: WARNING: Cannot find libraries for IDN support: IDN disabled
on the failed server, there is a libidn2.so:
checking for libidn2 options with pkg-config... found
configure: pkg-config: IDN_LIBS: "-lidn2"
configure: pkg-config: IDN_LDFLAGS: ""
configure: pkg-config: IDN_CPPFLAGS: ""
configure: pkg-config: IDN_DIR: ""
checking if idn2_lookup_ul can be linked... yes
checking idn2.h usability... yes
checking idn2.h presence... yes
checking for idn2.h... yes
But if the env has libidn2.so, why yadcc do not link it?
I don't know blade very well, it seems to be an internal tool in tencent.
I have add -lidn2 in linkflags in BLADE_ROOT, but it do not work.
=========================================================
Blade: Entering directory /home/zhut/davidwang/yadcc' Blade(info): Loading config file "/home/zhut/davidwang/yadcc/thirdparty/blade/blade.conf" Blade(info): Loading config file "/home/zhut/davidwang/yadcc/BLADE_ROOT" Blade(info): Loading BUILD files... Blade(info): Loading done. Blade(info): Analyzing dependency graph... Blade(info): Analyzing done. Blade(info): Generating backend build code... Blade(info): Generating done. Blade(info): Building... Blade(info): Adjust build jobs number(-j N) to be 8 [1/3] LINK BINARY build64_release/yadcc/scheduler/yadcc-scheduler FAILED: build64_release/yadcc/scheduler/yadcc-scheduler g++ -o build64_release/yadcc/scheduler/yadcc-scheduler -Wl,--no-as-needed -lpthread -static-libgcc -static-libstdc++ -Wl,--whole-archive build64_release/yadcc/api/libscheduler_proto.a build64_release/yadcc/api/libenv_desc_proto.a build64_release/flare/init/libopenssl_initializer.a build64_release/flare/rpc/protocol/http/builtin/libexposed_vars_handler.a build64_release/flare/rpc/protocol/http/builtin/libgflags_handler.a build64_release/flare/rpc/protocol/http/builtin/libmisc_handler.a build64_release/flare/rpc/protocol/http/builtin/liboptions_handler.a build64_release/flare/base/liboption.a build64_release/flare/rpc/protocol/http/builtin/libprof_cpu_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_reflect_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_statistics_handler.a build64_release/flare/rpc/protocol/http/builtin/libstatic_resource_http_handler.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_protocol.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libpoppy_protocol.a build64_release/flare/rpc/protocol/protobuf/libpoppy_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libproto_over_http_protocol.a build64_release/flare/rpc/protocol/protobuf/libqzone_protocol.a build64_release/flare/rpc/protocol/protobuf/libnslb_registration.a build64_release/flare/rpc/load_balancer/libround_robin.a build64_release/flare/rpc/name_resolver/liblist.a build64_release/flare/rpc/message_dispatcher/libcomposited.a build64_release/flare/rpc/protocol/protobuf/libbinlog_proto.a build64_release/flare/rpc/librpc_options_proto.a build64_release/flare/rpc/protocol/protobuf/libstd_protocol.a build64_release/flare/rpc/binlog/text_only/libdumper.a build64_release/flare/rpc/binlog/text_only/libbinlog_proto.a build64_release/flare/rpc/binlog/util/libproto_binlog_proto.a build64_release/flare/rpc/protocol/protobuf/librpc_meta_proto.a build64_release/flare/rpc/protocol/http/libhttp.a build64_release/flare/rpc/protocol/http/libbinlog_proto.a build64_release/flare/base/compression/libgzip.a build64_release/flare/base/compression/liblz4.a build64_release/flare/base/compression/libsnappy.a build64_release/flare/base/compression/libzstd.a -Wl,--no-whole-archive @build64_release/yadcc/scheduler/yadcc-scheduler.rsp -lresolv -lm -ldl -lpthread -lrt /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function
conn_free':
url.c:(.text+0x2a6): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2c2): undefined reference to
idn2_free'
/usr/bin/ld: url.c:(.text+0x2de): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2fa): undefined reference to
idn2_free'
/usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_idnconvert_hostname': url.c:(.text+0x135b): undefined reference to
idn2_check_version'
/usr/bin/ld: url.c:(.text+0x137b): undefined reference to idn2_lookup_ul' /usr/bin/ld: url.c:(.text+0x139b): undefined reference to
idn2_strerror'
/usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_free_idnconverted_hostname': url.c:(.text+0x13d1): undefined reference to
idn2_free'
/usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_connect': url.c:(.text+0x4769): undefined reference to
idn2_free'
/usr/bin/ld: url.c:(.text+0x4785): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x498e): undefined reference to
idn2_free'
/usr/bin/ld: url.c:(.text+0x49ac): undefined reference to idn2_free' collect2: error: ld returned 1 exit status [2/3] LINK BINARY build64_release/yadcc/cache/yadcc-cache FAILED: build64_release/yadcc/cache/yadcc-cache g++ -o build64_release/yadcc/cache/yadcc-cache -Wl,--no-as-needed -lpthread -static-libgcc -static-libstdc++ -Wl,--whole-archive build64_release/yadcc/api/libcache_proto.a build64_release/yadcc/cache/libdisk_cache_engine.a build64_release/yadcc/cache/libnull_cache_engine.a build64_release/flare/init/libopenssl_initializer.a build64_release/flare/rpc/protocol/http/builtin/libexposed_vars_handler.a build64_release/flare/rpc/protocol/http/builtin/libgflags_handler.a build64_release/flare/rpc/protocol/http/builtin/libmisc_handler.a build64_release/flare/rpc/protocol/http/builtin/liboptions_handler.a build64_release/flare/base/liboption.a build64_release/flare/rpc/protocol/http/builtin/libprof_cpu_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_reflect_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_statistics_handler.a build64_release/flare/rpc/protocol/http/builtin/libstatic_resource_http_handler.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_protocol.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libpoppy_protocol.a build64_release/flare/rpc/protocol/protobuf/libpoppy_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libproto_over_http_protocol.a build64_release/flare/rpc/protocol/protobuf/libqzone_protocol.a build64_release/flare/rpc/protocol/protobuf/libnslb_registration.a build64_release/flare/rpc/load_balancer/libround_robin.a build64_release/flare/rpc/name_resolver/liblist.a build64_release/flare/rpc/message_dispatcher/libcomposited.a build64_release/flare/rpc/protocol/protobuf/libbinlog_proto.a build64_release/flare/rpc/librpc_options_proto.a build64_release/flare/rpc/protocol/protobuf/libstd_protocol.a build64_release/flare/rpc/binlog/text_only/libdumper.a build64_release/flare/rpc/binlog/text_only/libbinlog_proto.a build64_release/flare/rpc/binlog/util/libproto_binlog_proto.a build64_release/flare/rpc/protocol/protobuf/librpc_meta_proto.a build64_release/flare/rpc/protocol/http/libhttp.a build64_release/flare/rpc/protocol/http/libbinlog_proto.a build64_release/flare/base/compression/libgzip.a build64_release/flare/base/compression/liblz4.a build64_release/flare/base/compression/libsnappy.a build64_release/flare/base/compression/libzstd.a -Wl,--no-whole-archive @build64_release/yadcc/cache/yadcc-cache.rsp -lresolv -lm -ldl -lpthread -lrt /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function
conn_free':
url.c:(.text+0x2a6): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2c2): undefined reference to
idn2_free'
/usr/bin/ld: url.c:(.text+0x2de): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2fa): undefined reference to
idn2_free'
/usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_idnconvert_hostname': url.c:(.text+0x135b): undefined reference to
idn2_check_version'
/usr/bin/ld: url.c:(.text+0x137b): undefined reference to idn2_lookup_ul' /usr/bin/ld: url.c:(.text+0x139b): undefined reference to
idn2_strerror'
/usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_free_idnconverted_hostname': url.c:(.text+0x13d1): undefined reference to
idn2_free'
/usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_connect': url.c:(.text+0x4769): undefined reference to
idn2_free'
/usr/bin/ld: url.c:(.text+0x4785): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x498e): undefined reference to
idn2_free'
/usr/bin/ld: url.c:(.text+0x49ac): undefined reference to idn2_free' collect2: error: ld returned 1 exit status [3/3] LINK BINARY build64_release/yadcc/daemon/yadcc-daemon FAILED: build64_release/yadcc/daemon/yadcc-daemon g++ -o build64_release/yadcc/daemon/yadcc-daemon -Wl,--no-as-needed -lpthread -static-libgcc -static-libstdc++ -Wl,--whole-archive build64_release/flare/init/libopenssl_initializer.a build64_release/flare/rpc/builtin/libtcmalloc_profiler_http_handler.a build64_release/yadcc/api/libcache_proto.a build64_release/yadcc/api/libdaemon_proto.a build64_release/yadcc/api/libscheduler_proto.a build64_release/yadcc/daemon/local/libmessages_proto.a build64_release/flare/rpc/protocol/http/builtin/libexposed_vars_handler.a build64_release/flare/rpc/protocol/http/builtin/libgflags_handler.a build64_release/flare/rpc/protocol/http/builtin/libmisc_handler.a build64_release/flare/rpc/protocol/http/builtin/liboptions_handler.a build64_release/flare/base/liboption.a build64_release/flare/rpc/protocol/http/builtin/libprof_cpu_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_form_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_reflect_handler.a build64_release/flare/rpc/protocol/http/builtin/librpc_statistics_handler.a build64_release/flare/rpc/protocol/http/builtin/libstatic_resource_http_handler.a build64_release/flare/rpc/protocol/http/libhttp.a build64_release/flare/rpc/protocol/http/libbinlog_proto.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_protocol.a build64_release/flare/rpc/protocol/protobuf/libbaidu_std_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libpoppy_protocol.a build64_release/flare/rpc/protocol/protobuf/libpoppy_rpc_meta_proto.a build64_release/flare/rpc/protocol/protobuf/libproto_over_http_protocol.a build64_release/flare/rpc/protocol/protobuf/libqzone_protocol.a build64_release/flare/rpc/protocol/protobuf/libnslb_registration.a build64_release/flare/rpc/load_balancer/libround_robin.a build64_release/flare/rpc/name_resolver/liblist.a build64_release/flare/rpc/message_dispatcher/libcomposited.a build64_release/flare/rpc/protocol/protobuf/libbinlog_proto.a build64_release/flare/rpc/librpc_options_proto.a build64_release/flare/rpc/protocol/protobuf/libstd_protocol.a build64_release/flare/rpc/binlog/text_only/libdumper.a build64_release/flare/rpc/binlog/text_only/libbinlog_proto.a build64_release/flare/rpc/binlog/util/libproto_binlog_proto.a build64_release/flare/rpc/protocol/protobuf/librpc_meta_proto.a build64_release/flare/base/compression/libgzip.a build64_release/flare/base/compression/liblz4.a build64_release/flare/base/compression/libsnappy.a build64_release/flare/base/compression/libzstd.a build64_release/yadcc/daemon/libcache_meta_proto.a build64_release/yadcc/api/libenv_desc_proto.a -Wl,--no-whole-archive @build64_release/yadcc/daemon/yadcc-daemon.rsp -lresolv -lm -ldl -lpthread -lrt /usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function
conn_free':
url.c:(.text+0x2a6): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2c2): undefined reference to
idn2_free'
/usr/bin/ld: url.c:(.text+0x2de): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x2fa): undefined reference to
idn2_free'
/usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_idnconvert_hostname': url.c:(.text+0x135b): undefined reference to
idn2_check_version'
/usr/bin/ld: url.c:(.text+0x137b): undefined reference to idn2_lookup_ul' /usr/bin/ld: url.c:(.text+0x139b): undefined reference to
idn2_strerror'
/usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_free_idnconverted_hostname': url.c:(.text+0x13d1): undefined reference to
idn2_free'
/usr/bin/ld: build64_release/thirdparty/curl/lib/libcurl.a(libcurl_la-url.o): in function Curl_connect': url.c:(.text+0x4769): undefined reference to
idn2_free'
/usr/bin/ld: url.c:(.text+0x4785): undefined reference to idn2_free' /usr/bin/ld: url.c:(.text+0x498e): undefined reference to
idn2_free'
/usr/bin/ld: url.c:(.text+0x49ac): undefined reference to `idn2_free'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
Blade(error): Build failure.
Blade(info): Cost time 4.43s
Blade(error): Failure
这是编译日志:
[root@st1-gitci-1 yadcc]# ./blade build yadcc/...
Blade: Entering directory `/root/source/yadcc'
Blade(info): Loading config file "/root/source/yadcc/thirdparty/blade/blade.conf"
Blade(info): Loading config file "/root/source/yadcc/BLADE_ROOT"
Blade(info): Loading BUILD files...
Blade(info): Loading done.
Blade(info): Analyzing dependency graph...
Blade(info): Analyzing done.
Blade(info): Generating backend build code...
Blade(info): Generating done.
Blade(info): Building...
Blade(info): Adjust build jobs number(-j N) to be 16
[1/1781] LINK BINARY build64_release/yadcc/client/cxx/yadcc-cxx
FAILED: build64_release/yadcc/client/cxx/yadcc-cxx
g++ -o build64_release/yadcc/client/cxx/yadcc-cxx -Wl,--no-as-needed -lpthread -static-libgcc -static-libstdc++ @build64_release/yadcc/client/cxx/yadcc-cxx.rsp -ldl -lpthread -static
/usr/bin/ld: cannot find -lpthread
/usr/bin/ld: cannot find -ldl
/usr/bin/ld: cannot find -lpthread
/usr/bin/ld: cannot find -lm
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
[2/1781] MAKE //thirdparty/ctemplate:ctemplate_build
ninja: build stopped: subcommand failed.
Blade(error): Build failure.
Blade(info): Cost time 23.6s
Blade(error): Failure
E0803 21:09:16.471520 8840 task_dispatcher.cc:314] [10.243.2.241:48228] Unrecognized compilation environment [7446218a5a1cd826cd663c99919545490da3c56af4ffd1d53d1554cd0768b0ee] is requested by [10.243.2.241].
文档中提到,“yadcc会按照命令行对源代码进行预处理,得到一个自包含的的预处理结果”
,如何理解“自包含”和yadcc是如何做到自包含的呢? @chen3feng @0x804d8000
$./blade build yadcc/...
thirdparty/protobuf/bin/x86_64/protoc-3.4.1: line 1: version: command not found
thirdparty/protobuf/bin/x86_64/protoc-3.4.1: line 2: oid: command not found
我运行的命令行是
./yadcc-daemon --scheduler_uri=flare://192.168.16.11:8336 token=lcy --poor_machine_threshold_processors=1
最终报错
F0316 15:04:56.197872 21353 execution_engine.cc:82] Check failed: parts.size() >= 3 (2 vs. 3) Unexpected cgroup setting: 0::/user.slice/user-0.slice/session-202.scope *** Check failure stack trace: *** @ 0x5632407132d3 (/root/yadcc/build64_release/yadcc/daemon/yadcc-daemon+0x80f2d2) @ 0x563240715826 (/root/yadcc/build64_release/yadcc/daemon/yadcc-daemon+0x811825) @ 0x563240712e8e (/root/yadcc/build64_release/yadcc/daemon/yadcc-daemon+0x80ee8d) @ 0x563240715dcf (/root/yadcc/build64_release/yadcc/daemon/yadcc-daemon+0x811dce) @ 0x56323ffc0639 (/root/yadcc/build64_release/yadcc/daemon/yadcc-daemon+0xbc638) @ 0x56323ffc0ea1 (/root/yadcc/build64_release/yadcc/daemon/yadcc-daemon+0xbcea0) @ 0x563240203572 (/root/yadcc/build64_release/yadcc/daemon/yadcc-daemon+0x2ff571) @ 0x5632401ea47f (/root/yadcc/build64_release/yadcc/daemon/yadcc-daemon+0x2e647e) @ 0x56324064deb3 (/root/yadcc/build64_release/yadcc/daemon/yadcc-daemon+0x749eb2) @ 0x5632406586aa (/root/yadcc/build64_release/yadcc/daemon/yadcc-daemon+0x7546a9) Aborted (core dumped)
yadcc
尝试使用一个daemon
未识别的编译器,守护进程等待五分钟放弃任务,并打印日志:
Task 3 cannot be started in time. Aborted.
,但是调度器仍不断打印 Unrecognized compilation environment
。
经过测试发现,将gcc编译之后的文件缓存至yadcc-cache没有问题,但是无法缓存g++编译之后的文件,并且使用YADCC_WARN_ON_NONCACHEABLE=1,这个参数也没有任何效果。
我架了4个编译节点, A:50核,B:50核, C:20核, D:20核。
当我make -j80的时候, A:50核占满,B:30核,C,和D,偶尔会有占用一两核。 C和D,在这种情况下,大部分时候没有编译任务。
想问下,有没有方案,让调度更平均一点,当make -j80的时候,按比例分配到节点
比如: A:28, B:28, C:12 D:12
yadcc/yadcc/client/libfakeroot/BUILD
Line 28 in 3c0f34a
构建了一个 arm 版本 yadcc, 其中发现资源库打包比较怪,打包的不是期望的构建出来的新库,
由于对于 blade 不熟悉, 没看出 src 中怎么引用其他库构建结果
只能是构建第一遍后,覆盖掉,然后再构建
请问这里是否可以优化呢?
编译环境:Intel CPU,Ubuntu22.04,gcc-11.3
yadcc v1.x 版本会拉取这个版本的flare-7a69d23,编译出错
/root/yadcc/build64_release/thirdparty/opentracing-cpp/opentracing-cpp-1.5.1/3rd_party/include/opentracing/catch2/catch.hpp:6631:45: error: size of array 'altStackMem' is not an integral constant-expression 6631 | char FatalConditionHandler::altStackMem[SIGSTKSZ] = {};
后面还会报一个error: macro "__has_attribute" requires an identifier
的错误。
手动将flare更新到master最新版本可以修复:https://github.com/Tencent/flare/tree/365d1ac3670d21a62212251364b0b9f7e15ed07f
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.