tencent / plato Goto Github PK
View Code? Open in Web Editor NEW腾讯高性能分布式图计算框架Plato
License: Other
腾讯高性能分布式图计算框架Plato
License: Other
Our data is larger than 4.2 billion.
Processing triggers for sgml-base (1.26+nmu4ubuntu1) ...
看源码的时候主要有两点不解:
1. shuffle里面由于多线程的关系,所以最后额外再来一次probe_once(),我理解这样是因为多线程一起接受,同一个源计算节点发送过来的数据在写buffer还没完成时另一个线程接收到了finish signal,这样要额外再test一下,但是在某些情况下,finish_signal 接收完成时多于1个线程还在写buffer,这样额外只调用一次probe_once()会不会还是有可能漏了一部分数据(因为vector长度默认为3,可能会出现)?
2. 我看了一下c++ reference,memcpy如果直接写一个类而非序列化的话应该是类得满足POD,即is_standard_layout 和 is_trival 都为true才行,代码中同时杂糅了两种不同的判断方法可能会有问题?
➜ plato git:(master) BAZEL_LINKOPTS=-static-libstdc++ CC=/usr/bin/mpicxx bazel build example/...
INFO: Analyzed 24 targets (39 packages loaded, 14033 targets configured).
INFO: Found 24 targets...
INFO: Deleting stale sandbox base /home/jsy/.cache/bazel/_bazel_jsy/336f117599031580e40b679ad0dfd7d6/sandbox
ERROR: /home/jsy/github/plato/example/BUILD:151:1: C++ compilation of rule '//example:bfs_simple' failed (Exit 1) mpicxx failed: error executing command
(cd /home/jsy/.cache/bazel/_bazel_jsy/336f117599031580e40b679ad0dfd7d6/sandbox/linux-sandbox/1/execroot/__main__ && \
exec env - \
PATH=/home/jsy/.sdkman/candidates/scala/current/bin:/home/jsy/.sdkman/candidates/maven/current/bin:/home/jsy/.sdkman/candidates/kotlin/current/bin:/home/jsy/.sdkman/candidates/groovy/current/bin:/home/jsy/bin:/usr/share/node-v10.16.0-linux-x64/bin:/usr/share/jdk1.8.0_211/bin:/usr/share/go//bin:/home/jsy/.cargo/bin:/home/jsy/bin:/home/jsy/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
PWD=/proc/self/cwd \
/usr/bin/mpicxx -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/example/_objs/bfs_simple/bfs_simple.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/example/_objs/bfs_simple/bfs_simple.pic.o' -fPIC -D__DUALMODE_DEBUG__ -iquote . -iquote bazel-out/k8-fastbuild/bin -iquote external/bazel_tools -iquote bazel-out/k8-fastbuild/bin/external/bazel_tools -Ibazel-out/k8-fastbuild/bin/3rd/glog/_virtual_includes/glog -Ibazel-out/k8-fastbuild/bin/3rd/gflags/_virtual_includes/gflags -Ibazel-out/k8-fastbuild/bin/3rd/yas/_virtual_includes/yas -Ibazel-out/k8-fastbuild/bin/3rd/hadoop2/_virtual_includes/hdfs-dynamic -Ibazel-out/k8-fastbuild/bin/3rd/jni/_virtual_includes/jni -isystem 3rd/boost/include -isystem bazel-out/k8-fastbuild/bin/3rd/boost/include -isystem plato/graph -isystem bazel-out/k8-fastbuild/bin/plato/graph -isystem plato/util -isystem bazel-out/k8-fastbuild/bin/plato/util -isystem plato/util/libcuckoo -isystem bazel-out/k8-fastbuild/bin/plato/util/libcuckoo -isystem plato/graph/state -isystem bazel-out/k8-fastbuild/bin/plato/graph/state -isystem plato/parallel -isystem bazel-out/k8-fastbuild/bin/plato/parallel -isystem plato/graph/partition -isystem bazel-out/k8-fastbuild/bin/plato/graph/partition -isystem 3rd/jni/include/linux -isystem bazel-out/k8-fastbuild/bin/3rd/jni/include/linux -isystem plato/graph/structure -isystem bazel-out/k8-fastbuild/bin/plato/graph/structure -isystem plato/algo/bfs -isystem bazel-out/k8-fastbuild/bin/plato/algo/bfs -isystem plato/engine -isystem bazel-out/k8-fastbuild/bin/plato/engine -g -O2 -Werror -g -fPIC -static-libstdc++ -static-libgcc -fopenmp -Wall '-std=c++11' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c example/bfs_simple.cc -o bazel-out/k8-fastbuild/bin/example/_objs/bfs_simple/bfs_simple.pic.o)
Execution platform: @bazel_tools//platforms:host_platform
Use --sandbox_debug to see verbose messages from the sandbox: mpicxx failed: error executing command
(cd /home/jsy/.cache/bazel/_bazel_jsy/336f117599031580e40b679ad0dfd7d6/sandbox/linux-sandbox/1/execroot/__main__ && \
exec env - \
PATH=/home/jsy/.sdkman/candidates/scala/current/bin:/home/jsy/.sdkman/candidates/maven/current/bin:/home/jsy/.sdkman/candidates/kotlin/current/bin:/home/jsy/.sdkman/candidates/groovy/current/bin:/home/jsy/bin:/usr/share/node-v10.16.0-linux-x64/bin:/usr/share/jdk1.8.0_211/bin:/usr/share/go//bin:/home/jsy/.cargo/bin:/home/jsy/bin:/home/jsy/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
PWD=/proc/self/cwd \
/usr/bin/mpicxx -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/example/_objs/bfs_simple/bfs_simple.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/example/_objs/bfs_simple/bfs_simple.pic.o' -fPIC -D__DUALMODE_DEBUG__ -iquote . -iquote bazel-out/k8-fastbuild/bin -iquote external/bazel_tools -iquote bazel-out/k8-fastbuild/bin/external/bazel_tools -Ibazel-out/k8-fastbuild/bin/3rd/glog/_virtual_includes/glog -Ibazel-out/k8-fastbuild/bin/3rd/gflags/_virtual_includes/gflags -Ibazel-out/k8-fastbuild/bin/3rd/yas/_virtual_includes/yas -Ibazel-out/k8-fastbuild/bin/3rd/hadoop2/_virtual_includes/hdfs-dynamic -Ibazel-out/k8-fastbuild/bin/3rd/jni/_virtual_includes/jni -isystem 3rd/boost/include -isystem bazel-out/k8-fastbuild/bin/3rd/boost/include -isystem plato/graph -isystem bazel-out/k8-fastbuild/bin/plato/graph -isystem plato/util -isystem bazel-out/k8-fastbuild/bin/plato/util -isystem plato/util/libcuckoo -isystem bazel-out/k8-fastbuild/bin/plato/util/libcuckoo -isystem plato/graph/state -isystem bazel-out/k8-fastbuild/bin/plato/graph/state -isystem plato/parallel -isystem bazel-out/k8-fastbuild/bin/plato/parallel -isystem plato/graph/partition -isystem bazel-out/k8-fastbuild/bin/plato/graph/partition -isystem 3rd/jni/include/linux -isystem bazel-out/k8-fastbuild/bin/3rd/jni/include/linux -isystem plato/graph/structure -isystem bazel-out/k8-fastbuild/bin/plato/graph/structure -isystem plato/algo/bfs -isystem bazel-out/k8-fastbuild/bin/plato/algo/bfs -isystem plato/engine -isystem bazel-out/k8-fastbuild/bin/plato/engine -g -O2 -Werror -g -fPIC -static-libstdc++ -static-libgcc -fopenmp -Wall '-std=c++11' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c example/bfs_simple.cc -o bazel-out/k8-fastbuild/bin/example/_objs/bfs_simple/bfs_simple.pic.o)
Execution platform: @bazel_tools//platforms:host_platform
Use --sandbox_debug to see verbose messages from the sandbox
In file included from example/bfs_simple.cc:26:0:
example/bfs_simple.cc:42:18: error: 'input_validator_registered' defined but not used [-Werror=unused-variable]
DEFINE_validator(input, &string_not_empty);
^
bazel-out/k8-fastbuild/bin/3rd/gflags/_virtual_includes/gflags/gflags/gflags.h:139:23: note: in definition of macro 'DEFINE_validator'
static const bool name##_validator_registered = \
^
cc1plus: all warnings being treated as errors
INFO: Elapsed time: 27.966s, Critical Path: 23.45s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
about bazel
➜ # about bazel
➜ plato git:(master) command -v bazel
/home/jsy/bin/bazel
➜ ~ # bazel version runs well in my home folder
➜ ~ bazel version
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Build label: 0.26.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue May 28 08:35:14 2019 (1559032514)
Build timestamp: 1559032514
Build timestamp as int: 1559032514
➜ # but bazel doesn't show it's version in my plato code base directory
➜ plato git:(master) pwd
/home/jsy/github/plato
➜ plato git:(master) bazel version
INFO: Options provided by the client:
Inherited 'common' options: --isatty=1 --terminal_columns=169
INFO: Reading rc options for 'version' from /home/jsy/github/plato/.bazelrc:
Inherited 'common' options: --color=yes --verbose_failures --strip=never
ERROR: Unrecognized option: --verbose_failures
about mpicxx
➜ # about my mpicxx
➜ plato git:(master) command -v mpicxx
/usr/bin/mpicxx
➜ plato git:(master) mpicxx --version
g++ (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
➜ plato git:(master) mpicxx -v
mpicxx for MPICH version 3.2
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.12' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12)
about my os
➜ ~ uname -a
Linux jsy-ubuntu 4.15.0-66-generic #75~16.04.1-Ubuntu SMP Tue Oct 1 14:01:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
➜ ~ screenfetch
./+o+- jsy@jsy-ubuntu
yyyyy- -yyyyyy+ OS: Ubuntu 16.04 xenial
://+//////-yyyyyyo Kernel: x86_64 Linux 4.15.0-66-generic
.++ .:/++++++/-.+sss/` Uptime: 6h 21m
.:++o: /++++++++/:--:/- Packages: 2223
o:+o+:++.`..```.-/oo+++++/ Shell: zsh 5.1.1
.:+o:+o/. `+sssoo+/ Resolution: 1920x1080
.++/+:+oo+o:` /sssooo. DE: Unity 7.4.5
/+++//+:`oo+o /::--:. WM: Compiz
\+/+o+++`o++o ++////. WM Theme: Radiance
.++.o+++oo+:` /dddhhh. GTK Theme: Radiance [GTK2/3]
.+.o+oo:. `oddhhhh+ Icon Theme: ubuntu-mono-light
\+.++o+o``-````.:ohdhhhhh+ Font: Ubuntu 11
`:o+++ `ohhhhhhhhyo++os: CPU: Intel Core i3-8100 CPU @ 3.6GHz
.o:`.syhhhhhhh/.oo++o` RAM: 11784MiB / 15775MiB
/osyyyyyyo++ooo+++/
````` +oo+++o\:
`oo++.
I am on a ubuntu 16.04 LTS, and I installed all dependencies and executed bash 3rdtools.sh install
successfully.
BAZEL_LINKOPTS=-static-libstdc++ CC=/usr/bin/mpicxx bazel build example/...
All things went well until I executed the command below
BAZEL_LINKOPTS=-static-libstdc++ CC=/usr/bin/mpicxx LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${PWD}/3rd/hadoop2/lib bazel test --test_env=LD_LIBRARY_PATH plato/...
the output shows a file named numa.h is missing, and I found that file does not exist in the project, anyone help?
In file included from plato/graph/base.cc:22:0:
./plato/graph/base.hpp:36:18: fatal error: numa.h: No such file or directory
compilation terminated.
INFO: Elapsed time: 446.216s, Critical Path: 1.59s
INFO: 3 processes: 3 linux-sandbox.
FAILED: Build did NOT complete successfully
//plato/algo/bnc/test:test_bnc NO STATUS
//plato/algo/cgm/test:test_cgm NO STATUS
//plato/algo/cnc/test:test_cnc NO STATUS
//plato/algo/hanp/test:test_hanp NO STATUS
//plato/algo/hyperanf/test:test_hyperanf NO STATUS
//plato/algo/nstepdegrees/test:test_nstepdegrees NO STATUS
//plato/engine/test:walk NO STATUS
//plato/graph/state/test:allgather_state NO STATUS
//plato/graph/state/test:dense_state NO STATUS
//plato/graph/state/test:sparse_state NO STATUS
//plato/graph/state/test:vertex_cache NO STATUS
//plato/graph/structure/test:bcsr NO STATUS
//plato/graph/structure/test:cbcsr NO STATUS
//plato/graph/structure/test:dcsc NO STATUS
//plato/graph/structure/test:edge_cache NO STATUS
//plato/graph/structure/test:hnbbcsr NO STATUS
//plato/graph/structure/test:tcsr NO STATUS
//plato/graph/test:message_passing NO STATUS
//plato/graph/test:parsers NO STATUS
//plato/graph/test:structure NO STATUS
//plato/parallel/test:broadcast NO STATUS
//plato/parallel/test:bsp NO STATUS
//plato/parallel/test:parallel NO STATUS
//plato/util/test:aliastable NO STATUS
//plato/util/test:archive NO STATUS
//plato/util/test:bitmap NO STATUS
//plato/util/test:intersection NO STATUS
//plato/util/test:libsvm NO STATUS
//plato/util/test:mmap_alloc NO STATUS
//plato/util/test:object_buffer NO STATUS
//plato/util/test:test_hyperloglog NO STATUS
//plato/util/test:thread_local_object NO STATUS
FAILED: Build did NOT complete successfully
运行run_ragerank_local.sh通过
运行run_ragerank.sh时报错,错误如图
我并未设置关于/3rd/jni6.0/这个路径的环境变量,为何要去这里找libjava.so,这个问题如何解决?
以下是我系统的env
XDG_SESSION_ID=15905
SPARK_HOME=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark
HOSTNAME=mirage-20.novalocal
TERM=xterm
SHELL=/bin/bash
HISTSIZE=1000
SSH_CLIENT=10.103.240.44 14266 22
CONDA_SHLVL=0
SSH_TTY=/dev/pts/0
USER=tseg
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:*.xspf=01;36:
CONDA_EXE=/home/tseg/anaconda3/bin/conda
_CE_CONDA=
MAIL=/var/spool/mail/tseg
PATH=/home/tseg/anaconda3/condabin:/home/tseg/lfy/jdk1.8.0_191/bin:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/spark/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/go/bin:/bin:/home/tseg/anaconda3/bin:/home/tseg/.local/bin:/home/tseg/bin
ANTLR4_HOME=/usr/local/lib/antlr-4.7.2-complete.jar
PWD=/home/tseg/lfy/jdk1.8.0_191/jre/lib/amd64
JAVA_HOME=/home/tseg/lfy/jdk1.8.0_191
LANG=zh_CN.UTF-8
_CE_M=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/home/tseg
GOROOT=/usr/local/go
CONDA_PYTHON_EXE=/home/tseg/anaconda3/bin/python
LOGNAME=tseg
CLASSPATH=:/usr/local/lib/antlr-4.7.2-complete.jar
XDG_DATA_DIRS=/home/tseg/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
SSH_CONNECTION=10.103.240.44 14266 192.168.111.199 22
LESSOPEN=||/usr/bin/lesspipe.sh %s
XDG_RUNTIME_DIR=/run/user/1002
DISPLAY=localhost:10.0
OLDPWD=/home/tseg/lfy/plato
_=/usr/bin/env
我根据集群环境修改的sh文件
I tried downloading dependencies as instructed.
# download and build staticlly linked libraries. ./3rdtools.sh distclean && ./3rdtools.sh install
But I came across an error like this:
~/plato/3rd ~/plato
~/plato
distclean 3rd done!
~/plato/.downloads ~/plato
--2020-03-30 01:19:42-- https://dl.bintray.com/boostorg/release/1.68.0/source/boost_1_68_0.tar.gz
Resolving dl.bintray.com (dl.bintray.com)... 52.41.180.114, 54.191.3.105
Connecting to dl.bintray.com (dl.bintray.com)|52.41.180.114|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2020-03-30 01:19:44 ERROR 403: Forbidden.
'wget -O boost_1_68_0.tar.gz https://dl.bintray.com/boostorg/release/1.68.0/source/boost_1_68_0.tar.gz' exec failed with code 8, abort install process!
Is there anything I can do about it? Thanks.
目前没有看到plato的license申请,那么plato采用哪种license,在拷贝,商用上有什么样的限制?
另外我们打算何时发布第一个版本?
系统是centos 7
./build.sh 的时候出错,看了一下:3rd/mpich/bin/目录下缺少mpicxx文件,这个文件是编译过程中生成的吗,如何解决这个问题?我已经将cmake和make都升级到了最新版本。
以下是完整报错信息:
-- The CXX compiler identification is GNU 4.8.5
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake Error at /usr/share/cmake/Modules/CMakeDetermineCCompiler.cmake:56 (message):
Could not find compiler set in environment variable CC:
/3rd/mpich/bin/mpicxx.
Call Stack (most recent call first):
CMakeLists.txt:100 (enable_language)
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_C_COMPILER_ENV_VAR
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_C_COMPILER
CMake Error: Could not find cmake module file: /home/tseg/lfy/plato/.downloads/gflags-2.2.1/build/CMakeFiles/2.8.12.2/CMakeCCompiler.cmake
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
See also "/home/tseg/lfy/plato/.downloads/gflags-2.2.1/build/CMakeFiles/CMakeOutput.log".
'CXXFLAGS=-fPIC cmake .. -DBUILD_SHARED_LIBS=OFF -DBUILD_STATIC_LIBS=ON -DCMAKE_INSTALL_PREFIX=/home/tseg/lfy/plato/3rd/gflags-2.2.1' exec failed with code 1, abort install process!
Hi,
I install the plato in WSL Ubuntu, and run the test while there goes the error:
F0220 15:04:03.325654 1031 object_buffer.hpp:980] Check failed: -1 != num WARNING: ftruncate64 failed, err code: 28, err msg: No space left on device
I found the cause of this bug.
see the example code below:
int main(int argc, char** argv) {
std::string cache_dir = ".cache_test/";
boost::filesystem::create_directories(cache_dir);
uint64_t capacity = std::min(512UL * GBYTES, boost::filesystem::space(cache_dir).capacity) / sizeof(char);
plato::temporary_file_t file_(".cache_test/");
uint64_t capacity_ = capacity;
printf("capacity_=%lld\nsizeof(size_t)=%d\n512UL * GBYTES=%lld\n", capacity_, sizeof(size_t), 512UL * GBYTES);
auto num = ftruncate64(file_.fd(), sizeof(char) * capacity_);
printf("num = %d\n", num);
return 1;
}
You'll find that the boost::filesystem::space(cache_dir).capacity
is much larger than the space remaining in disk C: where the wsl ubuntu system is installed. I guess that that boost space function returns all the space of the hole physicks disk consisting of other logic disks.
So I modified the 512UL to 10UL (smaller than the space remainning in disk C:) and it goes correctly.
But it is obviously not the solution.
Maybe we should check something related to system or logic disk?
when I run the command build.sh
or bazel test plato/...
, there comes the error:
[31 / 47] 2 / 32 tests; Compiling plato/parallel/test/parallel.cc; 136s processwrapper-sandbox ... (8 actions running)
ERROR: /home/bowen/plato-master/plato/parallel/test/BUILD:28:1: C++ compilation of rule '//plato/parallel/test:bsp' failed (Exit 1) gcc failed: error executing command
(cd /home/bowen/.cache/bazel/_bazel_bowen/c8dd2e7eb9809dcc4e477247b251c27a/sandbox/processwrapper-sandbox/94/execroot/__main__ && \
exec env - \
PATH='/home/bowen/.vscode-server/bin/ae08d5460b5a45169385ff3fd44208f431992451/bin:/home/bowen/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/mnt/f/Xshell 6/:/mnt/f/Microsoft MPI/Bin/:/mnt/f/CUDA_10_0/Development/bin:/mnt/f/CUDA_10_0/Development/libnvvp:/mnt/f/CUDA_10_0/Development/lib/x64:/mnt/f/Python/Python37/Scripts/:/mnt/f/Python/Python37/:/mnt/c/Program Files (x86)/Intel/iCLS Client/:/mnt/c/Program Files/Intel/iCLS Client/:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0/:/mnt/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/DAL:/mnt/c/Program Files/Intel/Intel(R) Management Engine Components/DAL:/mnt/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/IPT:/mnt/c/Program Files/Intel/Intel(R) Management Engine Components/IPT:/mnt/c/Users/MI/.dnx/bin:/mnt/c/Program Files/Microsoft DNX/Dnvm/:/mnt/c/Program Files/Microsoft SQL Server/130/Tools/Binn/:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/f/VSCode/bin:/mnt/f/graphviz_2_38/bin:/mnt/f/MATLAB/2015a/runtime/win64:/mnt/f/MATLAB/2015a/bin:/mnt/f/MATLAB/2015a/polyspace/bin:/mnt/f/CMake/bin:/mnt/c/Program Files/Intel/WiFi/bin/:/mnt/c/Program Files/Common Files/Intel/WirelessCommon/:/mnt/f/Git/cmd:/mnt/f/Microsoft MPI/Bin:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/f/swipl/bin:/mnt/f/Python/Python27:/mnt/f/Python/Python27/Scripts:/mnt/c/Program Files/dotnet/:/mnt/f/gcc-mingw-w64/x86_64-4.8.1-win32-seh-rt_v3-rev2/mingw64/bin:/mnt/c/Users/MI/AppData/Local/Microsoft/WindowsApps:/mnt/f/texlive/2019/bin/win32:/snap/bin:/usr/include' \
PWD=/proc/self/cwd \
/usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/plato/parallel/test/_objs/bsp/bsp.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/plato/parallel/test/_objs/bsp/bsp.pic.o' -fPIC -iquote . -iquote bazel-out/k8-fastbuild/bin -iquote external/bazel_tools -iquote bazel-out/k8-fastbuild/bin/external/bazel_tools -Ibazel-out/k8-fastbuild/bin/3rd/yas/_virtual_includes/yas -Ibazel-out/k8-fastbuild/bin/3rd/gflags/_virtual_includes/gflags -Ibazel-out/k8-fastbuild/bin/3rd/glog/_virtual_includes/glog -Ibazel-out/k8-fastbuild/bin/3rd/libunwind/_virtual_includes/libunwind -isystem 3rd/googletest/googlemock -isystem bazel-out/k8-fastbuild/bin/3rd/googletest/googlemock -isystem 3rd/googletest/googlemock/include -isystem bazel-out/k8-fastbuild/bin/3rd/googletest/googlemock/include -isystem 3rd/googletest/googletest -isystem bazel-out/k8-fastbuild/bin/3rd/googletest/googletest -isystem 3rd/googletest/googletest/include -isystem bazel-out/k8-fastbuild/bin/3rd/googletest/googletest/include -isystem plato/util -isystem bazel-out/k8-fastbuild/bin/plato/util -isystem plato/parallel -isystem bazel-out/k8-fastbuild/bin/plato/parallel -isystem plato/graph -isystem bazel-out/k8-fastbuild/bin/plato/graph -isystem 3rd/boost/include -isystem bazel-out/k8-fastbuild/bin/3rd/boost/include -isystem plato/util/libcuckoo -isystem bazel-out/k8-fastbuild/bin/plato/util/libcuckoo -O2 -Wall -Werror '-std=c++11' -g -fPIC -static-libstdc++ -static-libgcc -fopenmp -Wall '-std=c++11' -Ilegacy/3rd/mpich/include -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c plato/parallel/test/bsp.cc -o bazel-out/k8-fastbuild/bin/plato/parallel/test/_objs/bsp/bsp.pic.o)
Execution platform: @local_config_platform//:host
Use --sandbox_debug to see verbose messages from the sandbox
In file included from plato/parallel/test/bsp.cc:22:0:
./plato/parallel/bsp.hpp:47:10: fatal error: mpi.h: No such file or directory
#include "mpi.h"
^~~~~~~
compilation terminated.
[33 / 47] 2 / 32 tests; Compiling plato/parallel/test/parallel.cc; 162s processwrapper-sandbox ... (6 actions running)
INFO: Elapsed time: 182.930s, Critical Path: 180.68s
INFO: 8 processes: 8 processwrapper-sandbox.
FAILED: Build did NOT complete successfully
//plato/algo/bnc/test:test_bnc NO STATUS
//plato/algo/cgm/test:test_cgm NO STATUS
//plato/algo/cnc/test:test_cnc NO STATUS
//plato/algo/hanp/test:test_hanp NO STATUS
//plato/algo/hyperanf/test:test_hyperanf NO STATUS
//plato/algo/nstepdegrees/test:test_nstepdegrees NO STATUS
//plato/engine/test:walk NO STATUS
//plato/graph/state/test:allgather_state NO STATUS
//plato/graph/state/test:dense_state NO STATUS
//plato/graph/state/test:sparse_state NO STATUS
//plato/graph/state/test:vertex_cache NO STATUS
//plato/graph/structure/test:bcsr NO STATUS
//plato/graph/structure/test:cbcsr NO STATUS
//plato/graph/structure/test:dcsc NO STATUS
//plato/graph/structure/test:edge_cache NO STATUS
//plato/graph/structure/test:hnbbcsr NO STATUS
//plato/graph/structure/test:tcsr NO STATUS
//plato/graph/test:message_passing NO STATUS
//plato/graph/test:parsers NO STATUS
//plato/graph/test:structure NO STATUS
//plato/parallel/test:broadcast NO STATUS
//plato/parallel/test:parallel NO STATUS
//plato/util/test:aliastable NO STATUS
//plato/util/test:archive NO STATUS
//plato/util/test:bitmap NO STATUS
//plato/util/test:intersection NO STATUS
//plato/util/test:libsvm NO STATUS
//plato/util/test:object_buffer NO STATUS
//plato/util/test:thread_local_object NO STATUS
//plato/parallel/test:bsp FAILED TO BUILD
//plato/util/test:mmap_alloc PASSED in 1.6s
//plato/util/test:test_hyperloglog PASSED in 0.5s
Executed 2 out of 32 tests: 2 tests pass, 1 fails to build and 29 were skipped.
FAILED: Build did NOT complete successfully
It seems that the mpi.h cannot be found. But I don't know where to add the including path.
Could anyone help?
Currently, the graph format is quite simple:
id1, id2 [,weight]
Will Plato be supporting the graph whose nodes (or edges) have more properties, or even more sophisticated graph e.g. the heterogeneous graph?
我用一些数据集运行pagerank,把进程数设为2,显示只有一个进程在工作.
I1129 01:11:34.990377 41805 detail.hpp:155] partition-0, file-count: 1
I1129 01:11:34.990427 41805 detail.hpp:155] partition-1, file-count: 0
如果多进程的并行计算呢?
另外,把进程数设为3就直接报错了
I1129 01:18:00.082996 42052 detail.hpp:155] partition-0, file-count: 1
I1129 01:18:00.084538 42052 detail.hpp:155] partition-1, file-count: 0
I1129 01:18:00.084599 42052 detail.hpp:155] partition-2, file-count: 0
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 42052 RUNNING AT 5f8a5a0dd650
= EXIT CODE: 9
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
为什么会报错,错误日志在哪看?
看到run_pagerank.sh里面设了一些环境变量,INPUT,OUTPUT之类。
我理解mpi应该是通过ssh对每个节点执行可执行文件,那么是不是每个节点都要将这些环境变量设一遍,也就是说单纯运行run_pagerank.sh是无法进行多节点计算的,是不是这样?
vertex ids are more than 2^32 in some scenarios, uint32_t is not working
麻烦问一下:50亿顶点,100亿边的无向有权图跑fast_unfolding_simple估算需要多少内存?能否提供类似#53的公式?
目前把plato 跑在k8s集群上,为了保证资源共享使用,容器直接通过mpi通讯时,我们会指定ssh的通讯端口,比如horovod任务默认用openmpi 可以通过参数指定ssh通讯20001端口:” -mca plm_rsh_args '-p 20001' “ ;我查了下mpich手册,看到通过"export MPIR_CVAR_CH3_PORT_RANGE=low:hi" 指定tcp通讯端口,并不能指定ssh端口。
请教如果用mpich 来满足这种需求呢?
Each host of these three computing servers has 512GB memory, so the total memory is enough to hold the whole graph in memory.
could you give us more documents about how to build from the scource file?
Have you done any benchmarking against something as wide spread as Janus Graph?
This looks exciting but I'm curious if it's worth a solid days commitment of time.
Analyzing: 100 targets (33 packages loaded, 0 targets configured)
ERROR: /opt/plato/plato/parallel/BUILD:38:1: no such package '3rd/glog': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
I guess you should not just cite the Gemini paper for "inspiration and guide" in the README.md when you literally lifting and shifting codes from the author's git repo.
I cannot help feeling this looks a bit like another go-chassis vs go-micro scandal, only this time it was from Tencent.
我在运行node2vec_randomwalk算法2千万点,10亿边的数据时,由于内存占用较大导致服务器中其他进程OOM了。
查看了plato图算法程序的--help以及mpiexec的使用方法https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager,没有找到控制内存使用大小的参数。
请问plato是否有配置参数或者运行参数能控制图算法运行时的内存占用大小?
如果支持的话,请问是怎么做的?如何保证弱相关性。特别是异构图的情况下
when running run_pagerank.sh on the open twitter data(twitter_rv.zip, 41 million nodes and 1.47 billion edges), i got some failed logs.
check failed: edge->dst_ < graph_info.vertices_
plato/plato/graph/structure/dcsc.hpp, line 362
auto foreach_dests = [&](bsp_send_callback_t<vid_t> send) {
auto traversal = [&](size_t, edge_unit_spec_t* edge) {
CHECK(edge->src_ < graph_info.vertices_);
CHECK(edge->dst_ < graph_info.vertices_);
send(partitioner_->get_partition_id(edge->src_, edge->dst_), edge->dst_);
if (false == graph_info.is_directed_) { // cache friendly
send(partitioner_->get_partition_id(edge->dst_, edge->src_), edge->src_);
}
return true;
};
size_t chunk_size = 64;
while (cache.next_chunk(traversal, &chunk_size)) {
}
};
should the node id of graph data be continous from zero?
when building examples with the script below
BAZEL_LINKOPTS=-static-libstdc++ CC=mpicxx bazel build example/...
an error is raised
Use --sandbox_debug to see verbose messages from the sandbox
/usr/bin/ld.gold: error: cannot find -lstdc++
When running
BAZEL_LINKOPTS=-static-libstdc++ CC=/usr/bin/mpicxx bazel build example/...
I have encountered the following error messages:
Starting local Bazel server and connecting to it... INFO: Writing tracer profile to '/home/lichen/.cache/bazel/_bazel_lichen/d05772fe1c525891155d178390fae732/command.profile.gz' INFO: Analyzed 24 targets (42 packages loaded, 14016 targets configured). INFO: Found 24 targets... ERROR: /mnt/Heavy_Duty/lichen_workspace/plato/example/BUILD:3:1: undeclared inclusion(s) in rule '//example:bitmap_scan': this rule is missing dependency declarations for the following files included by 'example/bitmap_scan.cc': '/usr/lib/x86_64-linux-gnu/openmpi/include/mpi.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/mpi_portable_platform.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/mpicxx.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/constants.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/functions.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/datatype.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/exception.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/op.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/status.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/request.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/group.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/comm.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/win.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/file.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/errhandler.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/topology.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intercomm.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/info.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/datatype_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/functions_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/request_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/comm_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/topology_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intercomm_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/group_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/op_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/errhandler_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/status_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/info_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/win_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/file_inln.h' INFO: Elapsed time: 36.804s, Critical Path: 29.88s INFO: 8 processes: 8 linux-sandbox. FAILED: Build did NOT complete successfully
When I try bazel clean --expunge
I then get the following:
INFO: Writing tracer profile to '/home/lichen/.cache/bazel/_bazel_lichen/d05772fe1c525891155d178390fae732/command.profile.gz' INFO: Analyzed 24 targets (1 packages loaded, 75 targets configured). INFO: Found 24 targets... ERROR: /mnt/Heavy_Duty/lichen_workspace/plato/plato/graph/BUILD:3:1: undeclared inclusion(s) in rule '//plato/graph:base': this rule is missing dependency declarations for the following files included by 'plato/graph/base.cc': '/usr/lib/x86_64-linux-gnu/openmpi/include/mpi.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/mpi_portable_platform.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/mpicxx.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/constants.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/functions.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/datatype.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/exception.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/op.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/status.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/request.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/group.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/comm.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/win.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/file.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/errhandler.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/topology.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intercomm.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/info.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/datatype_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/functions_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/request_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/comm_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intracomm_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/topology_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/intercomm_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/group_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/op_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/errhandler_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/status_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/info_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/win_inln.h' '/usr/lib/x86_64-linux-gnu/openmpi/include/openmpi/ompi/mpi/cxx/file_inln.h' INFO: Elapsed time: 30.042s, Critical Path: 27.95s INFO: 0 processes. FAILED: Build did NOT complete successfully
Some environments
bazel
$ bazel version WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown". Build label: 1.0.1 Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar Build time: Mon Oct 21 10:53:29 2019 (1571655209) Build timestamp: 1571655209 Build timestamp as int: 1571655209
However, bazel version
does not work in plato
directory.
mpicxx
$ mpicxx --version g++ (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. $ mpicxx -v Using built-in specs. COLLECT_GCC=/usr/bin/g++ COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper OFFLOAD_TARGET_NAMES=nvptx-none OFFLOAD_TARGET_DEFAULT=1 Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.4.0-1ubuntu1~18.04.1' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
如题。
看代码,没有发现kerberos相关的配置和处理
We follow the test instruction:
BAZEL_LINKOPTS=-static-libstdc++ CC=mpicxx LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${PWD}/3rd/hadoop2/lib bazel test --test_env=LD_LIBRARY_PATH plato/...
and get the result below:
there are 3 warnings and 1 failure. The failure log gives
Is the failure acceptable? Does "plato" work normally?
when building examples with the script below
BAZEL_LINKOPTS=-static-libstdc++ CC=mpicxx bazel build example/...
an error is raised
./plato/graph/base.hpp:36:18: fatal error: numa.h: No such file or director
when running BAZEL_LINKOPTS=-static-libstdc++ CC=/usr/bin/mpicxx bazel build example/...
`WARNING: Waiting for server process to terminate (waited 5 seconds, waiting at most 60)
Starting local Bazel server and connecting to it...
INFO: Writing tracer profile to '/home/tian/.cache/bazel/_bazel_tian/d46c134286fdbb1c2928eea0643ff387/command.profile.gz'
INFO: Analyzed 24 targets (42 packages loaded, 14017 targets configured).
INFO: Found 24 targets...
INFO: Deleting stale sandbox base /home/tian/.cache/bazel/_bazel_tian/d46c134286fdbb1c2928eea0643ff387/sandbox
ERROR: /home/tian/wzb/plato/example/BUILD:20:1: Linking of rule '//example:function_call' failed (Exit 1) mpicxx failed: error executing command
(cd /home/tian/.cache/bazel/_bazel_tian/d46c134286fdbb1c2928eea0643ff387/sandbox/linux-sandbox/6/execroot/main &&
exec env -
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/tian/bin
PWD=/proc/self/cwd
/usr/bin/mpicxx @bazel-out/k8-fastbuild/bin/example/function_call-2.params)
Execution platform: @local_config_platform//:host
Use --sandbox_debug to see verbose messages from the sandbox
3rd/glog/lib/libglog.a(libglog_la-utilities.o):utilities.cc:function google::GetStackTrace(void**, int, int): error: undefined reference to '_Ux86_64_getcontext'
3rd/glog/lib/libglog.a(libglog_la-utilities.o):utilities.cc:function google::GetStackTrace(void**, int, int): error: undefined reference to '_ULx86_64_init_local'
3rd/glog/lib/libglog.a(libglog_la-utilities.o):utilities.cc:function google::GetStackTrace(void**, int, int): error: undefined reference to '_ULx86_64_get_reg'
3rd/glog/lib/libglog.a(libglog_la-utilities.o):utilities.cc:function google::GetStackTrace(void**, int, int): error: undefined reference to '_ULx86_64_step'
collect2: error: ld returned 1 exit status
INFO: Elapsed time: 165.786s, Critical Path: 154.06s
INFO: 2 processes: 2 linux-sandbox.
FAILED: Build did NOT complete successfully`
any example on node2vec including running scripts, graph data and so on.
建议加上python支持,现在还有几个人写C++?快加上python库的支持吧
封装一层
(graph) [root@gpudashujucbd010030 plato-master]# scripts/run_pagerank_local_ppd.sh
+++ dirname scripts/run_pagerank_local_ppd.sh
++ realpath scripts
check output stream failed https://github.com/Tencent/plato/blob/master/plato/util/foutput.cc#L125.
when building examples with the script below
BAZEL_LINKOPTS=-static-libstdc++ CC=mpicxx bazel build example/...
an error is raised
ERROR: /home/xxxxxx/plato-master/plato/graph/BUILD:3:1: undeclared inclusion(s) in rule '//plato/graph:base':
this rule is missing dependency declarations for the following files included by 'plato/graph/base.cc':
'/home/xxxxxx/openmpi/include/mpi.h'
'/home/xxxxxx/openmpi/include/mpi_portable_platform.h'
other MPI function has same problem
env: Ubuntu 18.04
//plato/util/test:object_buffer FAILED in 2.8s
Build completed, 1 test failed
/plato$ ./scripts/run_pagerank_local.sh
+++ dirname ./scripts/run_pagerank_local.sh
++ realpath ./scripts
I entered plato/bazel-bin directory, but I didn't find example directory.
安装完运行run_pagerank_local.sh和run_pagerank.sh(修改input/output)均通过。
但是运行lpa和tree_stat_simple样例出错
./bazel-bin/example/lpa --input /test/home/hadoop/tar/plato-master/data/graph/v100_e2150_ua_c3.csv --output /tmp/lpa
./bazel-bin/example/tree_stat_simple --input /test/home/hadoop/tar/plato-master/data/graph/v100_e2150_ua_c3.csv --output /tmp/tree
运行node2vec_randomwalk通过
./bazel-bin/example/node2vec_randomwalk --input /test/home/hadoop/tar/plato-master/data/graph/v100_e2150_ua_c3.csv --output /tmp/node2vec
问题:我设置了java环境,为什么还要到路径/test/home/hadoop/tar/plato-master/3rd/jni6.0/中找so文件?为什么同样的环境,有的用例能跑过,有的却跑不过呢?我看了#45的问题,#45最后也没有解决如何配置。
我继续通过ln -s的方式将/usr/lib/jvm/java-8-openjdk-amd64/中的so软链接到/test/home/hadoop/tar/plato-master/3rd/jni6.0/路径,最后还是报错
aggregate_message函数里,将当前partition的图的数据读出来,然后又按vid获取其所属的partition,将数据发出去。这样的话,不就是传给自己吗,如果是这样,岂不是多此一举?
请问计算规模可以水平扩展吗?会受单机内存的限制吗?如果有限制,那么支持最大多少顶点和边呢?
Plato code can give more detailed code comment information. Release more help documents.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.