Giter VIP home page Giter VIP logo

powergraph_update's Introduction

PowerGraph是2012年发表在OSDI会议上的一篇有关于图计算系统的文章。由于其本身包含了多种传统图算法,并且支持分布式计算,因此直到今日依然会成为众多论文的基线对比,或者是通过PG来验证一些算法的效果。

但是由于PowerGraph年久失修,目前无法直接通过github的源码来进行编译,主要的原因还是第三方库的URL失效。由于最近我们的论文有使用到PG计算框架,也是踩了许多坑,再次进行记录,希望可以方便后面还需要使用PG来进行测试的人员。

github地址:GitHub - BearBiscuit05/PowerGraph_update

源仓库地址:https://github.com/jegonzal/PowerGraph

  • Ubuntu16.04
  • gcc 5.4
  • g++5.4
  • jdk1.8
  • build-essential
  • Zlib
sudo apt-get update

sudo apt-get install gcc g++ build-essential libopenmpi-dev openmpi-bin default-jdk cmake zlib1g-dev git

git clone https://github.com/BearBiscuit05/PowerGraph_update.git

cd PowerGraph_update

./configure

cd release/toolkits/graph_analytics

make -j2

或者直接从docker hub拉取镜像

docker pull bearbiscuit/pgimage:latest

PowerGraph可以很方便的进行一些传统图计算任务的运行,经常作为论文的基线测试程序,用于测试分区算法的健壮性,因此,为了方便在PG上测试分区算法在实际分布式场景下的表现,我们对PG的代码进行了一些调整,使得PG可以直接接受来自外部的分区结果,以供应用。

以一个graph.txt文件为例,每一行包含三个参数,分别是src,dst,partid,其中partid表明对应的边(src,dst)最后被分配到哪一个分区中。三个参数之间需要用\t分隔。

#SRC\tDST\tpartid
1	2	0
2	3	1
3	0	1

由于我们的导入接口设置在random中,替代了原先的random方法,因此,运行的命令调整为

./pagerank --graph_opts="ingress=random" --graph /data/in_S5P.txt --format self_tsv

# 其中pagerank通过之前的make命令完成编译,路径在release/toolkits/graph_analytics下
# --graph参数接着的是图的文件路径
# format本意是告诉PG你的图文件结构,self_tsv便是上面提到的三参数文件结构

分布式PG运行流程

前一节中已经提供了详细的信息

docker pull bearbiscuit/pgimage:latest
git clone https://github.com/BearBiscuit05/PowerGraph_update.git

首先,docker集群必须在同一个子网下,因此我们需要先创建子网

docker network create --driver bridge --subnet 43.0.0.0/16 --gateway 43.0.0.1 pg_network

随后,我们修改runDocker.sh脚本中的内容,以调整至你所需要的内容(如映射PowerGraph,映射图数据)

在docker集群搭建完成后,进入任意容器,并进入PG程序运行的目录下,随后运行

mpiexec --allow-run-as-root -n 2 -hostfile /root/data/machines ./pagerank --graph_opts="ingress=random" --graph /root/data/test.edges --format self_tsv

powergraph_update's People

Contributors

ylow avatar jegonzal avatar dbickson avatar haijieg avatar dhruvbatra avatar timmuss avatar philleonard avatar dmcennis avatar prakritibanik avatar aroma123 avatar svilen-mihaylov-db avatar scott-vsi avatar andre-martins avatar bearbiscuit05 avatar fabiopetroni avatar sliverdragon37 avatar iglesias avatar dinahs avatar parkerzf avatar biancamo avatar cerquide avatar qiwang321 avatar realstolz avatar kelarini avatar hupili avatar darrengarvey avatar caio avatar boorad avatar lfzark avatar august782 avatar

Stargazers

Qianyu Cheng avatar Shangyou Wang avatar  avatar

Watchers

 avatar

Forkers

kssamwang

powergraph_update's Issues

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.