Giter VIP home page Giter VIP logo

a-simrank-algorithm-implementation-using-spark's Introduction

Spark SimRank Algorithm Implementation

This package includs 5 different SimRank implementations: DFS (depth-first search) MapReduce, naive MapReduce, delta MapReduce, matrix multiplication and PageRank-like Random Walk with Restart. You can choose different implementation through configuration.

This implementation is compatible with Spark 0.8.1+ version, you can compile using sbt assembly, before that please configure the correct Hadoop version in build.sbt.

How to Run

  1. Using graph_generate.py to generate random adjacency matrix, you can configure GRAPH_SIZE (number of vertices), EDGE_SIZE (number of edges) to control the matrix rank, this script will serialize matrix to file.
  2. Generate initial similarity matrix. Using ./run simrank.SimRankDataPrepare to generate data, it should be noted that two parameters graphASize and graphBSize, which specifies the vertices number of two sub-graphs in the bipartite graph, should be the same as step 1's generated result.
  3. Configure config/config.properties and run by ./run simrank.SimRankImpl.

Notes

  • Step 2 data preparation will generate one initial similarity matrix and one identity matrix, here similarity matrix is a upper triangluar matrix, implementation delta MapReduce will use this matrix as a initial input similarity matrix, for other implementations identity matrix would be enough to use as a initial input similarity matrix, you can skip step 2 if identity matrix is created by yourself.
  • Here we focused on matrix multiplication implementation, other implementations are implemented only for reference, may not be well tuned.

This implementation is open sourced under Apache License Version 2.0.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.