Giter VIP home page Giter VIP logo

mapreduce_itemcf's Introduction

基于MapReduce实现物品协同过滤算法(ItemCF)

具体参考博客

补充:hadoop三种执行方式

MR执行环境有两种:本地测试环境,服务器环境

本地测试环境(windows): 在windows的hadoop目录bin目录有一个winutils.exe

1、在windows下配置hadoop的环境变量
2、拷贝debug工具(winutils.ext)到HADOOP_HOME/bin
3、修改hadoop的源码 ,注意:确保项目的lib需要真实安装的jdk的lib
4、MR调用的代码需要改变:
	a、src不能有服务器的hadoop配置文件
	b、在调用是使用:
	  Configuration config = new  Configuration();
	  config.set("fs.defaultFS", "hdfs://node7:8020");
	  config.set("yarn.resourcemanager.hostname", "node7");

服务器环境: 首先需要在src下放置服务器上的hadoop配置文件

1、在本地直接调用,执行过程在服务器上(真正企业运行环境)

a、把MR程序打包(jar),直接放到本地
b、修改hadoop的源码 ,注意:确保项目的lib需要真实安装的jdk的lib
c、增加一个属性:config.set("mapred.jar", "C:\\Users\\Administrator\\Desktop\\wc.jar");
d、本地执行main方法,servlet调用MR。

2、直接在服务器上,使用命令的方式调用,执行过程也在服务器上

a、把MR程序打包(jar),传送到服务器上
b、通过: hadoop jar jar路径  类的全限定名
       如:hadoop jar wc.jar com.zxl.mr.wc.RunJob

注:一些jar包可以通过 java jar xxx.jar 运行,使用hadoop则默认会把hadoop所依赖包添加

mapreduce_itemcf's People

Contributors

luckyzxl2016 avatar

Watchers

James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.