alanfgates / programmingpig Goto Github PK

Data and example code for Programming Pig, by Alan F. Gates

Java 89.41% Perl 1.07% Python 9.52%

programmingpig's Introduction

This repository contains the example data and Pig Latin scripts for the book
_Programming Pig_ by Alan F. Gates, published by O'Reilly.  All data used in
the examples is in the public domain.  All Pig Latin scripts and associated 
user defined functions are released under the Apache 2.0 license.

In this repository you will find the code used to take the data from it source
and prepare if for the example in the setup directory.  The data directory
contains the cleansed data, ready for use in the examples.  The examples 
directory contains the example Pig Latin scripts, divided by chapters.  The 
udfs directory contains the UDFs used in the examples.

programmingpig's People

Stargazers

Watchers

Forkers

ddaniels katandtonic turtlelabs liuyunliu seekshreyas fengzhenxing coreyhyllested sagaratgithub jekey gy21 willieavendano thekorn joschwa9 hss4233 kungfupanda6 amitmahale jfmwz deepamadhavu vamsijkrishna maixiaohai schan002 bnamatherdhala vijju7kittu omkarv zcwfeng vijaybharat ckeating skangayam premzparadize kkunduru vrameshinbox rajeshre hanchensu which07 njvijay hnflee grueda neojason2013 deepujain strategist922 yuanke fone4u mazamgov diegojimeno pombredanne swathibhat28 veetil09 cherif19 denny9527 scottburger bugcy013 punsiitg ajeancharles alphamanimal hangelwen rajnishgoswami davidzh77 krishnakalyan3 hwhwell tong06portal pracheeragarwal-zz sheltowt liunkor pradeepvutukuri seanfranks igiffie hiteshtulsani ppaawwaarr saryamane mrs-hudson bcjohns lkhn-pjpt pierreau wangw5 yingchen aymar bhrgvishal agomariz ichbineinberliner gbortz27 sphalguna krishnasamudrala-tal tomakant eshmelkov mayurkale22 leoliuxd satyargudimetla shihuacai1989 tsuralik shivchoudhury sparrowkong tinychen433 mbhushan mesdsingh sellappan vikassarin krishnasrikanthk sksundaram mg398e sukantdas

programmingpig's Issues

ch2/average_dividend expects data file to be in the same directory

fix was to change dividends = load 'NYSE_dividends' as (exchange, symbol, date, dividend); to dividends = load '../../data/NYSE_dividends' as (exchange, symbol, date, dividend);

PageRank example getting stuck due to explosion of out/pagerank_data_N

I just tried the PageRank example from Chapter 9, but it gets stuck in the reduce phase of a process on the 5th iteration of the algorithm.

Digging a little deeper, it looks like the contents of the pagerank_data_N directories are exploding in size as N increases:

hdfs dfs -du -h out/
5        out/max_diff_1
18       out/max_diff_2
19       out/max_diff_3
18       out/max_diff_4
333.7 K  out/pagerank_data_1
768.6 K  out/pagerank_data_2
9.2 M    out/pagerank_data_3
2.2 G    out/pagerank_data_4
33.8 G   out/pagerank_data_5

Looking further into the contents of pagerank_data_N, it looks like a large number of duplicates end up in the output. I was able to fix by applying DISTINCT to new_pagerank:

new_pagerank = ...
new_pagerank_final = distinct new_pagerank; 
store new_pagerank_final into '$docs_out' using PigStorage('\t');
..

alanfgates / programmingpig Goto Github PK

programmingpig's Introduction

programmingpig's People

Stargazers

Watchers

Forkers

programmingpig's Issues

ch2/average_dividend expects data file to be in the same directory

PageRank example getting stuck due to explosion of out/pagerank_data_N

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent