rjurney / agile_data_code Goto Github PK

View Code? Open in Web Editor NEW

158.0 158.0 137.0 2.27 MB

Chapter-wise code for Agile Data the O'Reilly book

Python 0.55% JavaScript 98.74% Shell 0.03% CSS 0.68%

agile_data_code's People

Contributors

Stargazers

Watchers

Forkers

pdendek namdle jstirnaman hellojinjie benbeck aarmjohn ike-okonkwo daniel1028 cindeem dischord01 dancheah a-covar buysse timgushue aklimenko web5design alphamanimal imclab mdjmay dmwalker101 supermiemie stargaser lucafoschini joechong88 bencassedy billcary aburkleaux richarddunks vsl888 arunpn nolanbjohnson roycoding siddharthaiiti lukman1416 iainsproat gwillink wemara coralyh sungsoo anb2 niarfe juaneschutte wilsoncao jornu firthh foxgrover pbaiz geekjimbo ajdg aendinalt danpliske sudhirpatil skreaminskull sejii rsherman parsemime patou2013 jshorwitz ountzza waynedev dbl001 2dpodcast guzu92 dsdinter alouisos vincewen davparker marco-buttu kjspring hanifmahboobi binnie869 qingkaikong ssmaroju erenewong navarq xiaogan2005 amkkouz nsolai joebradly an100 sqlsc yenshikwan chriscornell2015 sanpi0205 rebecca-g tig3rkatze qtagtech irobla praveen766 marystreet krishnatray ngpestelos leoliujie andrewmlewis roszcz kirby49 floresfdev jgblouin sandortucakov luoqeng

agile_data_code's Issues

Cloud Setup in Amazon EMR

Still no success in loading the afro-1.7.4.jar and json-simple-1.1.jar into EMR. I've managed to upload the jar files in S3 under one bucket. From there, within EMR, I added new steps to load custom classes from the S3 location. However, it always result in fail. Error seen was as follow:

2014-01-08T15:03:09.422Z INFO Fetching jar file.
2014-01-08T15:03:10.550Z WARN Error fetching jar file. java.lang.RuntimeException: Error whilst fetching 's3://agile.data88/jars/build/ivy/lib/Pig'

File "./gmail.py", line 7, in <module> from lepl.apps.rfc3696 import EmailUtils ImportError: No module named lepl.apps.rfc3696

when i try to run this:

./gmail.py -m automatic -u [email protected] -p 'pwd' -s ./email.avro.schema -f '[Gmail]/All Mail' -o /tmp/test_mbox 2>&1 &

i get this error:

Traceback (most recent call last):
File "./gmail.py", line 7, in
from lepl.apps.rfc3696 import EmailUtils
ImportError: No module named lepl.apps.rfc3696

Please let me know if you have any suggestions.
Thanks
Sai

Error (snappy) installing requirements in virtualenv.

(virtualenv)DLs-MacBook-Pro:Agile_Data_Code davidlaxer$ pip install -r requirements.txt
Downloading/unpacking BareNecessities==0.2.8 (from -r requirements.txt (line 1))
Downloading BareNecessities-0.2.8.tar.gz
Running setup.py egg_info for package BareNecessities

Downloading/unpacking ESClient==0.5.3 (from -r requirements.txt (line 2))
Downloading ESClient-0.5.3.tar.gz
Running setup.py egg_info for package ESClient

Downloading/unpacking Flask==0.9 (from -r requirements.txt (line 3))
Downloading Flask-0.9.tar.gz (481kB): 481kB downloaded
Running setup.py egg_info for package Flask

warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'

Downloading/unpacking Jinja2==2.6 (from -r requirements.txt (line 4))
Downloading Jinja2-2.6.tar.gz (389kB): 389kB downloaded
Running setup.py egg_info for package Jinja2

warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
warning: no previously-included files matching '*.pyo' found under directory 'docs'

Downloading/unpacking LEPL==5.1.3 (from -r requirements.txt (line 5))
Downloading LEPL-5.1.3.tar.gz (203kB): 203kB downloaded
Running setup.py egg_info for package LEPL

Downloading/unpacking Mail==2.1.0 (from -r requirements.txt (line 6))
Downloading Mail-2.1.0.tar.gz
Running setup.py egg_info for package Mail

Downloading/unpacking Werkzeug==0.8.3 (from -r requirements.txt (line 7))
Downloading Werkzeug-0.8.3.tar.gz (1.1MB): 1.1MB downloaded
Running setup.py egg_info for package Werkzeug

warning: no files found matching '*' under directory 'werkzeug/debug/templates'
warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'

Downloading/unpacking distribute==0.6.31 (from -r requirements.txt (line 8))
Downloading distribute-0.6.31.tar.gz (643kB): 643kB downloaded
Running setup.py egg_info for package distribute

Downloading/unpacking python-snappy (from -r requirements.txt (line 9))
Downloading python-snappy-0.5.tar.gz
Running setup.py egg_info for package python-snappy

Downloading/unpacking avro==1.7.3 (from -r requirements.txt (line 10))
Downloading avro-1.7.3.tar.gz
Running setup.py egg_info for package avro

Obtaining pyelasticsearch from git+https://github.com/rhec/pyelasticsearch.git#egg=pyelasticsearch (from -r requirements.txt (line 11))
Cloning https://github.com/rhec/pyelasticsearch.git to /Users/davidlaxer/virtualenv/src/pyelasticsearch
Running setup.py egg_info for package pyelasticsearch

Downloading/unpacking pymongo==2.4.1 (from -r requirements.txt (line 12))
Downloading pymongo-2.4.1.tar.gz (275kB): 275kB downloaded
Running setup.py egg_info for package pymongo

Downloading/unpacking requests==1.0.4 (from -r requirements.txt (line 13))
Downloading requests-1.0.4.tar.gz (336kB): 336kB downloaded
Running setup.py egg_info for package requests

Downloading/unpacking simplejson==2.6.2 (from -r requirements.txt (line 14))
Downloading simplejson-2.6.2.tar.gz (53kB): 53kB downloaded
Running setup.py egg_info for package simplejson

Requirement already satisfied (use --upgrade to upgrade): wsgiref==0.1.2 in /Users/davidlaxer/anaconda/lib/python2.7 (from -r requirements.txt (line 15))
Downloading/unpacking numpy (from -r requirements.txt (line 16))
Downloading numpy-1.9.0.tar.gz (4.0MB): 4.0MB downloaded
Running setup.py egg_info for package numpy
Running from numpy source directory.

warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution

Downloading/unpacking honcho (from -r requirements.txt (line 17))
Downloading honcho-0.5.0.tar.gz
Running setup.py egg_info for package honcho

Downloading/unpacking scipy (from -r requirements.txt (line 18))
Downloading scipy-0.14.0.tar.gz (10.2MB): 10.2MB downloaded
Running setup.py egg_info for package scipy

warning: no previously-included files matching '*_subr_*.f' found under directory 'scipy/linalg/src/id_dist/src'
no previously-included directories found matching 'scipy/special/tests/data/boost'
no previously-included directories found matching 'scipy/special/tests/data/gsl'
no previously-included directories found matching 'doc/build'
no previously-included directories found matching 'doc/source/generated'
no previously-included directories found matching '*/__pycache__'
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.bak' found anywhere in distribution
warning: no previously-included files matching '*.swp' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution

Downloading/unpacking dotcloud (from -r requirements.txt (line 19))
Downloading dotcloud-0.9.8.tar.gz
Running setup.py egg_info for package dotcloud

Downloading/unpacking python-dateutil (from -r requirements.txt (line 20))
Downloading python-dateutil-2.2.tar.gz (259kB): 259kB downloaded
Running setup.py egg_info for package python-dateutil

Downloading/unpacking nltk (from -r requirements.txt (line 21))
Downloading nltk-3.0.0.tar.gz (962kB): 962kB downloaded
Running setup.py egg_info for package nltk

warning: no files found matching 'Makefile' under directory '*.txt'
warning: no previously-included files matching '*~' found anywhere in distribution

Requirement already satisfied (use --upgrade to upgrade): six in /Users/davidlaxer/virtualenv/lib/python2.7/site-packages (from pyelasticsearch->-r requirements.txt (line 11))
Downloading/unpacking argparse (from dotcloud->-r requirements.txt (line 19))
Downloading argparse-1.2.1.tar.gz (69kB): 69kB downloaded
Running setup.py egg_info for package argparse

warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.orig' found anywhere in distribution
warning: no previously-included files matching '*.rej' found anywhere in distribution
no previously-included directories found matching 'doc/_build'
no previously-included directories found matching 'env24'
no previously-included directories found matching 'env25'
no previously-included directories found matching 'env26'
no previously-included directories found matching 'env27'

Downloading/unpacking colorama (from dotcloud->-r requirements.txt (line 19))
Downloading colorama-0.3.2.tar.gz
Running setup.py egg_info for package colorama

Installing collected packages: BareNecessities, ESClient, Flask, Jinja2, LEPL, Mail, Werkzeug, distribute, python-snappy, avro, pyelasticsearch, pymongo, requests, simplejson, numpy, honcho, scipy, dotcloud, python-dateutil, nltk, argparse, colorama
Running setup.py install for BareNecessities

Running setup.py install for ESClient

Running setup.py install for Flask

warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'docs/_themes/.git'

Running setup.py install for Jinja2

warning: no previously-included files matching '*' found under directory 'docs/_build'
warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
warning: no previously-included files matching '*.pyo' found under directory 'docs'

Running setup.py install for LEPL

Running setup.py install for Mail

Running setup.py install for Werkzeug

warning: no files found matching '*' under directory 'werkzeug/debug/templates'
warning: no files found matching '*' under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'docs'
warning: no previously-included files matching '*.pyo' found under directory 'docs'
warning: no previously-included files matching '*.pyc' found under directory 'tests'
warning: no previously-included files matching '*.pyo' found under directory 'tests'
warning: no previously-included files matching '*.pyc' found under directory 'examples'
warning: no previously-included files matching '*.pyo' found under directory 'examples'
no previously-included directories found matching 'docs/_build'

Running setup.py install for distribute
Before install bootstrap.
Scanning installed packages
Setuptools installation detected at /Users/davidlaxer/virtualenv/lib/python2.7/site-packages
Non-egg installation
Moving elements out of the way...
Renaming /Users/davidlaxer/virtualenv/lib/python2.7/site-packages/setuptools-1.1.5-py2.7.egg-info to /Users/davidlaxer/virtualenv/lib/python2.7/site-packages/setuptools-1.1.5-py2.7.egg-info.OLD.1412614402.06
Renaming /Users/davidlaxer/virtualenv/lib/python2.7/site-packages/setuptools to /Users/davidlaxer/virtualenv/lib/python2.7/site-packages/setuptools.OLD.1412614402.07
Renaming /Users/davidlaxer/virtualenv/lib/python2.7/site-packages/pkg_resources.py to /Users/davidlaxer/virtualenv/lib/python2.7/site-packages/pkg_resources.py.OLD.1412614402.07
Could not find the /Users/davidlaxer/virtualenv/lib/python2.7/site-packages/site.py element of the Setuptools distribution
Patching complete.
Relaunching...
Before install bootstrap.
Scanning installed packages
Setuptools installation detected at /Users/davidlaxer/virtualenv/lib/python2.7/site-packages
Non-egg installation
Moving elements out of the way...
/Users/davidlaxer/virtualenv/lib/python2.7/site-packages/setuptools-1.1.5-py2.7.egg-info/EGG-INFO/PKG-INFO already patched.
/Users/davidlaxer/virtualenv/lib/python2.7/site-packages/setuptools-1.1.5-py2.7.egg-info already patched.
Installing easy_install script to /Users/davidlaxer/virtualenv/bin
Installing easy_install-2.7 script to /Users/davidlaxer/virtualenv/bin
After install bootstrap.
Creating /Users/davidlaxer/virtualenv/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info
Creating /Users/davidlaxer/virtualenv/lib/python2.7/site-packages/setuptools.pth
Running setup.py install for python-snappy
building '_snappy' extension
gcc -fno-strict-aliasing -I/Users/davidlaxer/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/davidlaxer/anaconda/include/python2.7 -c snappymodule.cc -o build/temp.macosx-10.5-x86_64-2.7/snappymodule.o
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ [enabled by default]
snappymodule.cc:45:28: warning: '_state' defined but not used [-Wunused-variable]
static struct module_state _state;
^
gcc -fno-strict-aliasing -I/Users/davidlaxer/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/davidlaxer/anaconda/include/python2.7 -c crc32c.c -o build/temp.macosx-10.5-x86_64-2.7/crc32c.o
gcc -bundle -undefined dynamic_lookup -L/Users/davidlaxer/anaconda/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.5-x86_64-2.7/snappymodule.o build/temp.macosx-10.5-x86_64-2.7/crc32c.o -L/Users/davidlaxer/anaconda/lib -lsnappy -o build/lib.macosx-10.5-x86_64-2.7/_snappy.so
ld: library not found for -lsnappy
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
Complete output from command /Users/davidlaxer/virtualenv/bin/python2 -c "import setuptools;file='/Users/davidlaxer/virtualenv/build/python-snappy/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/nj/nphdkhyj6s1dttb0pd9zb2wc0000gn/T/pip-ujZVlI-record/install-record.txt --single-version-externally-managed --install-headers /Users/davidlaxer/virtualenv/include/site/python2.7:
running install

running build

running build_py

creating build

creating build/lib.macosx-10.5-x86_64-2.7

copying snappy.py -> build/lib.macosx-10.5-x86_64-2.7

running build_ext

building '_snappy' extension

creating build/temp.macosx-10.5-x86_64-2.7

gcc -fno-strict-aliasing -I/Users/davidlaxer/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/davidlaxer/anaconda/include/python2.7 -c snappymodule.cc -o build/temp.macosx-10.5-x86_64-2.7/snappymodule.o

cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ [enabled by default]

snappymodule.cc:45:28: warning: '_state' defined but not used [-Wunused-variable]

static struct module_state _state;

gcc -fno-strict-aliasing -I/Users/davidlaxer/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/davidlaxer/anaconda/include/python2.7 -c crc32c.c -o build/temp.macosx-10.5-x86_64-2.7/crc32c.o

gcc -bundle -undefined dynamic_lookup -L/Users/davidlaxer/anaconda/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.5-x86_64-2.7/snappymodule.o build/temp.macosx-10.5-x86_64-2.7/crc32c.o -L/Users/davidlaxer/anaconda/lib -lsnappy -o build/lib.macosx-10.5-x86_64-2.7/_snappy.so

ld: library not found for -lsnappy

collect2: error: ld returned 1 exit status

error: command 'gcc' failed with exit status 1

Cleaning up...
Command /Users/davidlaxer/virtualenv/bin/python2 -c "import setuptools;file='/Users/davidlaxer/virtualenv/build/python-snappy/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/nj/nphdkhyj6s1dttb0pd9zb2wc0000gn/T/pip-ujZVlI-record/install-record.txt --single-version-externally-managed --install-headers /Users/davidlaxer/virtualenv/include/site/python2.7 failed with error code 1 in /Users/davidlaxer/virtualenv/build/python-snappy
Storing complete log in /Users/davidlaxer/.pip/pip.log
(virtualenv)David-Laxers-MacBook-Pro:Agile_Data_Code davidlaxer$ pip install snappy
Downloading/unpacking snappy
Could not find any downloads that satisfy the requirement snappy
Cleaning up...
No distributions at all found for snappy
Storing complete log in /Users/davidlaxer/.pip/pip.log
(virtualenv)David-Laxers-MacBook-Pro:Agile_Data_Code davidlaxer$ pip install python-snappy
Downloading/unpacking python-snappy
Downloading python-snappy-0.5.tar.gz
Running setup.py egg_info for package python-snappy

Installing collected packages: python-snappy
Running setup.py install for python-snappy
building '_snappy' extension
gcc -fno-strict-aliasing -I/Users/davidlaxer/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/davidlaxer/anaconda/include/python2.7 -c snappymodule.cc -o build/temp.macosx-10.5-x86_64-2.7/snappymodule.o
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ [enabled by default]
snappymodule.cc:45:28: warning: '_state' defined but not used [-Wunused-variable]
static struct module_state _state;
^
gcc -fno-strict-aliasing -I/Users/davidlaxer/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/davidlaxer/anaconda/include/python2.7 -c crc32c.c -o build/temp.macosx-10.5-x86_64-2.7/crc32c.o
gcc -bundle -undefined dynamic_lookup -L/Users/davidlaxer/anaconda/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.5-x86_64-2.7/snappymodule.o build/temp.macosx-10.5-x86_64-2.7/crc32c.o -L/Users/davidlaxer/anaconda/lib -lsnappy -o build/lib.macosx-10.5-x86_64-2.7/_snappy.so
ld: library not found for -lsnappy
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
Complete output from command /Users/davidlaxer/virtualenv/bin/python2 -c "import setuptools;file='/Users/davidlaxer/virtualenv/build/python-snappy/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/nj/nphdkhyj6s1dttb0pd9zb2wc0000gn/T/pip-9GakDF-record/install-record.txt --single-version-externally-managed --install-headers /Users/davidlaxer/virtualenv/include/site/python2.7:
running install

running build

running build_py

creating build

creating build/lib.macosx-10.5-x86_64-2.7

copying snappy.py -> build/lib.macosx-10.5-x86_64-2.7

running build_ext

building '_snappy' extension

creating build/temp.macosx-10.5-x86_64-2.7

cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ [enabled by default]

snappymodule.cc:45:28: warning: '_state' defined but not used [-Wunused-variable]

static struct module_state _state;

gcc -fno-strict-aliasing -I/Users/davidlaxer/anaconda/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/davidlaxer/anaconda/include/python2.7 -c crc32c.c -o build/temp.macosx-10.5-x86_64-2.7/crc32c.o

ld: library not found for -lsnappy

collect2: error: ld returned 1 exit status

error: command 'gcc' failed with exit status 1

ch07/pig/related_email_addresses.pig error/issues

Can someone verify that they can run ch07/pig/related_email_addresses.pig without any issues or error?

If so, please let me know what version of pig and mongo-hadoop that you are using.
I am currently using pig-0.12.0 and mongo-hadoop-r1.2.0. It seems there are bit of errors that I am currently encounter for the exercises.

thanks,

James

ch03 - p50 - Issues Installing mongo-Hadoop

Hi,

Trying to follow the instructions of (1) cloning mongo-hadoop.git and (2) "sbt package", I do not get any ".jar" file created thanks to that :(

I never used [sbt] before. But I am block here.

Here are my commands :

$sbt package
Unable to find a $JAVA_HOME at "/usr", continuing with system-provided Java...
[info] Set current project to mongo-hadoop (in build file:/Applications/MongoDB/mongo-hadoop/mongo-hadoop/)
[info] Updating {file:/Applications/MongoDB/mongo-hadoop/mongo-hadoop/}mongo-hadoop...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Packaging /Applications/MongoDB/mongo-hadoop/mongo-hadoop/target/scala-2.10/mongo-hadoop_2.10-0.1-SNAPSHOT.jar ...
[info] Done packaging.
[success] Total time: 2 s, completed 24 oct. 2014 00:48:23

$find .|grep jar
./gradle/wrapper/gradle-wrapper.jar
./target/scala-2.10/mongo-hadoop_2.10-0.1-SNAPSHOT.jar

Any insight appreciated...

libsnappy1 - package not found

This package cannot be found during the package compilation in dotcloud. Is there a new library to be added?

13:19:00.948826: [www] Reading package lists...
13:19:02.232602: [www] stdin: is not a tty
13:19:02.233042: [www] Reading package lists...
13:19:02.358834: [www] Building dependency tree...
13:19:02.359280: [www] Reading state information...
13:19:02.408824: [www] E: Couldn't find package libsnappy1

fatal error: 'snappy-c.h' file not found

On OSx Mavericks

During: pip install -r requirements.txt

snappymodule.cc:31:10: fatal error: 'snappy-c.h' file not found

include <snappy-c.h>

1 error generated.

I installed ant, maven and gfortran as suggested for Mavericks to no avail.

Installation of snappy into venv fails

Hi Russel,

I bought your book, and have given it quick read through about half way so far. I'm dying to analyze my mailbox, but the 'install -r requirements.txt' command fails while installing snappy (CentOS 6.5).

Any ideas? Thank you, Robert.

copying snappy.py -> build/lib.linux-x86_64-2.7
running build_ext
building '_snappy' extension
creating build/temp.linux-x86_64-2.7
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/ads/tools/python/2.7.7/include/python2.7 -c snappymodule.cc -o build/temp.linux-x86_64-2.7/snappymodule.o
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
snappymodule.cc:31:22: error: snappy-c.h: No such file or directory
snappymodule.cc:77: error: âsnappy_statusâ was not declared in this scope
snappymodule.cc:78: error: expected â,â or â;â before â{â token
snappymodule.cc:45: warning: â_stateâ defined but not used
snappymodule.cc:56: warning: âSnappyCompressErrorâ defined but not used
snappymodule.cc:57: warning: âSnappyUncompressErrorâ defined but not used
snappymodule.cc:58: warning: âSnappyInvalidCompressedInputErrorâ defined but not used
snappymodule.cc:59: warning: âSnappyCompressedLengthErrorâ defined but not used
snappymodule.cc:77: warning: âsnappy_strerrorâ defined but not used
error: command 'gcc' failed with exit status 1

Cleaning up...
Command /home/rstober/Agile_Data_Code/venv/bin/python2.7 -c "import setuptools, tokenize;file='/home/rstober/Agile_Data_Code/venv/build/python-snappy/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-HFIM2_-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/rstober/Agile_Data_Code/venv/include/site/python2.7 failed with error code 1 in /home/rstober/Agile_Data_Code/venv/build/python-snappy

ch07: Readme

replace
pig -l /tmp -x local -v -w publish_topics.pig
with
pig -l /tmp -x local -v -w publish_topics_per_email.pig

Suggestions for ch03/python/test_avro.py

Hello,
On page 41 in the O'Reilly book, the file (test_avro.py) given as Example 3-2 has the same name as the file to write avros in Python given as Example 3-1.

What is more, the OUTFILE_NAME intialisation is missing. Here is the script (I named it test_reading_avro.py) with the suggested correction:

from avro import schema, datafile, io
import pprint
# Test reading avros
rec_reader = io.DatumReader()

# Create a 'data file' (avro file) reader
OUTFILE_NAME = '/tmp/messages.avro'
df_reader = datafile.DataFileReader( open(OUTFILE_NAME), rec_reader)

# Read all records stored inside
pp = pprint.PrettyPrinter()
for record in df_reader:
    pp.pprint(record)

What is more, I run those scripts on Mac OS X, but they won't work on Microsoft Windows which has no /tmp directory.

Jean-Baptiste

Moved to mongo-hadoop

Timeout exeption message while running gmail.py

Hello,
I have a problem when I execute the gmail.py script (ch03/gmail/gmail.py): the email collection does not occur and the "Timeout exception occured!" message is displayed (tested on Mac OS X 10.6 and Linux Debian in a virtual machine running Windows).

Note that the timeout error is displayed even when running the script without specifying any parameters (to display the usage message).
Jean-Baptiste

[ch06] - nvd3 display is wrong

The original source code lead to a no-display of the hours on the bottom of the graphs.

ch07/pig/topics.pig - varaha error

Was executing the above using the command: pig -l /tmp -x local -v -w topics.pig. This was executed within the virtual environment. However, I'm getting an error of "could not instantiate 'varaha.text.TokenizeText' with arguments 'null'".

Pretty sure that varaha has already been compiled and can be located as per topics.pig. Not sure what else to look for.

Please advise. Thanks.

UnicodeDecodeError running test_avro.py on Windows

Hello,
I have a UnicodeDecodeError running test_avro.py on WIndows 7. Here is a copy of my Windows terminal, thank you for you help:

C:\Users\me\Documents\Agile_Data>python test_avro.py
Traceback (most recent call last):
  File "test_avro.py", line 50, in <module>
    for record in df_reader:
  File "c:\Python27\lib\site-packages\avro\datafile.py", line 362, in next
    datum = self.datum_reader.read(self.datum_decoder)
  File "c:\Python27\lib\site-packages\avro\io.py", line 445, in read
    return self.read_data(self.writers_schema, self.readers_schema, decoder)
  File "c:\Python27\lib\site-packages\avro\io.py", line 490, in read_data
    return self.read_record(writers_schema, readers_schema, decoder)
  File "c:\Python27\lib\site-packages\avro\io.py", line 690, in read_record
    field_val = self.read_data(field.type, readers_field.type, decoder)
  File "c:\Python27\lib\site-packages\avro\io.py", line 468, in read_data
    return decoder.read_utf8()
  File "c:\Python27\lib\site-packages\avro\io.py", line 233, in read_utf8
    return unicode(self.read_bytes(), "utf-8")
UnicodeDecodeError: 'utf8' codec can't decode byte 0x9c in position 0: invalid start byte

Jean-Baptiste

Ant Build Fails for Pig 0.13.0

Hi,

I am running OSX Mavericks 10.9.4 with Macports.
...

|                  |            modules            ||   artifacts   |
|       conf       | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
|      compile     |  109  |   21  |   0   |   17  ||   95  |   0   |
---------------------------------------------------------------------

[ivy:resolve]
[ivy:resolve] :: problems summary ::
[ivy:resolve] :::: WARNINGS
[ivy:resolve] [FAILED ] org.mortbay.jetty#jetty;6.1.26!jetty.zip: (0ms)
[ivy:resolve] ==== fs: tried
[ivy:resolve] /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip
[ivy:resolve] ==== maven2: tried
[ivy:resolve] http://repo2.maven.org/maven2/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip
[ivy:resolve] ==== jboss-maven2: tried
[ivy:resolve] http://repository.jboss.com/nexus/content/groups/public/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip
[ivy:resolve] ==== apache-snapshots: tried
[ivy:resolve] http://repository.apache.org/content/groups/snapshots-group/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] :: FAILED DOWNLOADS ::
[ivy:resolve] :: ^ see resolution messages for details ^ ::
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] :: org.mortbay.jetty#jetty;6.1.26!jetty.zip
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve]
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

BUILD FAILED
/Users/davidlaxer/pig-0.13.0/build.xml:1599: impossible to resolve dependencies:
resolve failed - see output for details

The .zip file jetty-6.1.26.zip can't be found.

However, the .jar files exist in my file system.

Total time: 23 seconds
David-Laxers-MacBook-Pro:pig-0.13.0 davidlaxer$ ls -l /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty/6.1.26
total 1104
-rw-r--r-- 1 davidlaxer staff 198 May 22 18:03 _remote.repositories
-rw-r--r-- 1 davidlaxer staff 539912 May 22 18:03 jetty-6.1.26.jar
-rw-r--r-- 1 davidlaxer staff 505 May 22 18:03 jetty-6.1.26.jar.lastUpdated
-rw-r--r-- 1 davidlaxer staff 40 May 22 18:03 jetty-6.1.26.jar.sha1
-rw-r--r-- 1 davidlaxer staff 425 May 22 18:03 jetty-6.1.26.pom
-rw-r--r-- 1 davidlaxer staff 376 May 22 18:03 jetty-6.1.26.pom.lastUpdated
-rw-r--r-- 1 davidlaxer staff 40 May 22 18:03 jetty-6.1.26.pom.sha1
David-Laxers-MacBook-Pro:pig-0.13.0 davidlaxer$

This StackOverflow seems relevant:
http://stackoverflow.com/questions/24869877/ant-build-fails-for-pig-0-13-0-for-hadoop-2

David-Laxers-MacBook-Pro:pig-0.13.0 davidlaxer$ cd contrib/piggybank/java/
David-Laxers-MacBook-Pro:java davidlaxer$ ant
Buildfile: /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/build.xml

init:

compile:
[echo] *** Compiling Pig UDFs ***
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/build.xml:94: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 167 source files to /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/build/classes
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:23: cannot find symbol
[javac] symbol : class Accumulator
[javac] location: package org.apache.pig
[javac] import org.apache.pig.Accumulator;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:24: cannot find symbol
[javac] symbol : class Algebraic
[javac] location: package org.apache.pig
[javac] import org.apache.pig.Algebraic;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:25: cannot find symbol
[javac] symbol : class EvalFunc
[javac] location: package org.apache.pig
[javac] import org.apache.pig.EvalFunc;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:26: cannot find symbol
[javac] symbol : class PigException
[javac] location: package org.apache.pig
[javac] import org.apache.pig.PigException;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:27: package org.apache.pig.backend.executionengine does not exist
[javac] import org.apache.pig.backend.executionengine.ExecException;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:28: package org.apache.pig.backend.hadoop.executionengine.physicalLayer does not exist
[javac] import org.apache.pig.backend.hadoop.executionengine.physicalLayer.PigProgressable;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:29: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.BagFactory;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:30: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.DataBag;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:31: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.Tuple;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:32: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.TupleFactory;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:33: package org.apache.pig.impl.logicalLayer.schema does not exist
[javac] import org.apache.pig.impl.logicalLayer.schema.Schema;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:118: cannot find symbol
[javac] symbol: class EvalFunc
[javac] public class ExtremalTupleByNthField extends EvalFunc implements
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:118: cannot find symbol
[javac] symbol: class Tuple
[javac] public class ExtremalTupleByNthField extends EvalFunc implements
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:119: cannot find symbol
[javac] symbol: class Algebraic
[javac] Algebraic, Accumulator {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:119: cannot find symbol
[javac] symbol: class Accumulator
[javac] Algebraic, Accumulator {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:119: cannot find symbol
[javac] symbol: class Tuple
[javac] Algebraic, Accumulator {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:119: interface expected here
[javac] Algebraic, Accumulator {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:142: cannot find symbol
[javac] symbol : class ExecException
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] public ExtremalTupleByNthField() throws ExecException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:148: cannot find symbol
[javac] symbol : class ExecException
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] throws ExecException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:153: cannot find symbol
[javac] symbol : class ExecException
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] throws ExecException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:168: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] public Tuple exec(Tuple input) throws IOException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:168: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] public Tuple exec(Tuple input) throws IOException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:177: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:177: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:212: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] Tuple intermediate = null;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:213: cannot find symbol
[javac] symbol : class DataBag
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] DataBag tempDb = BagFactory.getInstance().newDefaultBag();
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:214: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] Tuple parameterToExtreme = TupleFactory.getInstance().newTuple(tempDb);
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:217: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] public void accumulate(Tuple b) throws IOException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:253: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] public Tuple getValue() {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:290: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] protected final static Tuple extreme(int pind, int psign, Tuple input,
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:291: cannot find symbol
[javac] symbol : class PigProgressable
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] PigProgressable reporter) throws ExecException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:290: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] protected final static Tuple extreme(int pind, int psign, Tuple input,
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:291: cannot find symbol
[javac] symbol : class ExecException
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] PigProgressable reporter) throws ExecException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:351: cannot find symbol
[javac] symbol : class ExecException
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] throws ExecException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:265: cannot find symbol
[javac] symbol : class EvalFunc
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] public static final class HelperClass extends EvalFunc {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:265: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField
[javac] public static final class HelperClass extends EvalFunc {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:268: cannot find symbol
[javac] symbol : class ExecException
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField.HelperClass
[javac] public HelperClass() throws ExecException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:272: cannot find symbol
[javac] symbol : class ExecException
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField.HelperClass
[javac] public HelperClass(String fieldIndexString) throws ExecException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:277: cannot find symbol
[javac] symbol : class ExecException
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField.HelperClass
[javac] throws ExecException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:283: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField.HelperClass
[javac] public Tuple exec(Tuple input) throws IOException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/ExtremalTupleByNthField.java:283: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.ExtremalTupleByNthField.HelperClass
[javac] public Tuple exec(Tuple input) throws IOException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsDouble.java:23: cannot find symbol
[javac] symbol : class EvalFunc
[javac] location: package org.apache.pig
[javac] import org.apache.pig.EvalFunc;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsDouble.java:24: cannot find symbol
[javac] symbol : class PigWarning
[javac] location: package org.apache.pig
[javac] import org.apache.pig.PigWarning;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsDouble.java:25: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.DataType;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsDouble.java:26: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.Tuple;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsDouble.java:27: package org.apache.pig.impl.logicalLayer.schema does not exist
[javac] import org.apache.pig.impl.logicalLayer.schema.Schema;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsDouble.java:36: cannot find symbol
[javac] symbol: class EvalFunc
[javac] public class IsDouble extends EvalFunc {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsDouble.java:38: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.IsDouble
[javac] public Boolean exec(Tuple input) throws IOException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsDouble.java:56: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.IsDouble
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsDouble.java:56: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.IsDouble
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsFloat.java:23: cannot find symbol
[javac] symbol : class EvalFunc
[javac] location: package org.apache.pig
[javac] import org.apache.pig.EvalFunc;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsFloat.java:24: cannot find symbol
[javac] symbol : class PigWarning
[javac] location: package org.apache.pig
[javac] import org.apache.pig.PigWarning;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsFloat.java:25: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.DataType;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsFloat.java:26: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.Tuple;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsFloat.java:27: package org.apache.pig.impl.logicalLayer.schema does not exist
[javac] import org.apache.pig.impl.logicalLayer.schema.Schema;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsFloat.java:36: cannot find symbol
[javac] symbol: class EvalFunc
[javac] public class IsFloat extends EvalFunc {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsFloat.java:38: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.IsFloat
[javac] public Boolean exec(Tuple input) throws IOException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsFloat.java:56: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.IsFloat
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsFloat.java:56: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.IsFloat
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:23: cannot find symbol
[javac] symbol : class EvalFunc
[javac] location: package org.apache.pig
[javac] import org.apache.pig.EvalFunc;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:24: cannot find symbol
[javac] symbol : class PigWarning
[javac] location: package org.apache.pig
[javac] import org.apache.pig.PigWarning;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:25: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.DataType;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:26: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.Tuple;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:27: package org.apache.pig.impl.logicalLayer.schema does not exist
[javac] import org.apache.pig.impl.logicalLayer.schema.Schema;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:36: cannot find symbol
[javac] symbol: class EvalFunc
[javac] public class IsInt extends EvalFunc {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:38: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.IsInt
[javac] public Boolean exec(Tuple input) throws IOException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:56: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.IsInt
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsInt.java:56: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.IsInt
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsLong.java:23: cannot find symbol
[javac] symbol : class EvalFunc
[javac] location: package org.apache.pig
[javac] import org.apache.pig.EvalFunc;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsLong.java:24: cannot find symbol
[javac] symbol : class PigWarning
[javac] location: package org.apache.pig
[javac] import org.apache.pig.PigWarning;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsLong.java:25: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.DataType;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsLong.java:26: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.Tuple;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsLong.java:27: package org.apache.pig.impl.logicalLayer.schema does not exist
[javac] import org.apache.pig.impl.logicalLayer.schema.Schema;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsLong.java:36: cannot find symbol
[javac] symbol: class EvalFunc
[javac] public class IsLong extends EvalFunc {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsLong.java:38: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.IsLong
[javac] public Boolean exec(Tuple input) throws IOException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsLong.java:56: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.IsLong
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsLong.java:56: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.IsLong
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsNumeric.java:23: cannot find symbol
[javac] symbol : class EvalFunc
[javac] location: package org.apache.pig
[javac] import org.apache.pig.EvalFunc;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsNumeric.java:24: cannot find symbol
[javac] symbol : class PigWarning
[javac] location: package org.apache.pig
[javac] import org.apache.pig.PigWarning;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsNumeric.java:25: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.DataType;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsNumeric.java:26: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.Tuple;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsNumeric.java:27: package org.apache.pig.impl.logicalLayer.schema does not exist
[javac] import org.apache.pig.impl.logicalLayer.schema.Schema;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsNumeric.java:46: cannot find symbol
[javac] symbol: class EvalFunc
[javac] public class IsNumeric extends EvalFunc {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsNumeric.java:49: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.IsNumeric
[javac] public Boolean exec(Tuple input) throws IOException {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsNumeric.java:71: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.IsNumeric
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/IsNumeric.java:71: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.IsNumeric
[javac] public Schema outputSchema(Schema input) {
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:24: cannot find symbol
[javac] symbol : class Algebraic
[javac] location: package org.apache.pig
[javac] import org.apache.pig.Algebraic;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:25: cannot find symbol
[javac] symbol : class EvalFunc
[javac] location: package org.apache.pig
[javac] import org.apache.pig.EvalFunc;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:26: package org.apache.pig.backend.executionengine does not exist
[javac] import org.apache.pig.backend.executionengine.ExecException;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:27: package org.apache.pig.backend.hadoop.executionengine.physicalLayer does not exist
[javac] import org.apache.pig.backend.hadoop.executionengine.physicalLayer.PigProgressable;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:28: package org.apache.pig.data does not exist
[javac] import org.apache.pig.data.*;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:29: package org.apache.pig.impl.logicalLayer.schema does not exist
[javac] import org.apache.pig.impl.logicalLayer.schema.Schema;
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:46: cannot find symbol
[javac] symbol: class EvalFunc
[javac] public class MaxTupleBy1stField extends EvalFunc implements Algebraic
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:46: cannot find symbol
[javac] symbol: class Tuple
[javac] public class MaxTupleBy1stField extends EvalFunc implements Algebraic
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:46: cannot find symbol
[javac] symbol: class Algebraic
[javac] public class MaxTupleBy1stField extends EvalFunc implements Algebraic
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:76: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.MaxTupleBy1stField
[javac] public Schema outputSchema(Schema input)
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:76: cannot find symbol
[javac] symbol : class Schema
[javac] location: class org.apache.pig.piggybank.evaluation.MaxTupleBy1stField
[javac] public Schema outputSchema(Schema input)
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:118: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.MaxTupleBy1stField
[javac] public Tuple exec(Tuple input) throws IOException
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:118: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.MaxTupleBy1stField
[javac] public Tuple exec(Tuple input) throws IOException
[javac] ^
[javac] /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/evaluation/MaxTupleBy1stField.java:131: cannot find symbol
[javac] symbol : class Tuple
[javac] location: class org.apache.pig.piggybank.evaluation.MaxTupleBy1stField
[javac] protected static Tuple max(Tuple input, PigProgressable reporter) throws ExecException
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/HiveColumnarStorage.java uses unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 100 errors

BUILD FAILED
/Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/build.xml:94: Compile failed; see the compiler error output for details.

Total time: 7 seconds

ch08/web/index.py

For some weird reason, I'm not able to get thru in the web page to click and show the page with the chart showing the probabilities. Am getting error IndexError: Out of bound when the function self.smoothed[idx] is being evaluated. I'm dumping the screen of WSGI when it happen along with the variable and line of suspect. Please advise. Thanks.

GNU Fortran is now provided as part of GCC, and can be installed with: brew install gcc

ERROR 2998: Unhandled internal error. Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected

Hi,

I'm getting errors running this simple Pig test:

David-Laxers-MacBook-Pro:pig davidlaxer$ cat test.pig
/* Set Home Directory - where we install software */
%default HOME echo \$HOME

REGISTER /Users/davidlaxer/pig-0.13.0/build/ivy/lib/Pig/avro-1.7.5.jar
REGISTER /Users/davidlaxer/pig-0.13.0/build/ivy/lib/Pig/json-simple-1.1.jar
REGISTER /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/piggybank.jar

/* DEFINE AvroStorage org.apache.pig.piggybank.storage.avro.AvroStorage();*/

/* Load the emails in avro format (edit the path to match where you saved them) using the AvroStorage UDF from Piggybank */
messages = LOAD '/tmp/test_mbox' USING org.apache.pig.piggybank.storage.avro.AvroStorage();

DESCRIBE messages;
EXPLAIN messages;
ILLUSTRATE messages;
lmt = LIMIT messages 100;
dump messages;

STORE messages INTO '/tmp/messages' USING org.apache.pig.piggybank.storage.avro.AvroStorage();

Mac OS X 10.9.5

$ java -version
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)

$ pig -version
Apache Pig version 0.13.0 (r1606446)
compiled Jun 29 2014, 02:27:58

$ virtualenv --version
1.11.6
(virtualenv)David-Laxers-MacBook-Pro:pig davidlaxer$ env | grep PIG
PIG_HOME=/Users/davidlaxer/pig-0.13.0
PIG_CLASSPATH=/users/davidlaxer/hadoop-2.3.0-src/src/conf
(virtualenv)David-Laxers-MacBook-Pro:pig davidlaxer$ env | grep HADOOP
HADOOP_HOME=/Users/davidlaxer/hadoop-2.3.0-src
HADOOP_CONF_DIR=/Users/davidlaxer/hadoop-2.3.0-src/src/conf

$ head /tmp/test_mbox/part-1.avro
Objavro.schema?{"fields": [{"doc": "", "type": ["null", "string"], "name": "message_id"}, {"doc": "", "type": ["null", "string"], "name": "thread_id"}, {"type": ["string", "null"], "name": "in_reply_to"}, {"type": ["string", "null"], "name": "subject"}, {"type": ["string", "null"], "name": "body"}, {"type": ["string", "null"], "name": "date"}, {"type": {"fields": [{"doc": "", "type": ["null", "string"], "name": "real_name"}, {"doc": "", "type": ["null", "string"], "name": "address"}], "type": "record", "name": "from"}, "name": "from"}, {"doc": "", "type": ["null", {"items": ["null", {"fields": [{"doc": "", "type": ["null", "string"], "name": "real_name"}, {"doc": "", "type": ["null", "string"], "name": "address"}], "type": "record", "name": "to"}], "type": "array"}], "name": "tos"}, {"doc": "", "type": ["null", {"items": ["null", {"fields": [{"doc": "", "type": ["null", "string"], "name": "real_name"}, {"doc": "", "type": ["null", "string"], "name": "address"}], "type": "record", "name": "cc"}], "type": "array"}], "name": "ccs"}, {"doc": "", "type": ["null", {"items": ["null", {"fields": [{"doc": "", "type": ["null", "string"], "name": "real_name"}, {"doc": "", "type": ["null", "string"], "name": "address"}], "type": "record", "name": "bcc"}], "type": "array"}], "name": "bccs"}, {"doc": "", "type": ["null", {"items": ["null", {"fields": [{"doc": "", "type": ["null", "string"], "name": "real_name"}, {"doc": "", "type": ["null", "string"], "name": "address"}], "type": "record", "name": "reply_to"}], "type": "array"}], "name": "reply_tos"}], "type": "record", "name": "Email"}avro.codenullZB?;?ԑ???LY????CAG+1DhTqe94W5mof4ZYpM5XCMK8nRazcC7L6H4ySt8s4GQL4Hw@mail.gmail.com&1480877866094699206b11ae7ec2-c20f-4dcf-a3fc-a9190cc3415b@continuum.io?Re: [Anaconda Support] ipython notebook fails to launch in Anaconda 2.1?rHi, I got a similar problem with ipython notebook. But once I defined
PYTHONPATTH point to the site-packages. All work well. In the past, it
sounds no need to define PYTHONPATH, so maybe something changed.

BTW, I installed miniconda for python 3.4.1, and ipython, ipython-notebook.

Hope it helps,

Wanli Wu

...

(virtualenv)David-Laxers-MacBook-Pro:pig davidlaxer$ !p
pig -l /tmp -x local -w -v test.pig
2014-10-10 01:20:26,141 INFO [main] pig.ExecTypeProvider (ExecTypeProvider.java:selectExecType(41)) - Trying ExecType : LOCAL
2014-10-10 01:20:26,145 INFO [main] pig.ExecTypeProvider (ExecTypeProvider.java:selectExecType(43)) - Picked LOCAL as the ExecType
2014-10-10 01:20:26,265 [main] INFO org.apache.pig.Main - Apache Pig version 0.13.0 (r1606446) compiled Jun 29 2014, 02:27:58
2014-10-10 01:20:26,265 [main] INFO org.apache.pig.Main - Logging error messages to: /private/tmp/pig_1412922026132.log
2014-10-10 01:20:26.727 java[80092:1003] Unable to load realm info from SCDynamicStore
2014-10-10 01:20:26,735 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-10-10 01:20:27,409 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /Users/davidlaxer/.pigbootup not found
2014-10-10 01:20:27,441 [main] INFO org.apache.pig.tools.parameters.PreprocessorContext - Executing command : echo $HOME
2014-10-10 01:20:27,611 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 01:20:27,611 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-10-10 01:20:27,612 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-10-10 01:20:27,934 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 01:20:28,114 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 01:20:28,258 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 01:20:31,764 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
messages: {message_id: chararray,thread_id: chararray,in_reply_to: chararray,subject: chararray,body: chararray,date: chararray,from: (real_name: chararray,address: chararray),tos: {ARRAY_ELEM: (real_name: chararray,address: chararray)},ccs: {ARRAY_ELEM: (real_name: chararray,address: chararray)},bccs: {ARRAY_ELEM: (real_name: chararray,address: chararray)},reply_tos: {ARRAY_ELEM: (real_name: chararray,address: chararray)}}
2014-10-10 01:20:32,830 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}

-----------------------------------------------

New Logical Plan:

-----------------------------------------------

messages: (Name: LOStore Schema: message_id#26:chararray,thread_id#27:chararray,in_reply_to#28:chararray,subject#29:chararray,body#30:chararray,date#31:chararray,from#32:tuple(real_name#33:chararray,address#34:chararray),tos#35:bag{ARRAY_ELEM#36:tuple(real_name#37:chararray,address#38:chararray)},ccs#39:bag{ARRAY_ELEM#40:tuple(real_name#41:chararray,address#42:chararray)},bccs#43:bag{ARRAY_ELEM#44:tuple(real_name#45:chararray,address#46:chararray)},reply_tos#47:bag{ARRAY_ELEM#48:tuple(real_name#49:chararray,address#50:chararray)})
|
|---messages: (Name: LOLoad Schema: message_id#26:chararray,thread_id#27:chararray,in_reply_to#28:chararray,subject#29:chararray,body#30:chararray,date#31:chararray,from#32:tuple(real_name#33:chararray,address#34:chararray),tos#35:bag{ARRAY_ELEM#36:tuple(real_name#37:chararray,address#38:chararray)},ccs#39:bag{ARRAY_ELEM#40:tuple(real_name#41:chararray,address#42:chararray)},bccs#43:bag{ARRAY_ELEM#44:tuple(real_name#45:chararray,address#46:chararray)},reply_tos#47:bag{ARRAY_ELEM#48:tuple(real_name#49:chararray,address#50:chararray)})RequiredFields:null

-----------------------------------------------

Physical Plan:

-----------------------------------------------

messages: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-1
|
|---messages: Load(/tmp/test_mbox:org.apache.pig.piggybank.storage.avro.AvroStorage) - scope-0

--------------------------------------------------

Map Reduce Plan

--------------------------------------------------

No MR jobs. Fetch only.
2014-10-10 01:20:33,366 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 01:20:33,368 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-10-10 01:20:33,443 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[LoadTypeCastInserter, StreamTypeCastInserter], RULES_DISABLED=[AddForEach, ColumnMapKeyPrune, FilterLogicExpressionSimplifier, GroupByConstParallelSetter, LimitOptimizer, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter]}
2014-10-10 01:20:33,575 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-10-10 01:20:33,624 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-10-10 01:20:33,625 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-10-10 01:20:33,683 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job
2014-10-10 01:20:33,689 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
2014-10-10 01:20:33,689 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-10-10 01:20:33,690 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2014-10-10 01:20:34,210 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map - Aliases being processed per job phase (AliasName[line,offset]): M: messages[11,11] C: R:
2014-10-10 01:20:34,226 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 01:20:34,227 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
2014-10-10 01:20:34,310 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-10-10 01:20:34,316 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
2014-10-10 01:20:34,320 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at org.apache.pig.piggybank.storage.avro.PigAvroInputFormat.listStatus(PigAvroInputFormat.java:96)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:375)
at org.apache.pig.impl.io.ReadToEndLoader.init(ReadToEndLoader.java:190)
at org.apache.pig.impl.io.ReadToEndLoader.(ReadToEndLoader.java:146)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad.setUp(POLoad.java:95)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLoad.getNextTuple(POLoad.java:123)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:277)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.pig.pen.LocalMapReduceSimulator.launchPig(LocalMapReduceSimulator.java:202)
at org.apache.pig.pen.ExampleGenerator.getData(ExampleGenerator.java:259)
at org.apache.pig.pen.ExampleGenerator.readBaseData(ExampleGenerator.java:223)
at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:155)
at org.apache.pig.PigServer.getExamples(PigServer.java:1282)
at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:810)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.Illustrate(PigScriptParser.java:802)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:381)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:228)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:608)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Details also at logfile: /private/tmp/pig_1412922026132.log
(virtualenv)David-Laxers-MacBook-Pro:pig davidlaxer$ pig -x local
2014-10-10 08:49:33,765 INFO [main] pig.ExecTypeProvider (ExecTypeProvider.java:selectExecType(41)) - Trying ExecType : LOCAL
2014-10-10 08:49:33,794 INFO [main] pig.ExecTypeProvider (ExecTypeProvider.java:selectExecType(43)) - Picked LOCAL as the ExecType
2014-10-10 08:49:33,866 [main] INFO org.apache.pig.Main - Apache Pig version 0.13.0 (r1606446) compiled Jun 29 2014, 02:27:58
2014-10-10 08:49:33,866 [main] INFO org.apache.pig.Main - Logging error messages to: /Users/davidlaxer/Agile_Data_Code/ch03/pig/pig_1412948973860.log
2014-10-10 08:49:33,897 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /Users/davidlaxer/.pigbootup not found
2014-10-10 08:49:34,217 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 08:49:34,221 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-10-10 08:49:34,223 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-10-10 08:49:34.824 java[83134:1003] Unable to load realm info from SCDynamicStore
2014-10-10 08:49:34,844 [main] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-10-10 08:49:35,111 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-10-10 08:49:35,117 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
grunt> REGISTER /Users/davidlaxer/pig-0.13.0/build/ivy/lib/Pig/avro-1.7.5.jar
2014-10-10 08:49:58,340 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 08:49:58,342 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
grunt> REGISTER /Users/davidlaxer/pig-0.13.0/build/ivy/lib/Pig/json-simple-1.1.jar
2014-10-10 08:49:58,562 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 08:49:58,565 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
grunt> REGISTER /Users/davidlaxer/pig-0.13.0/contrib/piggybank/java/piggybank.jar
2014-10-10 08:49:58,745 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 08:49:58,754 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
grunt>
grunt> /* DEFINE AvroStorage org.apache.pig.piggybank.storage.avro.AvroStorage();/
grunt>
grunt> / Load the emails in avro format (edit the path to match where you saved them) using the AvroStorage UDF from Piggybank */
grunt> messages = LOAD '/tmp/test_mbox' USING org.apache.pig.piggybank.storage.avro.AvroStorage();
2014-10-10 08:49:59,251 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 08:49:59,253 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
grunt> DESCRIBE messages;
messages: {message_id: chararray,thread_id: chararray,in_reply_to: chararray,subject: chararray,body: chararray,date: chararray,from: (real_name: chararray,address: chararray),tos: {ARRAY_ELEM: (real_name: chararray,address: chararray)},ccs: {ARRAY_ELEM: (real_name: chararray,address: chararray)},bccs: {ARRAY_ELEM: (real_name: chararray,address: chararray)},reply_tos: {ARRAY_ELEM: (real_name: chararray,address: chararray)}}
grunt> EXPLAIN messages;
2014-10-10 08:50:22,370 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}

-----------------------------------------------

New Logical Plan:

-----------------------------------------------

messages: (Name: LOStore Schema: message_id#51:chararray,thread_id#52:chararray,in_reply_to#53:chararray,subject#54:chararray,body#55:chararray,date#56:chararray,from#57:tuple(real_name#58:chararray,address#59:chararray),tos#60:bag{ARRAY_ELEM#61:tuple(real_name#62:chararray,address#63:chararray)},ccs#64:bag{ARRAY_ELEM#65:tuple(real_name#66:chararray,address#67:chararray)},bccs#68:bag{ARRAY_ELEM#69:tuple(real_name#70:chararray,address#71:chararray)},reply_tos#72:bag{ARRAY_ELEM#73:tuple(real_name#74:chararray,address#75:chararray)})
|
|---messages: (Name: LOLoad Schema: message_id#51:chararray,thread_id#52:chararray,in_reply_to#53:chararray,subject#54:chararray,body#55:chararray,date#56:chararray,from#57:tuple(real_name#58:chararray,address#59:chararray),tos#60:bag{ARRAY_ELEM#61:tuple(real_name#62:chararray,address#63:chararray)},ccs#64:bag{ARRAY_ELEM#65:tuple(real_name#66:chararray,address#67:chararray)},bccs#68:bag{ARRAY_ELEM#69:tuple(real_name#70:chararray,address#71:chararray)},reply_tos#72:bag{ARRAY_ELEM#73:tuple(real_name#74:chararray,address#75:chararray)})RequiredFields:null

-----------------------------------------------

Physical Plan:

-----------------------------------------------

messages: Store(fakefile:org.apache.pig.builtin.PigStorage) - scope-1
|
|---messages: Load(/tmp/test_mbox:org.apache.pig.piggybank.storage.avro.AvroStorage) - scope-0

--------------------------------------------------

Map Reduce Plan

--------------------------------------------------

No MR jobs. Fetch only.
grunt> ILLUSTRATE messages;
2014-10-10 08:50:22,620 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-10-10 08:50:22,622 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 08:50:22,641 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-10-10 08:50:22,710 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[LoadTypeCastInserter, StreamTypeCastInserter], RULES_DISABLED=[AddForEach, ColumnMapKeyPrune, FilterLogicExpressionSimplifier, GroupByConstParallelSetter, LimitOptimizer, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter]}
2014-10-10 08:50:22,793 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-10-10 08:50:22,812 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-10-10 08:50:22,812 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-10-10 08:50:22,846 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job
2014-10-10 08:50:22,854 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
2014-10-10 08:50:22,883 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-10-10 08:50:22,883 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2014-10-10 08:50:23,137 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map - Aliases being processed per job phase (AliasName[line,offset]): M: messages[1,11] C: R:
2014-10-10 08:50:23,139 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
2014-10-10 08:50:23,140 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-10-10 08:50:23,141 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
2014-10-10 08:50:23,180 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-10-10 08:50:23,184 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
Details at logfile: /Users/davidlaxer/Agile_Data_Code/ch03/pig/pig_1412948973860.log
grunt> lmt = LIMIT messages 100;
grunt> dump messages;
2014-10-10 08:50:23,309 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2014-10-10 08:50:23,320 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[LoadTypeCastInserter, StreamTypeCastInserter], RULES_DISABLED=[AddForEach, ColumnMapKeyPrune, FilterLogicExpressionSimplifier, GroupByConstParallelSetter, LimitOptimizer, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter]}
2014-10-10 08:50:23,372 [main] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized
2014-10-10 08:50:23,409 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2081: Unable to setup the load function.
Details at logfile: /Users/davidlaxer/Agile_Data_Code/ch03/pig/pig_1412948973860.log
grunt>

error (imaplib.error) occurs when try to run gmail.py

Follow instruction in chapter 3 to run $ ./gmail.py -m automatic -u [email protected] -p 'xxxxxx' -s ./email.avro.schema -f '[Gmail]/All Mail' -o /tmp/my_inbox_directory 2>&1 &

To get data from gmail, here is the error:

$ Timeout exception occurred!
Warning: /tmp/my_inbox_directory already exists:
Warning: /tmp/my_inbox_directorytmp already exists:
Traceback (most recent call last):
File "./gmail.py", line 104, in
main()
File "./gmail.py", line 85, in main
status, count = slurper.init_folder(imap_folder)
File "/home/james/training/ADS/gmail/gmail_slurper.py", line 68, in init_folder
status, count = self.imap.select(folder)
File "/usr/lib/python2.7/imaplib.py", line 649, in select
typ, dat = self._simple_command(name, mailbox)
File "/usr/lib/python2.7/imaplib.py", line 1070, in _simple_command
return self._command_complete(name, self._command(name, *args))
File "/usr/lib/python2.7/imaplib.py", line 825, in _command
', '.join(Commands[name])))
imaplib.error: command SELECT illegal in state NONAUTH, only allowed in states AUTH, SELECTED

any idea?

Ch03 elasticsearch.pig fails with ElasticSearch 0.90.0

It works up until the end when it tries to query for data, for which I receive:

{
  "error" : "IndexMissingException[[sent_counts] missing]",
  "status" : 404
}

ElasticSearch was running in another tab, and the operations to add data to it seem to have worked. The book specifies elasticsearch 0.20.2, I have ElasticSearch Version: 0.90.0, which might be part of it.

Anyone have a workaround?

Issue getting sent_counts.pig to work

This seems like a trivial enough issue, but stuck for 2 days...

~/work/agile_data_science/Agile_Data_Code/ch03/pig$ pig -l /tmp -x local -v -w sent_counts.pig
2015-06-27 12:50:31,890 [main] INFO  org.apache.pig.Main - Apache Pig version 0.11.2-SNAPSHOT (rexported) compiled Jun 27 2015, 12:17:04
2015-06-27 12:50:31,891 [main] INFO  org.apache.pig.Main - Logging error messages to: /tmp/pig_1435389631827.log
2015-06-27 12:50:33,652 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /home/cisadmin/.pigbootup not found
2015-06-27 12:50:33,687 [main] INFO  org.apache.pig.tools.parameters.PreprocessorContext - Executing command : echo $HOME/Software/
2015-06-27 12:50:33,928 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2015-06-27 12:50:35,252 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1025:
<file sent_counts.pig, line 18, column 31> Invalid field projection. Projected field [from] does not exist.
2015-06-27 12:50:35,253 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.plan.PlanValidationException: ERROR 1025:
<file sent_counts.pig, line 18, column 31> Invalid field projection. Projected field [from] does not exist.
        at org.apache.pig.newplan.logical.expression.ProjectExpression.findColNum(ProjectExpression.java:183)
        at org.apache.pig.newplan.logical.expression.ProjectExpression.setColumnNumberFromAlias(ProjectExpression.java:166)
...

Done till now:

Followed all steps in ch03
Tested avro standalone
Pig is installed (used 0.11.1 itself for simplicity)
Tested avro is able to read my mail data file correctly:

(venv)~/work/agile_data_science$ python test_read_avro.py
{u'bccs': None,
 u'body': u"View this email in your browser\r\n[http://mailer.dzone.com/display.php?M=1636801&C=a8b08717c76733a0b595b9feb67f2798&S=5176&L=3&N=3141]\r\n\r\nDZone [http://mailer.dzone.com/link.php?M=1636801&N=5176&L=80&F=T]\r\n\r\n\r\nFREE RESEARCH GUIDE\r\nDZone Guide to Mobile Development - 2015 EDITION\r\n\r\n\r\nDownload the Guide:\r\n[http://mailer.dzone.com/link.php?M=1636801&N=5176&L=23988&F=T]\r\n\r\n\r\nWHAT'S INSIDE?\r\n\r\n -- Get an expert view of the mobile development landscape: from\r\n     MBaaS to creating the best user experience\r\n\r\n -- Key findings from our survey to over 500 developers that sheds\r\n     light on mobile development trends and patterns\r\n\r\n -- Follow our steps in the Mobile App Testing Checklist for a clear\r\n     process to a successful app\r\n\r\n -- Review our list of the best MADPs and mobile web frameworks\r\n\r\n\r\nREAD YOUR FREE GUIDE\r\n[http://mailer.dzone.com/link.php?M=1636801&N=5176&L=23989&F=T]\r\n\r\n\r\n\r\n\r\nThis email is was sent to: [email protected].\r\nUnsubscribe:\r\nhttp://mailer.dzone.com/unsubscribe.php?M=1636801&C=a8b08717c76733a0b595b9feb67f2798&L=3&N=5176\r\n\r\nDZone 150 Preston Executive Dr, Cary, NC 27513\r\n\xa92015 DZone, Inc.\r\n",
 u'ccs': None,
 u'date': u'2015-06-24T10:15:31',
 u'from': {u'address': u'[email protected]', u'real_name': u'DZone'},
 u'in_reply_to': u'None',
 u'message_id': u'0000014e2510bddf-c8ec36c7-6b97-4d96-b9cb-263917629524-000000@email.amazonses.com',
 u'reply_tos': [{u'address': u'[email protected]', u'real_name': None}],
 u'subject': u'Gir, how are you overcoming mobile development pain points?',
 u'thread_id': u'1504854338257655678',
 u'tos': [{u'address': u'[email protected]', u'real_name': u'Gir Man'}]}
....

test_read_avro.py:

from avro import schema, datafile, io
import pprint

# OUTFILE_NAME = '/tmp/messages.avro'
OUTFILE_NAME = '/home/cisadmin/work/agile_data_science/mails/part-1.avro'

# Test reading avros
rec_reader = io.DatumReader()

# Create a 'data file' (avro file) reader
df_reader = datafile.DataFileReader(
  open(OUTFILE_NAME),
    rec_reader
    )

# Read all records stored inside
pp = pprint.PrettyPrinter()
for record in df_reader:
      pp.pprint(record)

sent_counts.pig:

/* Set Home Directory - where we install software */
%default HOME `echo \$HOME/Software/`

REGISTER $HOME/pig/build/ivy/lib/Pig/avro-1.5.3.jar
REGISTER $HOME/pig/build/ivy/lib/Pig/json-simple-1.1.jar
REGISTER $HOME/pig/contrib/piggybank/java/piggybank.jar

DEFINE AvroStorage org.apache.pig.piggybank.storage.avro.AvroStorage();

rmf /tmp/sent_counts.txt

/* Load the emails in avro format (edit the path to match where you saved them) using the AvroStorage UDF from Piggybank */
messages = LOAD '$HOME/work/agile_data_science/mails/part-1.avro' USING AvroStorage();
...

What could I be missing?

Ch06: sent_distributions.pig issues/errors - ERROR 2244: Job failed, hadoop does not return any error message

Hello,

I am doing the exercise on Ch06. Try to run the sent_distributions.pig in local mode (some of the error outputs are pasted below). The /tmp/sent_distributions.avro was never created nor pushed to mongodb. Anyone has any ideas or suggestions?

thanks,
James

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
1.0.0 0.12.0 james 2014-04-18 08:17:49 2014-04-18 08:17:59 GROUP_BY,FILTER,UNION

Some jobs have failed! Stop running all dependent jobs

Job Stats (time in seconds):
JobId Alias Feature Outputs
job_local_0001 emails,filtered,has_reply_to MULTI_QUERY,MAP_ONLY

Failed Jobs:
JobId Alias Feature Message Outputs
job_local_0002 1-2,all_froms,froms,pairs,reply_to,reply_to_froms,sent_times GROUP_BY,COMBINER Message: Job failed! Error - NA

Input(s):
Successfully read records from: "/tmp/my_inbox_directory"

Output(s):

Job DAG:
job_local_0001 -> job_local_0002,
job_local_0002 -> null,
null

2014-04-18 08:17:59,602 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Some jobs have failed! Stop running all dependent jobs
2014-04-18 08:17:59,605 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
2014-04-18 08:17:59,605 [main] ERROR org.apache.pig.tools.grunt.GruntParser - org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:148)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)

Details also at logfile: /tmp/pig_1397809064581.log

ch08: Readme

replace
pig -l /tmp -x local -v -w p_reply.pig
with
pig -l /tmp -x local -v -w p_reply_given_from_to.pig

ch09: p_reply_given_topics.pig (Path is not properly set)

HOME is not properly setup (similar to other chapters)
DEFINE TokenizeText varaha.text.TokenizeText('1', '1') is missing ;

ch09/p_reply_given_time_of_day.pig not used

Looking thru Chapter 9 in the book, I've not found how this pig file is utilised. Is there some missing codes or instructions I should perform to check this out?

ch04/pigs/sent_counts.pig

I finally managed to get AWS S3, EMR and DotCloud setup. I'm on pg 77 in the book to run the sent_counts.pig, albeit AWS style.

The following lines are what I've changed to:
/* Set Home Directory - where we install software Pig */
%default HOME '/home/hadoop/.versions/pig-0.11.1.1'

REGISTER $HOME/build/ivy/lib/Pig/avro-1.7.4.jar
REGISTER $HOME/build/ivy/lib/Pig/json-simple-1.1.jar
REGISTER $HOME/contrib/piggybank/java/piggybank.jar

When I ssh into the EMR site, i can't find avro or json-simple. Is there a separate location for them, or if not, how do I build from scratch, or upload them into AWS?

Pig Error: Pig 0.11.2/Hadoop 0.21.0

pig -l /tmp -x local -v -w sent_counts.pig
2014-08-13 22:26:16,744 [main] INFO org.apache.pig.Main - Apache Pig version 0.11.2-SNAPSHOT (rUnversioned directory) compiled Aug 13 2014, 21:53:01
2014-08-13 22:26:16,745 [main] INFO org.apache.pig.Main - Logging error messages to: /private/tmp/pig_1407990376721.log
2014-08-13 22:26:17,005 [main] INFO org.apache.hadoop.security.Groups - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
2014-08-13 22:26:17,222 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /Users/davidlaxer/.pigbootup not found
2014-08-13 22:26:17,271 [main] INFO org.apache.pig.tools.parameters.PreprocessorContext - Executing command : echo $HOME
2014-08-13 22:26:17,443 [main] WARN org.apache.hadoop.conf.Configuration - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
2014-08-13 22:26:17,445 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-08-13 22:26:17,449 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:17,450 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:17,451 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:17,452 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:17,538 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:17,539 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:17,608 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:17,608 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:17,687 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:17,696 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:17,783 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:17,788 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:18,926 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:18,930 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:18,996 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
2014-08-13 22:26:19,670 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:19,671 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:19,672 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:19,767 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:19,767 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:19,768 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:19,874 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:19,874 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:19,886 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:19,980 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:19,981 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:19,987 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:20,018 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:20,018 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:20,039 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:20,145 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:20,158 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:20,159 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:20,210 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:20,210 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:20,211 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:20,322 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:20,322 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:20,323 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:20,412 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,ORDER_BY,FILTER
2014-08-13 22:26:20,497 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:20,521 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for messages: $0, $1, $2, $3, $4, $5, $8, $9, $10
2014-08-13 22:26:20,561 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:20,562 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:20,583 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:20,583 [main] WARN org.apache.hadoop.conf.Configuration - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
2014-08-13 22:26:20,803 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-08-13 22:26:20,874 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:20,875 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:20,905 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2014-08-13 22:26:20,952 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 3
2014-08-13 22:26:20,952 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 3
2014-08-13 22:26:20,959 [main] WARN org.apache.hadoop.conf.Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2014-08-13 22:26:20,961 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2014-08-13 22:26:20,973 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:21,014 [main] WARN org.apache.pig.backend.hadoop20.PigJobControl - falling back to default JobControl (not using hadoop 0.20 ?)
java.lang.NoSuchFieldException: runnerState
at java.lang.Class.getDeclaredField(Class.java:2057)
at org.apache.pig.backend.hadoop20.PigJobControl.(PigJobControl.java:51)
at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:97)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:285)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
at org.apache.pig.PigServer.execute(PigServer.java:1239)
at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:604)
at org.apache.pig.Main.main(Main.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
2014-08-13 22:26:21,052 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:21,055 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-08-13 22:26:21,106 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
2014-08-13 22:26:21,107 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-08-13 22:26:21,107 [main] WARN org.apache.hadoop.conf.Configuration - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
2014-08-13 22:26:21,107 [main] WARN org.apache.hadoop.conf.Configuration - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
2014-08-13 22:26:21,114 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2014-08-13 22:26:21,116 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=23891952
2014-08-13 22:26:21,117 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2014-08-13 22:26:21,118 [main] WARN org.apache.hadoop.conf.Configuration - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2014-08-13 22:26:21,260 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-08-13 22:26:21,274 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2014-08-13 22:26:21,275 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2014-08-13 22:26:21,275 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Distributed cache not supported or needed in local mode. Setting key [pig.schematuple.local.dir] with code temp directory: /var/folders/nj/nphdkhyj6s1dttb0pd9zb2wc0000gn/T/1407990381274-0
2014-08-13 22:26:21,488 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-08-13 22:26:21,491 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
2014-08-13 22:26:21,492 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.NoSuchMethodError: org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:296)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
at org.apache.pig.PigServer.execute(PigServer.java:1239)
at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:604)
at org.apache.pig.Main.main(Main.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

Details also at logfile: /private/tmp/pig_1407990376721.log
David-Laxers-MacBook-Pro:pig davidlaxer$

sudo ant clean jar-withouthadoop
Buildfile: /Users/davidlaxer/pig-0.11.1-src/build.xml

clean:
[delete] Deleting directory /Users/davidlaxer/pig-0.11.1-src/build

clean:

ivy-download:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar
[get] To: /Users/davidlaxer/pig-0.11.1-src/ivy/ivy-2.2.0.jar
[get] Not modified - so not downloaded

ivy-init-dirs:
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/build/ivy
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/build/ivy/lib
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/build/ivy/report
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/build/ivy/maven

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:
[ivy:configure] :: Ivy 2.2.0 - 20100923230623 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /Users/davidlaxer/pig-0.11.1-src/ivy/ivysettings.xml

ivy-resolve:
[ivy:resolve] :: resolving dependencies :: org.apache.pig#pig;0.11.2-SNAPSHOT
[ivy:resolve] confs: [master, default, runtime, compile, test, javadoc, releaseaudit, jdiff, checkstyle, buildJar, hadoop20, hadoop23]
[ivy:resolve] found com.sun.jersey#jersey-core;1.8 in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-core;1.0.0 in maven2
[ivy:resolve] found commons-cli#commons-cli;1.2 in maven2
[ivy:resolve] found xmlenc#xmlenc;0.52 in maven2
[ivy:resolve] found commons-httpclient#commons-httpclient;3.0.1 in maven2
[ivy:resolve] found commons-codec#commons-codec;1.4 in maven2
[ivy:resolve] found org.apache.commons#commons-math;2.1 in maven2
[ivy:resolve] found commons-configuration#commons-configuration;1.6 in maven2
[ivy:resolve] found commons-collections#commons-collections;3.2.1 in maven2
[ivy:resolve] found commons-lang#commons-lang;2.4 in maven2
[ivy:resolve] found commons-logging#commons-logging;1.1.1 in maven2
[ivy:resolve] found commons-digester#commons-digester;1.8 in maven2
[ivy:resolve] found commons-beanutils#commons-beanutils;1.7.0 in maven2
[ivy:resolve] found commons-beanutils#commons-beanutils-core;1.8.0 in maven2
[ivy:resolve] found commons-net#commons-net;1.4.1 in maven2
[ivy:resolve] found oro#oro;2.0.8 in maven2
[ivy:resolve] found org.mortbay.jetty#jetty;6.1.26 in maven2
[ivy:resolve] found org.mortbay.jetty#jetty-util;6.1.26 in maven2
[ivy:resolve] found org.mortbay.jetty#servlet-api;2.5-20081211 in maven2
[ivy:resolve] found tomcat#jasper-runtime;5.5.12 in maven2
[ivy:resolve] found tomcat#jasper-compiler;5.5.12 in maven2
[ivy:resolve] found org.mortbay.jetty#jsp-api-2.1;6.1.14 in maven2
[ivy:resolve] found org.mortbay.jetty#servlet-api-2.5;6.1.14 in maven2
[ivy:resolve] found org.mortbay.jetty#jsp-2.1;6.1.14 in maven2
[ivy:resolve] found org.eclipse.jdt#core;3.1.1 in maven2
[ivy:resolve] found ant#ant;1.6.5 in maven2
[ivy:resolve] found commons-el#commons-el;1.0 in maven2
[ivy:resolve] found net.java.dev.jets3t#jets3t;0.7.1 in maven2
[ivy:resolve] found net.sf.kosmosfs#kfs;0.3 in maven2
[ivy:resolve] found hsqldb#hsqldb;1.8.0.10 in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-test;1.0.0 in maven2
[ivy:resolve] found org.apache.ftpserver#ftplet-api;1.0.0 in maven2
[ivy:resolve] found org.apache.mina#mina-core;2.0.0-M5 in maven2
[ivy:resolve] found org.slf4j#slf4j-api;1.5.2 in maven2
[ivy:resolve] found org.apache.ftpserver#ftpserver-core;1.0.0 in maven2
[ivy:resolve] found org.apache.ftpserver#ftpserver-deprecated;1.0.0-M2 in maven2
[ivy:resolve] found commons-io#commons-io;2.3 in maven2
[ivy:resolve] found org.apache.httpcomponents#httpclient;4.1 in maven2
[ivy:resolve] found org.apache.httpcomponents#httpcore;4.1 in maven2
[ivy:resolve] found log4j#log4j;1.2.16 in maven2
[ivy:resolve] found org.slf4j#slf4j-log4j12;1.6.1 in maven2
[ivy:resolve] found org.apache.avro#avro;1.5.3 in maven2
[ivy:resolve] found com.thoughtworks.paranamer#paranamer;2.3 in maven2
[ivy:resolve] found org.xerial.snappy#snappy-java;1.0.3.2 in maven2
[ivy:resolve] found org.slf4j#slf4j-api;1.6.1 in maven2
[ivy:resolve] found com.googlecode.json-simple#json-simple;1.1 in maven2
[ivy:resolve] found com.jcraft#jsch;0.1.38 in maven2
[ivy:resolve] found jline#jline;0.9.94 in maven2
[ivy:resolve] found net.java.dev.javacc#javacc;4.2 in maven2
[ivy:resolve] found org.codehaus.groovy#groovy-all;1.8.6 in maven2
[ivy:resolve] found org.codehaus.jackson#jackson-mapper-asl;1.8.8 in maven2
[ivy:resolve] found org.codehaus.jackson#jackson-core-asl;1.8.8 in maven2
[ivy:resolve] found org.fusesource.jansi#jansi;1.9 in maven2
[ivy:resolve] found joda-time#joda-time;1.6 in maven2
[ivy:resolve] found com.google.guava#guava;11.0 in maven2
[ivy:resolve] found org.python#jython-standalone;2.5.3 in maven2
[ivy:resolve] found rhino#js;1.7R2 in maven2
[ivy:resolve] found org.antlr#antlr;3.4 in maven2
[ivy:resolve] found org.antlr#antlr-runtime;3.4 in maven2
[ivy:resolve] found org.antlr#stringtemplate;3.2.1 in maven2
[ivy:resolve] found antlr#antlr;2.7.7 in maven2
[ivy:resolve] found org.antlr#ST4;4.0.4 in maven2
[ivy:resolve] found org.apache.zookeeper#zookeeper;3.3.3 in maven2
[ivy:resolve] found dk.brics.automaton#automaton;1.11-8 in maven2
[ivy:resolve] found org.jruby#jruby-complete;1.6.7 in maven2
[ivy:resolve] found org.apache.hbase#hbase;0.90.0 in maven2
[ivy:resolve] found org.vafer#jdeb;0.8 in maven2
[ivy:resolve] found org.mockito#mockito-all;1.8.4 in maven2
[ivy:resolve] found xalan#xalan;2.7.1 in maven2
[ivy:resolve] found xalan#serializer;2.7.1 in maven2
[ivy:resolve] found xml-apis#xml-apis;1.3.04 in maven2
[ivy:resolve] found xerces#xercesImpl;2.10.0 in maven2
[ivy:resolve] found xml-apis#xml-apis;1.4.01 in maven2
[ivy:resolve] found junit#junit;4.11 in maven2
[ivy:resolve] found org.hamcrest#hamcrest-core;1.3 in maven2
[ivy:resolve] found org.jboss.netty#netty;3.2.2.Final in maven2
[ivy:resolve] found org.apache.hive#hive-exec;0.8.0 in maven2
[ivy:resolve] found junit#junit;3.8.1 in maven2
[ivy:resolve] found com.google.code.p.arat#rat-lib;0.5.1 in maven2
[ivy:resolve] found commons-collections#commons-collections;3.2 in maven2
[ivy:resolve] found commons-lang#commons-lang;2.1 in maven2
[ivy:resolve] found jdiff#jdiff;1.0.9 in maven2
[ivy:resolve] found checkstyle#checkstyle;4.2 in maven2
[ivy:resolve] found commons-beanutils#commons-beanutils-core;1.7.0 in maven2
[ivy:resolve] found commons-cli#commons-cli;1.0 in maven2
[ivy:resolve] found commons-logging#commons-logging;1.0.3 in maven2
[ivy:resolve] found org.codehaus.jackson#jackson-mapper-asl;1.0.1 in maven2
[ivy:resolve] found org.codehaus.jackson#jackson-core-asl;1.0.1 in maven2
[ivy:resolve] found com.sun.jersey#jersey-bundle;1.8 in maven2
[ivy:resolve] found com.sun.jersey#jersey-server;1.8 in maven2
[ivy:resolve] found com.sun.jersey.contribs#jersey-guice;1.8 in maven2
[ivy:resolve] found commons-httpclient#commons-httpclient;3.1 in maven2
[ivy:resolve] found javax.servlet#servlet-api;2.5 in maven2
[ivy:resolve] found javax.ws.rs#jsr311-api;1.1.1 in maven2
[ivy:resolve] found com.google.protobuf#protobuf-java;2.4.0a in maven2
[ivy:resolve] found javax.inject#javax.inject;1 in maven2
[ivy:resolve] found javax.xml.bind#jaxb-api;2.2.2 in maven2
[ivy:resolve] found com.sun.xml.bind#jaxb-impl;2.2.3-1 in maven2
[ivy:resolve] found com.google.inject#guice;3.0 in maven2
[ivy:resolve] found com.google.inject.extensions#guice-servlet;3.0 in maven2
[ivy:resolve] found aopalliance#aopalliance;1.0 in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-annotations;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-auth;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-common;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-hdfs;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-core;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-jobclient;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server-tests;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-app;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-shuffle;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-common;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-api;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-common;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server-web-proxy;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server-common;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server-nodemanager;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server-resourcemanager;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-hs;2.0.0-alpha in maven2
[ivy:resolve] :: resolution report :: resolve 6281ms :: artifacts dl 178ms
[ivy:resolve] :: evicted modules:
[ivy:resolve] junit#junit;3.8.1 by [junit#junit;4.11] in [javadoc, default, test, buildJar, runtime]
[ivy:resolve] commons-logging#commons-logging;1.0.3 by [commons-logging#commons-logging;1.1.1] in [javadoc, default, compile, test, buildJar, hadoop20, runtime]
[ivy:resolve] commons-codec#commons-codec;1.2 by [commons-codec#commons-codec;1.4] in [javadoc, default, compile, test, buildJar, hadoop20, runtime]
[ivy:resolve] commons-logging#commons-logging;1.1 by [commons-logging#commons-logging;1.1.1] in [javadoc, default, compile, test, buildJar, hadoop20, runtime]
[ivy:resolve] commons-codec#commons-codec;1.3 by [commons-codec#commons-codec;1.4] in [javadoc, default, compile, test, buildJar, hadoop20, runtime]
[ivy:resolve] commons-httpclient#commons-httpclient;3.1 by [commons-httpclient#commons-httpclient;3.0.1] in [javadoc, default, compile, test, buildJar, hadoop20, runtime]
[ivy:resolve] org.codehaus.jackson#jackson-mapper-asl;1.0.1 by [org.codehaus.jackson#jackson-mapper-asl;1.8.8] in [javadoc, default, compile, test, buildJar, runtime]
[ivy:resolve] org.slf4j#slf4j-api;1.5.2 by [org.slf4j#slf4j-api;1.6.1] in [javadoc, default, compile, test, buildJar, runtime]
[ivy:resolve] org.apache.mina#mina-core;2.0.0-M4 by [org.apache.mina#mina-core;2.0.0-M5] in [javadoc, default, compile, test, buildJar, hadoop20, runtime]
[ivy:resolve] org.apache.ftpserver#ftplet-api;1.0.0-M2 by [org.apache.ftpserver#ftplet-api;1.0.0] in [javadoc, default, compile, test, buildJar, hadoop20, runtime]
[ivy:resolve] org.apache.ftpserver#ftpserver-core;1.0.0-M2 by [org.apache.ftpserver#ftpserver-core;1.0.0] in [javadoc, default, compile, test, buildJar, hadoop20, runtime]
[ivy:resolve] org.apache.mina#mina-core;2.0.0-M2 by [org.apache.mina#mina-core;2.0.0-M5] in [javadoc, default, compile, test, buildJar, hadoop20, runtime]
[ivy:resolve] commons-cli#commons-cli;1.0 by [commons-cli#commons-cli;1.2] in [javadoc, default, compile, test, buildJar, runtime]
[ivy:resolve] org.codehaus.jackson#jackson-mapper-asl;1.7.3 by [org.codehaus.jackson#jackson-mapper-asl;1.8.8] in [javadoc, default, compile, test, buildJar, runtime]
[ivy:resolve] org.antlr#antlr-runtime;3.3 by [org.antlr#antlr-runtime;3.4] in [javadoc, default, compile, test, buildJar, runtime]
[ivy:resolve] xml-apis#xml-apis;1.3.04 by [xml-apis#xml-apis;1.4.01] in [javadoc, default, test, buildJar, runtime]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| master | 0 | 0 | 0 | 0 || 0 | 0 |
| default | 91 | 0 | 0 | 16 || 76 | 0 |
| runtime | 91 | 0 | 0 | 16 || 76 | 0 |
| compile | 82 | 0 | 0 | 14 || 69 | 0 |
| test | 91 | 0 | 0 | 16 || 76 | 0 |
| javadoc | 91 | 0 | 0 | 16 || 76 | 0 |
| releaseaudit | 3 | 0 | 0 | 0 || 3 | 0 |
| jdiff | 3 | 0 | 0 | 0 || 3 | 0 |
| checkstyle | 6 | 0 | 0 | 0 || 6 | 0 |
| buildJar | 91 | 0 | 0 | 16 || 76 | 0 |
| hadoop20 | 48 | 0 | 0 | 9 || 39 | 0 |
| hadoop23 | 39 | 0 | 0 | 0 || 41 | 0 |
---------------------------------------------------------------------

ivy-compile:
[ivy:retrieve] :: retrieving :: org.apache.pig#pig
[ivy:retrieve] confs: [compile]
[ivy:retrieve] 69 artifacts copied, 0 already retrieved (66821kB/2244ms)
[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
[ivy:cachepath] :: loading settings :: file = /Users/davidlaxer/pig-0.11.1-src/ivy/ivysettings.xml

init:
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/src-gen/org/apache/pig/impl/logicalLayer/parser
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/src-gen/org/apache/pig/tools/pigscript/parser
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/src-gen/org/apache/pig/tools/parameters
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/build/classes
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/build/test/classes
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/test/org/apache/pig/test/utils/dotGraph/parser
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/src-gen/org/apache/pig/data/parser
[move] Moving 1 file to /Users/davidlaxer/pig-0.11.1-src/build/ivy/lib/Pig

cc-compile:
[javacc] Java Compiler Compiler Version 4.2 (Parser Generator)
[javacc](type "javacc" with no arguments for help)
[javacc] Reading from file /Users/davidlaxer/pig-0.11.1-src/src/org/apache/pig/tools/pigscript/parser/PigScriptParser.jj . . .
[javacc] File "TokenMgrError.java" does not exist. Will create one.
[javacc] File "ParseException.java" does not exist. Will create one.
[javacc] File "Token.java" does not exist. Will create one.
[javacc] File "JavaCharStream.java" does not exist. Will create one.
[javacc] Parser generated successfully.
[javacc] Java Compiler Compiler Version 4.2 (Parser Generator)
[javacc](type "javacc" with no arguments for help)
[javacc] Reading from file /Users/davidlaxer/pig-0.11.1-src/src/org/apache/pig/tools/parameters/PigFileParser.jj . . .
[javacc] Warning: Lookahead adequacy checking not being performed since option LOOKAHEAD is more than 1. Set option FORCE_LA_CHECK to true to force checking.
[javacc] File "TokenMgrError.java" does not exist. Will create one.
[javacc] File "ParseException.java" does not exist. Will create one.
[javacc] File "Token.java" does not exist. Will create one.
[javacc] File "JavaCharStream.java" does not exist. Will create one.
[javacc] Parser generated with 0 errors and 1 warnings.
[javacc] Java Compiler Compiler Version 4.2 (Parser Generator)
[javacc](type "javacc" with no arguments for help)
[javacc] Reading from file /Users/davidlaxer/pig-0.11.1-src/src/org/apache/pig/tools/parameters/ParamLoader.jj . . .
[javacc] File "TokenMgrError.java" is being rebuilt.
[javacc] File "ParseException.java" is being rebuilt.
[javacc] File "Token.java" is being rebuilt.
[javacc] File "JavaCharStream.java" is being rebuilt.
[javacc] Parser generated successfully.
[jjtree] Java Compiler Compiler Version 4.2 (Tree Builder)
[jjtree](type "jjtree" with no arguments for help)
[jjtree] Reading from file /Users/davidlaxer/pig-0.11.1-src/test/org/apache/pig/test/utils/dotGraph/DOTParser.jjt . . .
[jjtree] File "Node.java" does not exist. Will create one.
[jjtree] File "SimpleNode.java" does not exist. Will create one.
[jjtree] File "DOTParserTreeConstants.java" does not exist. Will create one.
[jjtree] File "JJTDOTParserState.java" does not exist. Will create one.
[jjtree] Annotated grammar generated successfully in /Users/davidlaxer/pig-0.11.1-src/test/org/apache/pig/test/utils/dotGraph/parser/DOTParser.jj
[javacc] Java Compiler Compiler Version 4.2 (Parser Generator)
[javacc](type "javacc" with no arguments for help)
[javacc] Reading from file /Users/davidlaxer/pig-0.11.1-src/test/org/apache/pig/test/utils/dotGraph/parser/DOTParser.jj . . .
[javacc] File "TokenMgrError.java" does not exist. Will create one.
[javacc] File "ParseException.java" does not exist. Will create one.
[javacc] File "Token.java" does not exist. Will create one.
[javacc] File "SimpleCharStream.java" does not exist. Will create one.
[javacc] Parser generated successfully.

prepare:
[mkdir] Created dir: /Users/davidlaxer/pig-0.11.1-src/src-gen/org/apache/pig/parser

genLexer:

genParser:

genTreeParser:

gen:

compile:
[echo] *** Building Main Sources ***
[echo] *** To compile with all warnings enabled, supply -Dall.warnings=1 on command line ***
[echo] *** If all.warnings property is supplied, compile-sources-all-warnings target will be executed ***
[echo] *** Else, compile-sources (which only warns about deprecations) target will be executed ***

compile-sources:
[javac] /Users/davidlaxer/pig-0.11.1-src/build.xml:531: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 787 source files to /Users/davidlaxer/pig-0.11.1-src/build/classes
[javac] warning: [options] bootstrap class path not set in conjunction with -source 1.5
[javac] warning: [options] source value 1.5 is obsolete and will be removed in a future release
[javac] warning: [options] target value 1.5 is obsolete and will be removed in a future release
[javac] warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 4 warnings
[copy] Copying 1 file to /Users/davidlaxer/pig-0.11.1-src/build/classes/org/apache/pig/tools/grunt
[copy] Copying 1 file to /Users/davidlaxer/pig-0.11.1-src/build/classes/org/apache/pig/tools/grunt

compile-sources-all-warnings:

jar-withouthadoop:

jar-withouthadoopWithSvn:

ivy-download:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar
[get] To: /Users/davidlaxer/pig-0.11.1-src/ivy/ivy-2.2.0.jar
[get] Not modified - so not downloaded

ivy-init-dirs:

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:

ivy-resolve:

ivy-buildJar:
[ivy:retrieve] :: retrieving :: org.apache.pig#pig
[ivy:retrieve] confs: [buildJar]
[ivy:retrieve] 9 artifacts copied, 67 already retrieved (9631kB/208ms)

buildJar-withouthadoop:
[echo] svnString Unversioned directory
[jar] Building jar: /Users/davidlaxer/pig-0.11.1-src/build/pig-0.11.2-SNAPSHOT.jar
[jar] Building jar: /Users/davidlaxer/pig-0.11.1-src/build/pig-0.11.2-SNAPSHOT-withouthadoop.jar
[copy] Copying 1 file to /Users/davidlaxer/pig-0.11.1-src

jar-withouthadoopWithOutSvn:

BUILD SUCCESSFUL
Total time: 1 minute 55 seconds
David-Laxers-MacBook-Pro:pig-0.11.1-src davidlaxer$ ant test
Buildfile: /Users/davidlaxer/pig-0.11.1-src/build.xml

test:

ivy-download:
[get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar
[get] To: /Users/davidlaxer/pig-0.11.1-src/ivy/ivy-2.2.0.jar
[get] Not modified - so not downloaded

ivy-init-dirs:

ivy-probe-antlib:

ivy-init-antlib:

ivy-init:
[ivy:configure] :: Ivy 2.2.0 - 20100923230623 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /Users/davidlaxer/pig-0.11.1-src/ivy/ivysettings.xml

ivy-resolve:
[ivy:resolve] :: resolving dependencies :: org.apache.pig#pig;0.11.2-SNAPSHOT
[ivy:resolve] confs: [master, default, runtime, compile, test, javadoc, releaseaudit, jdiff, checkstyle, buildJar, hadoop20, hadoop23]
[ivy:resolve] found com.sun.jersey#jersey-core;1.8 in default
[ivy:resolve] found org.apache.hadoop#hadoop-core;1.0.0 in maven2
[ivy:resolve] found commons-cli#commons-cli;1.2 in fs
[ivy:resolve] found xmlenc#xmlenc;0.52 in default
[ivy:resolve] found commons-httpclient#commons-httpclient;3.0.1 in maven2
[ivy:resolve] found commons-codec#commons-codec;1.4 in fs
[ivy:resolve] found org.apache.commons#commons-math;2.1 in default
[ivy:resolve] found commons-configuration#commons-configuration;1.6 in default
[ivy:resolve] found commons-collections#commons-collections;3.2.1 in fs
[ivy:resolve] found commons-lang#commons-lang;2.4 in fs
[ivy:resolve] found commons-logging#commons-logging;1.1.1 in fs
[ivy:resolve] found commons-digester#commons-digester;1.8 in fs
[ivy:resolve] found commons-beanutils#commons-beanutils;1.7.0 in fs
[ivy:resolve] found commons-beanutils#commons-beanutils-core;1.8.0 in default
[ivy:resolve] found commons-net#commons-net;1.4.1 in maven2
[ivy:resolve] found oro#oro;2.0.8 in fs
[ivy:resolve] found org.mortbay.jetty#jetty;6.1.26 in fs
[ivy:resolve] found org.mortbay.jetty#jetty-util;6.1.26 in fs
[ivy:resolve] found tomcat#jasper-runtime;5.5.12 in maven2
[ivy:resolve] found tomcat#jasper-compiler;5.5.12 in maven2
[ivy:resolve] found org.mortbay.jetty#jsp-api-2.1;6.1.14 in default
[ivy:resolve] found org.mortbay.jetty#servlet-api-2.5;6.1.14 in default
[ivy:resolve] found org.mortbay.jetty#jsp-2.1;6.1.14 in default
[ivy:resolve] found org.eclipse.jdt#core;3.1.1 in default
[ivy:resolve] found ant#ant;1.6.5 in maven2
[ivy:resolve] found commons-el#commons-el;1.0 in default
[ivy:resolve] found net.java.dev.jets3t#jets3t;0.7.1 in default
[ivy:resolve] found net.sf.kosmosfs#kfs;0.3 in maven2
[ivy:resolve] found hsqldb#hsqldb;1.8.0.10 in default
[ivy:resolve] found org.apache.hadoop#hadoop-test;1.0.0 in maven2
[ivy:resolve] found org.apache.ftpserver#ftplet-api;1.0.0 in maven2
[ivy:resolve] found org.apache.mina#mina-core;2.0.0-M5 in fs
[ivy:resolve] found org.slf4j#slf4j-api;1.5.2 in maven2
[ivy:resolve] found org.apache.ftpserver#ftpserver-core;1.0.0 in maven2
[ivy:resolve] found org.apache.ftpserver#ftpserver-deprecated;1.0.0-M2 in maven2
[ivy:resolve] found commons-io#commons-io;2.3 in maven2
[ivy:resolve] found org.apache.httpcomponents#httpclient;4.1 in maven2
[ivy:resolve] found org.apache.httpcomponents#httpcore;4.1 in maven2
[ivy:resolve] found log4j#log4j;1.2.16 in maven2
[ivy:resolve] found org.slf4j#slf4j-log4j12;1.6.1 in maven2
[ivy:resolve] found org.apache.avro#avro;1.5.3 in maven2
[ivy:resolve] found com.thoughtworks.paranamer#paranamer;2.3 in maven2
[ivy:resolve] found org.xerial.snappy#snappy-java;1.0.3.2 in maven2
[ivy:resolve] found org.slf4j#slf4j-api;1.6.1 in maven2
[ivy:resolve] found com.googlecode.json-simple#json-simple;1.1 in default
[ivy:resolve] found com.jcraft#jsch;0.1.38 in fs
[ivy:resolve] found jline#jline;0.9.94 in fs
[ivy:resolve] found net.java.dev.javacc#javacc;4.2 in maven2
[ivy:resolve] found org.codehaus.groovy#groovy-all;1.8.6 in maven2
[ivy:resolve] found org.codehaus.jackson#jackson-mapper-asl;1.8.8 in maven2
[ivy:resolve] found org.codehaus.jackson#jackson-core-asl;1.8.8 in maven2
[ivy:resolve] found org.fusesource.jansi#jansi;1.9 in maven2
[ivy:resolve] found joda-time#joda-time;1.6 in maven2
[ivy:resolve] found com.google.guava#guava;11.0 in maven2
[ivy:resolve] found org.python#jython-standalone;2.5.3 in maven2
[ivy:resolve] found rhino#js;1.7R2 in maven2
[ivy:resolve] found org.antlr#antlr;3.4 in default
[ivy:resolve] found org.antlr#antlr-runtime;3.4 in default
[ivy:resolve] found org.antlr#stringtemplate;3.2.1 in default
[ivy:resolve] found antlr#antlr;2.7.7 in fs
[ivy:resolve] found org.antlr#ST4;4.0.4 in default
[ivy:resolve] found org.apache.zookeeper#zookeeper;3.3.3 in maven2
[ivy:resolve] found dk.brics.automaton#automaton;1.11-8 in maven2
[ivy:resolve] found org.jruby#jruby-complete;1.6.7 in maven2
[ivy:resolve] found org.apache.hbase#hbase;0.90.0 in maven2
[ivy:resolve] found org.vafer#jdeb;0.8 in maven2
[ivy:resolve] found org.mockito#mockito-all;1.8.4 in maven2
[ivy:resolve] found xalan#xalan;2.7.1 in maven2
[ivy:resolve] found xalan#serializer;2.7.1 in maven2
[ivy:resolve] found xml-apis#xml-apis;1.3.04 in fs
[ivy:resolve] found xerces#xercesImpl;2.10.0 in maven2
[ivy:resolve] found xml-apis#xml-apis;1.4.01 in maven2
[ivy:resolve] found junit#junit;4.11 in fs
[ivy:resolve] found org.hamcrest#hamcrest-core;1.3 in fs
[ivy:resolve] found org.jboss.netty#netty;3.2.2.Final in maven2
[ivy:resolve] found org.apache.hive#hive-exec;0.8.0 in maven2
[ivy:resolve] found junit#junit;3.8.1 in fs
[ivy:resolve] found com.google.code.p.arat#rat-lib;0.5.1 in maven2
[ivy:resolve] found commons-collections#commons-collections;3.2 in fs
[ivy:resolve] found commons-lang#commons-lang;2.1 in fs
[ivy:resolve] found jdiff#jdiff;1.0.9 in fs
[ivy:resolve] found checkstyle#checkstyle;4.2 in maven2
[ivy:resolve] found commons-beanutils#commons-beanutils-core;1.7.0 in maven2
[ivy:resolve] found commons-cli#commons-cli;1.0 in fs
[ivy:resolve] found commons-logging#commons-logging;1.0.3 in fs
[ivy:resolve] found org.codehaus.jackson#jackson-mapper-asl;1.0.1 in default
[ivy:resolve] found org.codehaus.jackson#jackson-core-asl;1.0.1 in default
[ivy:resolve] found com.sun.jersey#jersey-bundle;1.8 in maven2
[ivy:resolve] found com.sun.jersey#jersey-server;1.8 in default
[ivy:resolve] found com.sun.jersey.contribs#jersey-guice;1.8 in maven2
[ivy:resolve] found commons-httpclient#commons-httpclient;3.1 in fs
[ivy:resolve] found javax.servlet#servlet-api;2.5 in fs
[ivy:resolve] found javax.ws.rs#jsr311-api;1.1.1 in maven2
[ivy:resolve] found com.google.protobuf#protobuf-java;2.4.0a in default
[ivy:resolve] found javax.inject#javax.inject;1 in maven2
[ivy:resolve] found javax.xml.bind#jaxb-api;2.2.2 in maven2
[ivy:resolve] found com.sun.xml.bind#jaxb-impl;2.2.3-1 in default
[ivy:resolve] found com.google.inject#guice;3.0 in maven2
[ivy:resolve] found com.google.inject.extensions#guice-servlet;3.0 in maven2
[ivy:resolve] found aopalliance#aopalliance;1.0 in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-annotations;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-auth;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-common;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-hdfs;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-core;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-jobclient;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server-tests;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-app;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-shuffle;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-common;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-api;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-common;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server-web-proxy;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server-common;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server-nodemanager;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-yarn-server-resourcemanager;2.0.0-alpha in maven2
[ivy:resolve] found org.apache.hadoop#hadoop-mapreduce-client-hs;2.0.0-alpha in maven2
[ivy:resolve] :: resolution report :: resolve 13603ms :: artifacts dl 5473ms
[ivy:resolve] :: evicted modules:
[ivy:resolve] junit#junit;3.8.1 by [junit#junit;4.11] in [default, test, runtime, javadoc, buildJar]
[ivy:resolve] commons-logging#commons-logging;1.0.3 by [commons-logging#commons-logging;1.1.1] in [default, hadoop20, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] commons-codec#commons-codec;1.2 by [commons-codec#commons-codec;1.4] in [default, hadoop20, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] commons-logging#commons-logging;1.1 by [commons-logging#commons-logging;1.1.1] in [default, hadoop20, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] commons-codec#commons-codec;1.3 by [commons-codec#commons-codec;1.4] in [default, hadoop20, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] commons-httpclient#commons-httpclient;3.1 by [commons-httpclient#commons-httpclient;3.0.1] in [default, hadoop20, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] org.codehaus.jackson#jackson-mapper-asl;1.0.1 by [org.codehaus.jackson#jackson-mapper-asl;1.8.8] in [default, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] org.slf4j#slf4j-api;1.5.2 by [org.slf4j#slf4j-api;1.6.1] in [default, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] org.apache.mina#mina-core;2.0.0-M4 by [org.apache.mina#mina-core;2.0.0-M5] in [default, hadoop20, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] org.apache.ftpserver#ftplet-api;1.0.0-M2 by [org.apache.ftpserver#ftplet-api;1.0.0] in [default, hadoop20, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] org.apache.ftpserver#ftpserver-core;1.0.0-M2 by [org.apache.ftpserver#ftpserver-core;1.0.0] in [default, hadoop20, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] org.apache.mina#mina-core;2.0.0-M2 by [org.apache.mina#mina-core;2.0.0-M5] in [default, hadoop20, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] commons-cli#commons-cli;1.0 by [commons-cli#commons-cli;1.2] in [default, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] org.codehaus.jackson#jackson-mapper-asl;1.7.3 by [org.codehaus.jackson#jackson-mapper-asl;1.8.8] in [default, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] org.antlr#antlr-runtime;3.3 by [org.antlr#antlr-runtime;3.4] in [default, test, compile, runtime, javadoc, buildJar]
[ivy:resolve] xml-apis#xml-apis;1.3.04 by [xml-apis#xml-apis;1.4.01] in [default, test, runtime, javadoc, buildJar]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| master | 0 | 0 | 0 | 0 || 0 | 0 |
| default | 90 | 0 | 0 | 16 || 75 | 0 |
| runtime | 90 | 0 | 0 | 16 || 75 | 0 |
| compile | 81 | 0 | 0 | 14 || 68 | 0 |
| test | 90 | 0 | 0 | 16 || 75 | 0 |
| javadoc | 90 | 0 | 0 | 16 || 75 | 0 |
| releaseaudit | 3 | 0 | 0 | 0 || 3 | 0 |
| jdiff | 3 | 0 | 0 | 0 || 3 | 0 |
| checkstyle | 6 | 0 | 0 | 0 || 6 | 0 |
| buildJar | 90 | 0 | 0 | 16 || 75 | 0 |
| hadoop20 | 47 | 0 | 0 | 9 || 38 | 0 |
| hadoop23 | 39 | 0 | 0 | 0 || 41 | 0 |
---------------------------------------------------------------------
[ivy:resolve]
[ivy:resolve] :: problems summary ::
[ivy:resolve] :::: WARNINGS
[ivy:resolve] [FAILED ] org.mortbay.jetty#jetty;6.1.26!jetty.zip: (0ms)
[ivy:resolve] ==== local: tried
[ivy:resolve] /Users/davidlaxer/.ivy2/local/org.mortbay.jetty/jetty/6.1.26/zips/jetty.zip
[ivy:resolve] ==== maven2: tried
[ivy:resolve] http://repo2.maven.org/maven2/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip
[ivy:resolve] ==== jboss-maven2: tried
[ivy:resolve] http://repository.jboss.com/nexus/content/groups/public/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip
[ivy:resolve] ==== apache-snapshots: tried
[ivy:resolve] http://repository.apache.org/content/groups/snapshots-group/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] :: FAILED DOWNLOADS ::
[ivy:resolve] :: ^ see resolution messages for details ^ ::
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] :: org.mortbay.jetty#jetty;6.1.26!jetty.zip
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve]
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

BUILD FAILED
/Users/davidlaxer/pig-0.11.1-src/build.xml:941: The following error occurred while executing this line:
/Users/davidlaxer/pig-0.11.1-src/build.xml:1723: impossible to resolve dependencies:
resolve failed - see output for details

Total time: 23 seconds

find ~ -name '6.1.26' -ls
42361118 1056 -rw-r--r-- 1 davidlaxer staff 540234 Dec 22 2012 /Users/davidlaxer/.bundler/ruby/1.8/sunspot-78717a338942/sunspot_solr/solr/lib/jetty-6.1.26-patched-JETTY-1340.jar
42361119 352 -rw-r--r-- 1 davidlaxer staff 177393 Dec 22 2012 /Users/davidlaxer/.bundler/ruby/1.8/sunspot-78717a338942/sunspot_solr/solr/lib/jetty-util-6.1.26-patched-JETTY-1340.jar
80795382 8 -rw-r--r-- 1 davidlaxer staff 2189 May 22 18:03 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/jetty/ivy-6.1.26.xml
82151461 8 -rw-r--r-- 1 davidlaxer staff 425 May 22 18:03 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/jetty/ivy-6.1.26.xml.original
80795336 8 -rw-r--r-- 1 davidlaxer staff 838 Aug 13 20:52 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/jetty/ivydata-6.1.26.properties
80796510 1056 -rw-r--r-- 1 davidlaxer staff 539912 Nov 10 2010 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/jetty/jars/jetty-6.1.26.jar
80795393 16 -rw-r--r-- 1 davidlaxer staff 7327 May 22 18:02 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/jetty-util/ivy-6.1.26.xml
82151464 8 -rw-r--r-- 1 davidlaxer staff 3876 May 22 18:02 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/jetty-util/ivy-6.1.26.xml.original
80795391 8 -rw-r--r-- 1 davidlaxer staff 900 Aug 13 20:52 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/jetty-util/ivydata-6.1.26.properties
80796764 352 -rw-r--r-- 1 davidlaxer staff 177131 Nov 10 2010 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/jetty-util/jars/jetty-util-6.1.26.jar
80795380 16 -rw-r--r-- 1 davidlaxer staff 5000 May 22 18:02 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/project/ivy-6.1.26.xml
82151467 24 -rw-r--r-- 1 davidlaxer staff 9205 May 22 18:02 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/project/ivy-6.1.26.xml.original
80795345 8 -rw-r--r-- 1 davidlaxer staff 620 Aug 9 18:00 /Users/davidlaxer/.ivy2/cache/org.mortbay.jetty/project/ivydata-6.1.26.properties
75248719 0 drwxr-xr-x 9 davidlaxer staff 306 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty/6.1.26
75248840 1056 -rw-r--r-- 1 davidlaxer staff 539912 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar
75248805 8 -rw-r--r-- 1 davidlaxer staff 505 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar.lastUpdated
75248848 8 -rw-r--r-- 1 davidlaxer staff 40 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar.sha1
75248723 8 -rw-r--r-- 1 davidlaxer staff 425 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.pom
75248720 8 -rw-r--r-- 1 davidlaxer staff 376 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.pom.lastUpdated
75248725 8 -rw-r--r-- 1 davidlaxer staff 40 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.pom.sha1
75248696 0 drwxr-xr-x 9 davidlaxer staff 306 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26
75248830 352 -rw-r--r-- 1 davidlaxer staff 177131 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar
75248804 8 -rw-r--r-- 1 davidlaxer staff 505 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar.lastUpdated
75248835 8 -rw-r--r-- 1 davidlaxer staff 40 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar.sha1
75248705 8 -rw-r--r-- 1 davidlaxer staff 3876 May 22 18:02 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.pom
75248698 8 -rw-r--r-- 1 davidlaxer staff 505 May 22 18:02 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.pom.lastUpdated
75248707 8 -rw-r--r-- 1 davidlaxer staff 40 May 22 18:02 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.pom.sha1
75248709 0 drwxr-xr-x 6 davidlaxer staff 204 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/project/6.1.26
75248715 24 -rw-r--r-- 1 davidlaxer staff 9205 May 22 18:02 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/project/6.1.26/project-6.1.26.pom
75248711 8 -rw-r--r-- 1 davidlaxer staff 505 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/project/6.1.26/project-6.1.26.pom.lastUpdated
75248717 8 -rw-r--r-- 1 davidlaxer staff 40 May 22 18:03 /Users/davidlaxer/.m2/repository/org/mortbay/jetty/project/6.1.26/project-6.1.26.pom.sha1
42356239 1056 -rw-r--r-- 1 davidlaxer staff 540234 Dec 22 2012 /Users/davidlaxer/chorus/vendor/solr/lib/jetty-6.1.26-patched-JETTY-1340.jar
42356240 352 -rw-r--r-- 1 davidlaxer staff 177393 Dec 22 2012 /Users/davidlaxer/chorus/vendor/solr/lib/jetty-util-6.1.26-patched-JETTY-1340.jar
75177741 1056 -rw-r--r-- 1 davidlaxer staff 539912 Mar 31 02:49 /Users/davidlaxer/Downloads/hadoop-2.4.0/share/hadoop/common/lib/jetty-6.1.26.jar
75177736 352 -rw-r--r-- 1 davidlaxer staff 177131 Mar 31 02:49 /Users/davidlaxer/Downloads/hadoop-2.4.0/share/hadoop/common/lib/jetty-util-6.1.26.jar
75184493 1056 -rw-r--r-- 1 davidlaxer staff 539912 Mar 31 02:49 /Users/davidlaxer/Downloads/hadoop-2.4.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar
75184491 352 -rw-r--r-- 1 davidlaxer staff 177131 Mar 31 02:49 /Users/davidlaxer/Downloads/hadoop-2.4.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar
75184734 1056 -rw-r--r-- 1 davidlaxer staff 539912 Mar 31 02:49 /Users/davidlaxer/Downloads/hadoop-2.4.0/share/hadoop/tools/lib/jetty-6.1.26.jar
75184730 352 -rw-r--r-- 1 davidlaxer staff 177131 Mar 31 02:49 /Users/davidlaxer/Downloads/hadoop-2.4.0/share/hadoop/tools/lib/jetty-util-6.1.26.jar
75184619 1056 -rw-r--r-- 1 davidlaxer staff 539912 Mar 31 02:49 /Users/davidlaxer/Downloads/hadoop-2.4.0/share/hadoop/yarn/lib/jetty-6.1.26.jar
75184615 352 -rw-r--r-- 1 davidlaxer staff 177131 Mar 31 02:49 /Users/davidlaxer/Downloads/hadoop-2.4.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar
82312261 3400 -rw-r--r-- 1 davidlaxer staff 1740076 Aug 13 21:01 /Users/davidlaxer/Downloads/jetty-hadoop-fix-6.1.26.cloudera.zip
82312276 0 drwxr-xr-x 16 davidlaxer staff 544 Aug 13 21:04 /Users/davidlaxer/jetty-hadoop-fix-6.1.26.cloudera

CH03: 'imaplib.error' when getting Gmail Email w/ script:

The following command (with my username & password):

./gmail.py -m automatic -u [email protected] -p 'mypassword' -s ./email.avro.schema -f '[Gmail]/All Mail' -o /tmp/test_mbox 2>&1 &

And see the following error. Any Ideas?:

Traceback (most recent call last):
  File "./gmail.py", line 104, in <module>
    main()
  File "./gmail.py", line 85, in main
    status, count = slurper.init_folder(imap_folder)
  File "/Users/William/github/Agile_Data_Code/ch03/gmail/gmail_slurper.py", line 68, in init_folder
    status, count = self.imap.select(folder)      
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/imaplib.py", line 649, in select
    typ, dat = self._simple_command(name, mailbox)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/imaplib.py", line 1070, in _simple_command
    return self._command_complete(name, self._command(name, *args))
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/imaplib.py", line 825, in _command
    ', '.join(Commands[name])))
imaplib.error: command SELECT illegal in state NONAUTH, only allowed in states AUTH, SELECTED

gmail.py - Handling of Unicode

I'm still getting the following error even though the slurp function already has handling of unicode error. The following is the error I'm encountering, which stops the read of my mailbox.

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

Is there an extra handling required at the slurp function?

ch03: gmail.py - error

I'd try run from ch03 as ./gmail.py -m automatic -u [email protected] -p 'my_password_' -s ./email.avro.schema -f '[Gmail]/All Mail' -o /tmp/test_mbox 2>&1
and have error message :
File "./gmail.py", line 7, in
from lepl.apps.rfc3696 import Email
ImportError: No module named lepl.apps.rfc3696

are there any advices?

resetting imap for ABORT while running gmail.py

Hello,

I am running this command to retrieve the gmail emails
python gmail.py -m automatic -u [email protected] -p xxxxx -s email.avro.schema -f '[Gmail]/All Mail' -o outputgmail

but I am getting:
Timeout exception occurred!
Warning: outputgmail already exists:
Warning: outputgmailtmp already exists:
Folder '[Gmail]/All Mail' SELECT status: OK
Folder '[Gmail]/All Mail has 63768' emails...

Connected to folder [Gmail]/All Mail and downloading 63768 emails...

resetting imap for ABORT
Folder '[Gmail]/All Mail' SELECT status: OK
Folder '[Gmail]/All Mail has 63768' emails...

IMAP RESET: OK 63768
resetting imap for ABORT
Folder '[Gmail]/All Mail' SELECT status: OK
Folder '[Gmail]/All Mail has 63768' emails...

(and it continues ...).

Any ideas of what's wrong? I have looked everywhere on internet and I have no clue of what's the issue.

Thank you.

AttributeError: 'GmailSlurper' object has no attribute 'imap'

I get the following error

(venv)vagrant@agiledata:~/agiledata/book-code/ch03/gmail$ ./gmail.py -m automatic -u [email protected] -p xxxxxxx -s ./email.avro.schema -f '[Gmail]/All Mail' -o /tmp/test_mbox2
Timeout exception occurred!
Traceback (most recent call last):
File "./gmail.py", line 104, in
main()
File "./gmail.py", line 85, in main
status, count = slurper.init_folder(imap_folder)
File "/home/vagrant/agiledata/book-code/ch03/gmail/gmail_slurper.py", line 58, in init_folder
status, count = self.imap.select(folder)
AttributeError: 'GmailSlurper' object has no attribute 'imap'

ch09/classify.py source code is missing

Can you please help upload? Thanks.

rjurney / agile_data_code Goto Github PK

agile_data_code's People

Contributors

Stargazers

Watchers

Forkers

agile_data_code's Issues

include <snappy-c.h>

-----------------------------------------------

New Logical Plan:

-----------------------------------------------

-----------------------------------------------

Physical Plan:

-----------------------------------------------

--------------------------------------------------

Map Reduce Plan

--------------------------------------------------

-----------------------------------------------

New Logical Plan:

-----------------------------------------------

-----------------------------------------------

Physical Plan:

-----------------------------------------------

--------------------------------------------------

Map Reduce Plan

--------------------------------------------------

Recommend Projects

Recommend Topics

Recommend Org