Giter VIP home page Giter VIP logo

hive-udfs's People

Contributors

danielmuller avatar k-kawa avatar

Stargazers

avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

k-kawa devonjones ysgit mambistats tmrvivian ncnarayanakumar certrialabs garyhuang-study

hive-udfs's Issues

UDF gives IOException after 63 ips

I've 30k+ ips in my hive table.
I am using lite free databases. UDF successfully executes till 63 rows.

When i try to execute with ip of 64th row.
SELECT geoip('64.233.173.231','COUNTRY_CODE','/home/dhruv/hive_loganalysis_udf_jars/GeoLite2-City.mmdb');
returns
FAILED: UnsupportedOperationException null

Can't load GeoLite2-City.mmdb from hdfs

I'd like to load the GeoLite2-City.mmdb file from HDFS but hive-udfs can't read it because it's not clear what the file path is. The only way I can get it to work is execute 'list files', copy the tmp directory location then use that in the function.

hive> ADD jar hdfs:///resources/jars/hive-geoip-udf-0.1-SNAPSHOT.jar;                                                                                                        [4/1829]
converting to local hdfs:///resources/jars/hive-geoip-udf-0.1-SNAPSHOT.jar
Added [/tmp/0fd54f8d-e3eb-4cfe-823f-8d1a0ce7c13a_resources/hive-geoip-udf-0.1-SNAPSHOT.jar] to class path
Added resources: [hdfs:///resources/jars/hive-geoip-udf-0.1-SNAPSHOT.jar]

hive> ADD FILE hdfs:///resources/data/geoip/GeoLite2-City.mmdb;
converting to local hdfs:///resources/data/geoip/GeoLite2-City.mmdb
Added resources: [hdfs:///resources/data/geoip/GeoLite2-City.mmdb]

hive> CREATE TEMPORARY FUNCTION geoip as 'com.spuul.hive.GeoIP2';
OK
Time taken: 0.537 seconds

hive> select geoip('8.8.8.8', 'CITY', 'GeoLite2-City.mmdb');
OK
Time taken: 1.258 seconds, Fetched: 1 row(s)

hive> select geoip('8.8.8.8', 'CITY', './GeoLite2-City.mmdb');
OK
Time taken: 0.165 seconds, Fetched: 1 row(s)

hive> list files;
/tmp/0fd54f8d-e3eb-4cfe-823f-8d1a0ce7c13a_resources/GeoLite2-City.mmdb

hive> select geoip('8.8.8.8', 'CITY', '/tmp/0fd54f8d-e3eb-4cfe-823f-8d1a0ce7c13a_resources/GeoLite2-City.mmdb');
OK
Mountain View
Time taken: 0.253 seconds, Fetched: 1 row(s)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.