Comments (2)
I haven't made any effort around profiling and optimization, so the lousy performance you're observing isn't surprising. (In all my use cases, Hive/Presto/Hadoop do all the heavy lifting, and I'm just using Python for some light weight post-processing.)
Some factors:
hadoop fs -cat
is compiled Java code instead of Python. I wouldn't be surprised if the python Hive client was CPU bound.hadoop fs -cat
just dumps the raw byte stream instead of doing things like serialization, decompression, etc.
All that said, large transfers might be better satisfied with Hive queries / MR jobs / distcp.
from pyhive.
OK, thanks for the response and the work on this library. If I find anything I'll let you know.
from pyhive.
Related Issues (20)
- kyuubi support HOT 3
- Project is currently unsupported? HOT 1
- pyhive.exc.DatabaseError MISSING_CATALOG_NAME
- Connection Error(TTransportException) to Hive using SQLAlchemy API
- Either set supports_statement_cache to True or False HOT 4
- precision and scale not supported for DECIMAL type
- Security Vulneratbility CVE-2022-40899 on future HOT 1
- Incompatibility with SQLAlchemy 2.0.0+ HOT 10
- TypeError: not all arguments converted during string formatting
- Unable to connect to Azure HDinsight Hive DB using PyHive python
- Release 0.7.0
- PyHive, Presto connector returning wrong resultset HOT 3
- metadata.reflect() not working causing `DESCRIBE default` because of possible bug in get_table_names() HOT 1
- Release of 0.7.1 HOT 1
- UnboundLocalError: local variable 'i' referenced before assignment
- Error connecting to Kerberos using pyHive
- Python 3.11 TTransportException: Could not start SASL: None of the mechanisms listed meet all required properties
- conflict with python12
- how to manually stophive task
- hive.Connection TTransportException
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyhive.