Comments (2)
itamarst
commented on June 20, 2024
1
OK, I have this mostly working, will finish up and do PR on Monday.
from polars.
itamarst
commented on June 20, 2024
Based on the above, it seems like the best initial approach is probably:
- Create a wrapper global allocator for Rust, that wraps jemalloc/mimalloc depending on platform, and registers memory with tracemalloc.
- Use this wrapper only for dev profiles (doing it in release would also be nice and probably low overhead, but requires benchmarking so probably out of scope for this issue).
- Add some PyArrow utility.
- Write tests for the above, to make sure they're actually tracking things!
- Write test infrastructure that allows asserting peak memory doesn't get above a certain level.
- Write a demonstration test focusing on
read_parquet().
from polars.
Related Issues (20)
- Incorrect result from projection_pushdown HOT 6
- np.NaT should get converted to null when converting from NumPy HOT 2
- Panic when building a Series of lists without an inner dtype
- Why two struct series with different type are equal?
- Upsample fills time_column but not groups HOT 5
- Setup CodSpeed for PR benchmarks HOT 1
- polars' maximum length reached. Consider installing 'polars-u64-idx' with a dataframe of 12K rows HOT 3
- join_asof should allow for strict inequalities on the "on" column
- Inconsistent result from comm_subplan_elim HOT 2
- Polars write_database Connection Issue with Redshift: Unrecognized Configuration Parameter HOT 5
- Support `group_by` on columns of type `List(String)` HOT 3
- Ability to disable jemalloc HOT 3
- `.reshape((-1, -1))` causes PanicException HOT 3
- Schema contradiction with polars.LazyFrame.fill_null(matches_supertype=True)
- Regression in common subplan elimination when cached node is used HOT 2
- Python Polars Import Fail on Aarch64 Linux HOT 6
- [DRAFT] Predicate pushdown broken by lazy `pl.concat` HOT 12
- Regression after 0.20.19: vec_hash_combine operation not supported for dtype list[u8] when joining with secondary column on list[numeric] HOT 1
- Implement window context in `df.sort(col("a").over("b"))`
- Be able to add negative values in dt.offset_by wherever in the string HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from polars.