Comments (5)
Do you mean to use Spark's memory management system? Then we need to define a set of APIs for native library where they can be implemented based on native implementation. when the library allocate memory it needs to notify spark's memory management system. This is really important part for spark. In Gazelle engine we spent quite much effort to implement this. We can port them to Gazelle-jni.
Thank you for your reminder, Let's add to readme.
from gluten.
Do you mean to use Spark's memory management system? Then we need to define a set of APIs for native library where they can be implemented based on native implementation. when the library allocate memory it needs to notify spark's memory management system. This is really important part for spark. In Gazelle engine we spent quite much effort to implement this. We can port them to Gazelle-jni.
Thank you for your reminder, Let's add to readme.
Hi @FelixYBW yep I think it's a critical part for this project. One more thing I have wondered is if this memory management is same in different engines. As you say, if users choose Gazelle engine, memory management will work well because you have defined-well API and spent much effort for it. But if users choose velox or another engine, since it may implement unified memory management by other ways instead of theses APIs you defined, does users have to do many works to unify memory management in this Spark-gazelleJni-mayAnotherEngine system?
In other words, my question is how to unify memory management API in different native engine?
from gluten.
It may depends on the native library. In theory we can implement a memory pool then every native engine use this pool to allocate large memory block. The pool register memory to Spark's memory management service. Currently Gazelle's one is enhanced from Arrow's. Let's see if it's possible to pull it out for general use.
from gluten.
Now we use [OPPRO-174] #264 to unify the memory management
from gluten.
We already implemented the memory allocation interface. Every memory allocation needs to check if Spark has enough memory for this task thread. Now both Arrow and Velox memory are allocated using the interface. Clickhouse backend is still WIP.
from gluten.
Related Issues (20)
- [VL] Can't run spark.plugins=io.glutenproject.GlutenPlugin with release jar HOT 2
- [CORE] Shouldn't pushdown undeterministic filter to scan node
- [VL] Add CPP unit test for UDF HOT 3
- [VL] No extra SortExec needed before WindowExec if window type is `sort` HOT 6
- [VL] memory leak when running task HOT 2
- [CH] Allow non-const arguments for replaceAll and replaceRegexpAll HOT 1
- [VL] Performance tunning for facilities in Iterators.scala
- [CH] It's easy to occur OOM in window functions
- Minor cleanup for BasicScanExecTransformer.scala
- libvelox.so issue while running Spark tpcds/tpch workload with Gluten
- [VL] Results are mismatch with vanilla Spark when use regexp_replace('a{bc', '\\{', '\\[') HOT 5
- [VL] Results are mismatch with vanilla Spark when using unix_timestamp(yyyy-MM-dd HH:mm:ss,SSS)
- Deperate isTransformable API from TransformHitRule
- [CH] Crash on ExpandTransform::work()
- [CH] Diff result for upper('ır') HOT 1
- Rename transform-hint to fallback-hint HOT 2
- [CH] Support config native local engine default logger level from system properties
- [CH] Disable GlutenSortShuffleSuite in clickhouse backend.
- [VL] Using Velox udf will fail with yarn-client mode when driver and executors are not on the same host
- [CH]mismatched headers in broadcast join HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gluten.