[DISCUSS] Any ideas about unified memory management for different native engines? about gluten HOT 5 CLOSED

Do you mean to use Spark's memory management system? Then we need to define a set of APIs for native library where they can be implemented based on native implementation. when the library allocate memory it needs to notify spark's memory management system. This is really important part for spark. In Gazelle engine we spent quite much effort to implement this. We can port them to Gazelle-jni.

Thank you for your reminder, Let's add to readme.

from gluten.

Do you mean to use Spark's memory management system? Then we need to define a set of APIs for native library where they can be implemented based on native implementation. when the library allocate memory it needs to notify spark's memory management system. This is really important part for spark. In Gazelle engine we spent quite much effort to implement this. We can port them to Gazelle-jni.

Thank you for your reminder, Let's add to readme.

Hi @FelixYBW yep I think it's a critical part for this project. One more thing I have wondered is if this memory management is same in different engines. As you say, if users choose Gazelle engine, memory management will work well because you have defined-well API and spent much effort for it. But if users choose velox or another engine, since it may implement unified memory management by other ways instead of theses APIs you defined, does users have to do many works to unify memory management in this Spark-gazelleJni-mayAnotherEngine system?

In other words, my question is how to unify memory management API in different native engine?

from gluten.

It may depends on the native library. In theory we can implement a memory pool then every native engine use this pool to allocate large memory block. The pool register memory to Spark's memory management service. Currently Gazelle's one is enhanced from Arrow's. Let's see if it's possible to pull it out for general use.

from gluten.

Now we use [OPPRO-174] #264 to unify the memory management

from gluten.

We already implemented the memory allocation interface. Every memory allocation needs to check if Spark has enough memory for this task thread. Now both Arrow and Velox memory are allocated using the interface. Clickhouse backend is still WIP.

from gluten.