does pygdf store data in vram? how to clean data in Vram? about cudf HOT 4 CLOSED

pygdf still requires data to fit in GPU memory. Our out-of-core solution will be provided by https://github.com/gpuopenanalytics/dask_gdf, which uses pygdf.DataFrame as a building block with an interface similar to dask.dataframe. However, dask_gdf is in earlier stages than pygdf and the docs are a bit lacking at the moment.

Pygdf has a delayed memory deallocation machinery inherited from numba. When a DataFrame is no longer retained by any reference, it goes into a pool for pending deallocation. The memory is deallocated as soon as there are 10 pending deallocation, pending deallocation holds on 20% of total gpu memory, or the numba gpu allocator observes a CUDA_ERROR_OUT_OF_MEMORY.

from cudf.

Thank you for your answer, so is pygdf not suitable for big data analisis? should I use dask_gdf for big data analisis? How does dask_gdf work?

from cudf.

Since big-data analytics is a big umbrella term, I will use the term out-of-core for your usecase where the data cannot fit in ram. We designed pygdf and dask_gdf such that pygdf handles in-core operations and dask_gdf handles out-of-core operations. Recall that dask_gdf uses pygdf as the building block, so it is possible to put pygdf in another out-of-core execution framework (i.e Spark).

For your usecase, you might be more interested in dask_gdf. To learn about dask_gdf, it good to start with learning about dask and the dask.dataframe at http://dask.pydata.org/. The dask_gdf is just dask.dataframe that uses pygdf.DataFrame as the building block instead of the pandas.DataFrame. We currently don't have a documentation page for dask_gdf but we are working on it.

Lastly, pygdf and dask_gdf are both in alpha stage. We encourage early adopters to experiment with them and give us feedback. But, it is not ready for production deployment yet.

from cudf.

thank you for your answer. I will try it. looking foward to see the final stage of this project

from cudf.