Comments (3)
I'm working on composing a solution for this. A question in the meantime: if the result under- or overflows, what behavior is needed to match spark-rapids?
E.g. for floating -> decimal, should we return one of 0 / INT_MIN / INT_MAX? for decimal -> floating do we set 0 / +-inf? Or do we null the field entirely?
from cudf.
The code in the new conversion PR has been modified so that the cuDF-specific code wraps around the core of the conversion algorithm. In spark-rapids-jni, we can similarly wrap around this core to perform the spark-specific rounding that we need.
This cuDF draft PR has the code for the spark-specific rounding that we need, wrapping around the core of the cuDF conversion code. This code is just in cuDF for my ease of testing, and should be migrated to spark-rapids-jni for full integration.
from cudf.
The cuDF conversion PR has been merged.
from cudf.
Related Issues (20)
- [BUG] compute-sanitizer failure on `gtests/PARQUET_TEST` preventing its use in code reading parquet files HOT 7
- [FEA] Add integration testing of cudf.pandas
- [BUG] Illegal Memory Access w/ New cuDF Polars backend
- [FEA] Add testing to help defend against thread-safety errors
- [QST] Cannot dlopen some GPU libraries [can't find cuda driver]. Is this a lower linux kernel version issue? HOT 2
- Update update-version.sh for pylibcudf
- [FEA] HOT 1
- Consider changing the `column_metadata` expectations when converting list types to arrow
- [FEA] Accelerate CI/CD by building and testing only the changed modules HOT 1
- [BUG] Method called on moved-from object (json.hpp:723:73)
- [BUG] When finding `testing` then `enable_language(CUDA)` is called from the CMake config
- [BUG] cudaErrorIllegalAddress while copying values to nullable string series HOT 1
- [FEA] Consider default nullability setting when converting cudf data to arrow C Data interface HOT 4
- [FEA] Enforce unsigned integer codes for cudf Python `CategoricalColumn`
- cudf::rank not passed enough parameters in list/struct rank benchmarks
- [BUG] TDIGEST_MERGE group by aggregation scales very badly HOT 1
- [BUG] low memory json reader running into an `OverflowError` HOT 2
- [FEA] Allow to run groupby/reduction with externally derived aggregations HOT 1
- [BUG] `cudf.testing.assert_*_equal` raises AssertionError for equivalent `DecimalDtype`d objects HOT 1
- [FEA] Please expose `int2ip` as a public method
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cudf.