Comments (2)
TomAugspurger
commented on August 23, 2024
Thanks. It might be a while before I can look at this.
_check_dsk is verifying things about your task graph. It looks like this is saying you have multiple layers that use the same key. I'm not sure if you're making your own layers / keys, but if you are you'll want to use dask.base.tokenize on all the arguments going into whatever that function is.
from dask-ml.
Dan-Yeh
commented on August 23, 2024
Thanks for your reply.
Yes, _check_dsk has a side effect of checking an overlapping key in the task graph.
But it's the nature of this computation.
Task Graph and Details of Layers
The items of layers from the output
Layer: empty_like-848136d20f353a621c3fb6120e93a97c
('empty_like-848136d20f353a621c3fb6120e93a97c', 0, 0)
Layer: setitem-af567baa0d133fb2414b22009feedc71
('uniform-92c494621cf5b5df81f5fc85445fdfbf', 0)
('sin-b72ce9449b1942fdf9eade38a01fb89e', 0)
('setitem-af567baa0d133fb2414b22009feedc71', 0, 0)
Layer: setitem-c722836162c6db55933255d42e1117d9
('uniform-cd1ff3741c1350d3149e974c0a71b242', 0)
('setitem-c722836162c6db55933255d42e1117d9', 0, 0)
Layer: setitem-63195b1fdf99be7d4bf37a116ed79ef1
('mul-d7bb22edc92ff9263d91f04ac910d2d8', 0)
('uniform-92c494621cf5b5df81f5fc85445fdfbf', 0)
('sign-6d3aa1923db2461ffc8668e5d496fcbc', 0)
('cos-514e590b2a8b5ed03c3f5dc4551ff230', 0)
('sub-5397dd1b68f17458187bee3512c849ca', 0)
('setitem-63195b1fdf99be7d4bf37a116ed79ef1', 0, 0)
We could observe that ('uniform-92c494621cf5b5df81f5fc85445fdfbf', 0) was used in layer 2 and 4, which is the expected behavior.
Also check sklearn make_s_curve
If we only want to check if outcomes are deterministic, maybe it's ok to skip checking the graph overlapping here?
from dask-ml.
Related Issues (20)
- Mistake.
- Add backward compatibility for supported version of scikit-learn
- Bug in ColumnTransformer HOT 2
- HashingVectorizer behaves differently from FeatureHasher HOT 1
- sklearn handles text labels differently than ml_dask on OneHotEncoding
- Import dask_ml with python 3.10 failed due to conflict with dask.distributed HOT 4
- Python 3.11 support HOT 2
- LogisticRegression.score returns an empty dask array
- Incremental does not handle dask arrays of ndim>2 in estimator training HOT 2
- loading dask_ml gives error contextualversionconflict with sklearn HOT 4
- For a single record data frame train_test_split() sometimes assigns this single record to test set. HOT 2
- The `log_loss`-function crashes when using mixed types
- Area under the receiving operating characteristic curve (AUROC) calculation. HOT 2
- The latest version doesn't support perceptron model
- sklearn StandardScaler vs dask StandardScaler. HOT 1
- Nearest Neighbors
- `TypeError` when predicting non-array data with `dask-expr` HOT 6
- Undeclared runtime dependency on setuptools HOT 1
- Documentation on PCA expected max memory usage HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dask-ml.