Comments (6)
Thanks for the detailed issue @pgr. I can confirm the error on my own machine.
Interestingly, I just made an issue today that will resolve this: #4. The problem is the data is in sorted order.
import random
random.shuffle(problem_data)
td = tdigest.TDigest()
td.batch_update(problem_data)
works fine. I'll add this as an additional constraint/problem to be addressed in #4.
from tdigest.
Oh wait, I was wrong; the data is not sorted. Strange then that the shuffle
solves the issue...
from tdigest.
I can narrow it down to just this data: https://gist.github.com/CamDavidsonPilon/2357b9d1b2d1401640fc
from tdigest.
K got it. It was because I was using an unbalanced binary tree in bintrees: https://github.com/CamDavidsonPilon/tdigest/blob/master/tdigest/tdigest.py#L4
If ordered data was passed in, as it was in both our examples (a subset in yours), then the tree becomes very lopsided. When we try to call iter_items
on the tree, we hit the upper limit of 64 depth (hardcoded in bintrees as MAXSTACK
). You can see the calls here: https://searchcode.com/codesearch/view/55567668/
Solution is the change to FastRBTree
in bintrees. I tested this and it works on our data. Only downside is is that it will be slightly slower. I feel this is a fair tradeoff.
from tdigest.
Fixed in latest commit!
from tdigest.
Confirmed, cheers!.
from tdigest.
Related Issues (20)
- Sometimes a test fails (test_uniform)
- tdigest.__version__ returns wrong version
- tdigest fails to serialize in spark
- Horribly slow on PyPy HOT 2
- Negative trimmed mean
- difference between java implementation and python HOT 8
- Updates to algorithm in latest paper [help needed] HOT 3
- wheel incorrectly contains `.pyc` files HOT 3
- Not able to install python package HOT 1
- CDF benchmark HOT 1
- well done HOT 1
- Is the code still maintained? HOT 1
- python2.7 ypeError: unbound method _cython_3_0_0a9.cython_function_or_method object must be called with RBTree instance as first argument (got AccumulationTree instance instead)
- pyspark example misses import
- Corner case where values are identical? HOT 1
- Best way to serialize and load HOT 1
- Scale function
- Tag request for git project: tdigest 0.5.2.2 HOT 1
- Conda Distribution
- Alternative faster t-digest implementation HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tdigest.