Virtualization fails above 50k recods in IE about slickgrid HOT 34 CLOSED

If you think about it, a "virtualization" of the scrollbar is already taking place in the browser (because the scroll thumb won't go below certain size for usability reasons, there comes a time when the ratio thubSize:scrollSize is not equal to the ratio viewportHeight:containerHeight -- for very large container heights the thumb size would have to be sub-pixel to truly represent how high the viewport is relative to the container). I "feel" we should be able to apply the same mechanism as well. SlickGrid's advantage is that the rows are positioned using "top:value" style. So the problem really is how to compute the value.

Right now, the computation is pretty straightforward. You know the height of rows, you know the scroll position, so it's simple to compute what rows are in viewport and what is their offset in the scroll container.

Let's consider now the row is 30px high and we have 100k records. That means computed height of the scroll container is 3 million pixels. We agree that this is too much for IE. However, what if we set the scroll container height to only 1 million pixels and applied coeficient of 3 (3mil/1mil) to all computations? I.e. if the reported scroll position is 500k, for us it would be 500k * 3 (to determine what rows to display).

I think there are two important scenarios: 1. scroll position changes, grid needs to display items at the approximate location the scroll thumb represents (as I said above, for large heights the thumb already only approximates position); 2. grid changes displayed records (e.g. cursor moves out of viewport, or clients calls function to bring a record to view), scroll thumb needs to be moved to a position that approximates position of the displayed items.

I (again) feel the 2nd scenario is more difficult and would probably require "scroll position offset". But I really need to build a working prototype because I'm just guessing at this point :-).

from slickgrid.

The problem isn't the logic behind the virtualization and not even the implementation.
Let me clarify what I meant there.
There are 2 reasons why other similar virtual rendering grids that have a separate scrollbar control driving the canvas aren't nearly as fast and as smooth as SlickGrid. The first is that continuously updating absolute positions of multiple elements is slow. The second is that browsers throttle firing the onscroll event for the scrolled content used to create the scrollbar we want. The result is a lag that you can see in the SlickGrid when you do synchronized horizontal scrolling of headers and cells. Some browsers, like Firefox, are particularly bad at that.

SlickGrid uses the fact that the browser can do bitmap caching on scrollable panes and can respond to user moving the scrollbar very quickly with minimal lag. All SlickGrid has to do is to step in at the right times and clean up old rows and put the new ones in the right place.

Virtualizing the scrollbar will negate that advantage. The workaround I was talking about involved breaking the canvas into 1M-pixel "pages" and virtualizing the transition between them while still using native scrolling while withing a page.

from slickgrid.

I see your point. The grid is nice and fast because you're allowing the browser to just "uncover" rows during scrolling, your main task being to make sure it has something to uncover (as opposed to moving things around using absolute positioning).

To preserve the speed, I don't really see a straightforward solution. Do you think the hybrid solution combining virtual pages and native scrolling within a page would work?

from slickgrid.

It's doable, but may get messy. Perhaps there are alternative routes there.
The limit seems to be in the CSS implementation. While you cannot set numeric style values to anything greater than 0x123456, the rest of the rendering engine doesn't have any issues with greater values (at least up to ~10M pixels).

For example, you can create 10 individual 1M-pixel-high elements and that will force the container to be 10M pixels high:

    var o = document.getElementById("mydiv");

    for (var i=0; i<10; i++) {
        var x = document.createElement("div");
        x.style.background = "yellow";
        x.style.height = "1000000px";
        o.appendChild(x);
    }

    // reports 10M pixels correctly
    alert(o.clientHeight);

(changing the loop limit to >10 breaks everything)

We can use this to force the height of the scrollable pane to correct values, but we still have the problem of not being able to set the "top" of the individual row to put it in the right place. Need to experiment with it some more.

from slickgrid.

Just for reference, another type of failure due to height overflow is documented in issue #23 (this time in Firefox 3.6+).

from slickgrid.

Have we checked that the Firefox 3.6+ issue can be resolved in the same way, by forcing container height with multiple children instead of CSS?

from slickgrid.

At this point, this is just brainstorming. How important is this anyway? I mean, I realize that it is, but what is the priority?

from slickgrid.

Bragging rights important? :-)))

"We are the only grid on The Net that can handle arbitrary row count" ;-)

There's always the pager you can fall back on. I would be a bit more comfortable if the cutoff point was in hundreds-of-thousands rows range (for 30px row height), but the 40k or so is ok. For me the priority of this issue is medium-low.

from slickgrid.

The priority for us is very high because we chose to use slickgrid precisely for its ability to handle "arbitrary row count". For us, "arbitrary row count" might go up to hundreds of millions of rows.

from slickgrid.

Can't quite imagine the scrollbar being useful at those row counts.
But I hear you guys. Will do some experimenting in the coming weeks.

from slickgrid.

I don't think the current implementation will ever be able to handle hundreds of millions of rows. Sooner or later, height limit will rear its ugly head in every browser.

from slickgrid.

Just pushed an experimental fix using the hack discussed above - http://github.com/mleibman/SlickGrid/commit/1c4860abcb9247011bb386b1e665f66e862b2aa8

There is still an upper limit there that varies with browsers. Chrome fares the best. IE8 breaks somewhere around 200K rows. FF is the weird one here. In the formatting example, it handles 1M rows just fine, but breaks on 400K. I suspect there is an internal conversion to floating point in position calculations there.

All-in-all, this does raise the upper limit considerably, but it still breaks down pretty quickly.
David, I'm interested in hearing your thoughts on how you plan to present the 100M-row grid to the user and still maintain any sort of reasonable navigation. The way I see it, you'd be using the scrollbar mostly for rough/relative navigation since the precision there would be extremely low (1px = hundreds of thousands of rows). The only remaining useful functionality there would be page up/down and row up/down using the scrollbar arrows. With that in mind, a combination of a graphical pager component (ex. http://www.extjs.com/deploy/dev/examples/grid/progress-bar-pager.html) and a native scrollbar for in-page navigation would work much better. You can still have a relatively large page size - maybe around 10K or 100K rows.

Thoughts?

from slickgrid.

Another good example of a slider pager - http://www.extjs.com/deploy/dev/examples/grid/sliding-pager.html

from slickgrid.

Is the 200K/400K limit even after the workaround? That's no good... I wonder if a third layer (divs of divs of divs) might work :)

You've understood my use-case perfectly: the grid is so that users can just navigate the data roughly. I'd guess the most useful regions are the beginning and the end, but we still want to provide a grid that supports the whole data.

As for pagers and sliders, the reason we chose to use slickgrid in the first place was to avoid the use of pagers and sliders.

from slickgrid.

Other than virtualizing scroll bar, it isn't possible to guarantee unlimited range of a virtual grid. Sooner or later browser height limitations will be encountered. And in case of SlickGrid virtualization of scroll bar is not easy/possible as it is using scrollbar offset to position rows before/after current viewport so that they are just "uncovered" by browser as you scroll down.

from slickgrid.

I think I have a solution - http://jsfiddle.net/SDa2B/
It's going to take a while to integrate this into SlickGrid, and there are areas that need to be figured out still, but I believe this does solve the problem.

Comments?

from slickgrid.

Added comments:
http://jsfiddle.net/SDa2B/4/

from slickgrid.

Implemented in a separate branch for now.
http://github.com/mleibman/SlickGrid/tree/unlimited-rows

Please test.

from slickgrid.

Branch merged in. Closing :)

from slickgrid.

Wanted to say thank you for this! Even six years later, this proved useful. I ported this chunk of logic over to React and it's working very nicely. https://github.com/Extensis/slick-scroll

from slickgrid.

Just looked at the fiddle also, and works brilliantly, even after six years 😄

@quicksnap you can overcome the Safari momentum scrolling issue by adding pointer-events: none; to the content/rows during scrolling.

from slickgrid.

@JouniK Holy crap, I'm going to try that out! Thanks!

from slickgrid.

It works! Thank you so much @JouniK!

from slickgrid.

http://jsfiddle.net/SDa2B/263/

from slickgrid.

hi, can i build sclickgrid programmatically?

from slickgrid.

There is an npm package: npm install slickgrid

But see the manifesto

from slickgrid.

i want to create slickgrid with my data but the scroll is horizontal

from slickgrid.

or this http://jsfiddle.net/SDa2B/263/ any example of horizontal scroll?

from slickgrid.

don't think that's possible. SlickGrid uses the browser built in scroll bar.

If you still want to use SlickGrid my repo is current, check out the readme. this repo is no longer active. download the zip package. you should be able to run the examples (in the examples folder) locally. if you still have problems, please open an issue there.

from slickgrid.

i will open new issue for example scroll. what i can do that?

from slickgrid.

scroll horizontal i mean

from slickgrid.

open an issue in my repo.
but i don't know what you mean by horizontal scroll, you'll need to explain that. If you mean using a horizontal scroll bar to scroll through data, you definitely can't do that with any version of SlickGrid.

from slickgrid.

@6pac Horizontal scrolling is also fast and done very well in SlickGrid http://mleibman.github.io/SlickGrid/examples/example-spreadsheet.html

from slickgrid.

@thadguidry sure, that's scrolling through column data horizontally, using the browser's built in scrolling. what i meant is, you can't scroll vertically through the row data, using a horizontal scroll bar. that's how i interpreted this request. i might be wrong :-)

from slickgrid.