This is clearly stated in the documentation, but perhaps it needs to be made clearer with a few examples. One possible gotcha here is that you cannot always mix string and number types, since 1 <= "a" and "a" <= 1 both return false (the "a" is always coerced to a number and becomes NaN). You can of course mix types if the strings coerce to non-NaN numbers.

Despite adding support for NaN values in version 1.1.3, mixing types will still produce unexpected results. I'm now tempted to revert support for NaN values in version 1.2.0 for a few reasons:

• Since we require values to be naturally-orderable anyway, supporting NaN is an exception to this rule.
• Supporting NaN values causes other problems, since Crossfilter has an inherent concept of natural ordering, which allows fast filtering of ranges using binary search (bisect). Although we now artificially sort NaN values to the end, it is still not possible to filter for NaNs, and moreover, top(k) somewhat unexpectedly returns NaN value records first (because they are sorted to the end).
• The support for NaN or undefined values was intended to avoid catastrophic failure in this situation, but as the same catastrophe happens for mixed types, it seems better to focus on documentation rather than work around it. It would certainly be possible to work around the issue for mixed types, but it seems like this would just be putting off the inevitable: the user needs to fix their value accessor.

I'm looking at the example on http://square.github.com/tesseract/ and thinking of different ways I'd like to be able to filter the data. Is there any possible way to filter on the flight cities? Let's say I want to just see data for flights from PHX to ONT. Or perhaps all flights that have a destination of SAN.

Of course we could filter the data that is returned from the server but since I already have all the data loaded client side it would be nice to be able to do this type of filtering client site.

Any thoughts/ideas on this?

I can't seem to find a way to properly destroy a crossfilter object.
I have a single page application that allows a user to select new data sets.
When the data is retrieved, I would like to be able to destroy the previous crossfilter object, to allow garbage collection, and create the new one.

Am I barking up the wrong tree here?

In "Example: Airline on-time performance" the first element in the flight table is out of order in some cases. For example, if you set the upper limit in the date filter to March 11 the top row is at 6:55 AM but is preceded by a row at 11:45 PM which is later.

Any chance you could add min and max properties to the dimension? Since you already have the bounds, i'm imagining getting the ends of the values using the bounds values should return the right min and max respectively, right?

After few failed attempts to read the crossfilter code in one sitting, I realized that a "Technical overview" wiki page could be immensely helpful.

In the documentation it states

a grouping intersects the crossfilter's current filters, except for the associated dimension's filter. Thus, group methods consider only records that satisfy every filter except this dimension's filter. So, if the crossfilter of payments is filtered by type and total, then group by total only observes the filter by type.

What is the reasoning behind that and what is the way around it?

I've been playing with Crossfilter for a bit, working with my own data in the flights sample. One of the features I needed is unions on filters, so I tried updating it to use the latest version of Crossfilter (both master and 1.2.0) but noticed some data sneaking past the filters.

I just tried the same with the original gh-pages sample, just replacing the crossfilter.v1.min.js with the latest crossfilter.min.js (afaik there haven't been any breaking API changes?) as my own data is private, and it was also broken.

Here is 2 screenshots showing obviously wrong results.

Chrome 27.0.1425.0 dev, OS X

Hello,
is it possible to select multiple regions in the diagram with crossfilter?

Thank you in advance.

Mic

The bar widths of crossfilter seem to mess up (i.e. are much larger than usual) if you're trying to use a smaller range of values (e.g. 0 to 1 or -1 to 1)

Any way around this?

Is it possible to use the reduce function to find the max of the group? It seems the problem would be that when records are removed, if the record removed happened to be that contributing the current maximum, you have have to re-run the the map/reduce in order to find the new maximum.

Is there a way to force the map/reduce to happen again?

In the documentation it states

a grouping intersects the crossfilter's current filters, except for the associated dimension's filter. Thus, group methods consider only records that satisfy every filter except this dimension's filter. So, if the crossfilter of payments is filtered by type and total, then group by total only observes the filter by type.

What is the reasoning behind that and what is the way around it?

Hey, I'm really sorry, I think I accidentally deleted the wiki page.

I came across some difficulties tracking updates to the charts and filters when linking crossfilter with other apis like leafjs and google maps. I tried different workarounds, but I guess a built in event listener model would be best.

Probably the following events would be best:

• init
• filter update
• filter reset

It may be possible to limit the listener to a specific dimension and get updates only in case it is affected by the filter.

Do you guys think this is reasonable or is there already a way to get those events?

See here: http://code.google.com/p/tesseract-ocr/ and also here: http://news.ycombinator.com/item?id=3724592

Hi again, would it be possible to create a running total chart based on reducedSum as shown below?

http://pixeltradr.com/crossfilter/runningTotal.png

I'm attempting to combine a crossfilter chart with a force directed graph, however, the existing examples and documentation don't clearly explain how to best access the filtered data values.

In my example, I have a long list of links that are ordered by time. Given some selected time range, I'd like to then plot a force directed graph using only the selected links and nodes reachable by the links. Based on the current documentation and examples found, my best guess would be to do the following:

var linkFilter = crossFilter(linkList),
       linkDimension = linkFilter.dimension( function(d) { return d.time; } ),
       linkGroup = linkDimension.group();

// Attach a force directed graph to an svg element and register it with the rend method that calls plotGraph.

function plotGraph(div) {
    val links = linkDimension.top(Infinity),
           nodes = acessibleNodes(links);

    // Update force graph with current links and nodes.
} 

This appears to be how the barchart method is updating itself upon each update, however this approach becomes significantly slower as the filtered region grows to include more data points as each filtered region requires essentially passing through all the data points in that group. Is there a more efficient way to update the links and nodes in a graph? Is there something somewhat similar to enter and exit in d3 that allows one to make modifications using only the items in the filter that entering or exiting the current filter?

As an example of what i'm trying to do and having problems with, i've included a minimal copy in this jsFiddle.

Hi there, congrats on an awesome charting plugin.

I was wondering if it was possible to create a graph with flight origin name (string) on the X axis
as shown in the image below.

http://www.pixeltradr.com/crossfilter/crossfilterTest.png

Ive managed to chart the data using a new column - flight origin ID for each flight name,
however i would prefer the graph to show the flight origin string name instead of a number on the X axis

http://www.pixeltradr.com/crossfilter/

I have a single crossfilter with various charts that the user can configure dynamically in javascript. Each chart contains a crossfilter group with an appropriate map/reduce function. As they select different metrics and group by options, the old group is discarded and a new group is created with a call to:

crossfilter.group()

This works great, except for one problem - as each group is constructed, listeners are added to the crossfilter:

  // The group listens to the crossfilter for when any dimension changes, so
  // that it can update the associated reduce values. It must also listen to
  // the parent dimension for when data is added, and compute new keys.
  filterListeners.push(update);
  indexListeners.push(add);

However these listeners are never removed. So predictably, groups add up, and the app quickly becomes slow to respond as each filter is updating hundreds of zombie groups that are not being used by any controls, calling reduce.add() or reduce.remove() on them.

Also, since the group is added to the crossfilter, we have a circular reference and these groups are never garbage collected.

Not good.

Solution:

The group class should have a some kind of method to mark for destruction - remove(), destroy(), etc.

I'm not intimately familiar with the code, but it seems all that this needs to do is remove it's update & add from crossfilter.filterListeners[] and dimension.indexListeners[] collections on the crossfilter. Voila, no more events firing to update zombie groups, and the objects will be removed from memory.

For groupAll the crossfilter.dataListeners[] needs to be cleaned up.

On https://github.com/square/crossfilter/wiki/API-Reference a number of headings are being displayed as raw text, instead of being correctly formatted with markdown.

e.g., # crossfilter.**add**(*records*)

This may be simple - but I'm new to d3, but how can I modify the crossfilter example to add y-axes with updating axes values? I've managed to add y axes with the following additions to the chart function:

var yaxis = d3.svg.axis()
.scale(y)
.orient("left")
.ticks(5)
.tickFormat(function (v) { return v / 2; });

and

g.append("g")
.attr("class", "yaxis")
.call(yaxis)

This works fine, I get get axes on all the chart with the correct initial values. However, when the data is filtered the y axis tick values are not updated to reflect the data on the filtered charts. Any help on this would be very much appreciated.

Thanks :-)

I was interested in seeing if I could extend the Crossfilter example to render other chart styles than bar charts. Since Crossfilter is built upon D3.js, I thought it might as well take advantage of the different visualization styles available through d3, such as line charts, pie charts, tree graphs and chord diagrams.

I was able to modify the example js to render the charts as line charts:

http://i.imgur.com/VJ16O.jpg

Unfortunately, the charts redraws as opposed to replacing the previous data when filtering:

http://i.imgur.com/WEU4N.jpg

The code is available here: https://gist.github.com/3350666

I believe the problem is with how I am attaching the data to the svg:

g.append("svg:path").datum(group.all()).attr("d",linePath(height,width,group.all())).attr("class", "data3");

I had to pass the group as a param in the linePath method as the datum wasn't accessible.

Any suggestions would be greatly appreciated.

Awesome library!

I believe that there is an inconsistency since filter(null) and filterExact(null) do not produce the same result.
Although it is explicitly documented that filter(null) calls filterAll(), I believe this should be reconsidered. Calling filter with any other scalar value (string or number) will call filterExact.
It makes more sense that calling filter with no range at all, or passing undefined, would call filterAll. And calling filter with a null value calls filterExact with null.

This simplifies code when attempting to use filter() for all value types, instead of having to do the checks for the one exception.

Additionally, filterRange (with an empty array) should call filterAll()
Instead it filters out everything.

Am I mistaken for thinking about this with a "filter" mindset, given that the functions are named "filter"?
Or should I be thinking more in an intersection of values? Where an intersection with an empty set is also an empty set.

I am trying to extend the flightdata example with day-of-week chart and eventually ordinal charts. I suppose this is a rather obvious next step... are there any working examples of this?

I have managed to do a simple day-of-week chart using Date.getDay() with a 0-6 linear range, but this looks quite lame of course. It would be nicer to have an x-axis with actual Monday-Sunday labels, and a bit wider bars than for the dates. I do think brushing makes sense though for day-of-the-week.

Also I am looking for a working example of crossfilter with an ordinal or nominal scale, either with a bar or pie chart. This one should probably not use brushing but subsetting by clicking the individual bars.

Hi,

Not sure if this the right place - but has anybody done much work linking Cube and Crossfilter.

There was an offhand reference to it here - https://github.com/square/cube/issues/60

However, I can't seem to find much online, and I'm not sure what's the "proper" way to do this?

Cheers,
Victor

It seems that the sorting methods generate a maximum recursion error with big datasets. I'm testing with a dataset of about 50,000 author names.

The JavaScript console shows:

Uncaught RangeError: Maximum call stack size exceeded
insertionsort
sort
quicksort
sort
quicksort
sort
quicksort
sort
....

BTW Thanks for the great library!

There's obviously a bug in Crossfilter or D3 somewhere. In the example at

http://square.github.io/crossfilter/

when it loads it is supposed to show the top 40 ordered by time. Instead it shows Feb28, from 11:58PM down to 10:45PM (those 39 are correct) followed by an event at 05:01PM. This last one is clearly wrong, since if you apply filters to choose just Feb28 and say Arrival Delay >100 then the data shows that there are other times on Feb 28 , e.g. 10:43PM down to 09:41PM that should have been in the original top 40 instead of 05:01PM.

It would be great if it was possible to not only add more data records to an existing tesseract, but also to be able to actually remove data again.

tesseract.remove() would remove all data records that remain with current filters applied. So after calling tesseract.remove() a call to tesseract.groupAll().value() would yield 0.

I do not know if the CS and Math behind tesseract allow for simple removing of data records, but there are plenty of practical use-cases for such a functionality.

Will be great to see a typescript definition file for crossfilter.
d3.js has a definition file at DefinitelyTyped

I've tested the example in IE7 & IE8 and both display the JS error copied below.

Webpage error details

User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)
Timestamp: Tue, 20 Mar 2012 16:06:30 UTC


Message: Object doesn't support this property or method
Line: 708
Char: 1
Code: 0
URI: http://square.github.com/tesseract/d3.v2.js


Message: Object doesn't support this property or method
Line: 210
Char: 1
Code: 0
URI: http://square.github.com/tesseract/

Would it be possible to launch crossfilter in a web worker to avoid slowing down the main process and freezing the UI ?

I tried but I get a DATA_CLONE_ERR: DOM Exception 25 error. I guess some data is stored in the DOM or in memory.

Regards,
David

Currently, NaN and undefined values are treated as if they are greater than all other values, thus sorting in ascending order puts them at the end of the internal values array. As a result, the current behaviour is:

• top(k): records with a NaN value come first (because the internal values array is traversed in reverse).
• bottom(k): records with a NaN value come last.

However, it might be more useful for records with a NaN value to always come last in both top(k) and bottom(k). For comparison, R’s order puts NA values last by default (they can also be made to come first, or ignored altogether).

On the other hand, perhaps it’s more consistent to maintain top(Infinity).reverse() === bottom(Infinity).

Related: group.{top,bottom}(k) don’t support NaN or undefined reduce values at the moment.

Missing values are quite common in some situations. The expected behavior would be to either always filter them, or classify as a special group. The current behavior however yields highly unexpected results.

Here a jsfiddle to illustrate this: http://jsfiddle.net/VgvCB/

It would be great to filter on multiple values at once i.e. analogous to SQL's IN (…). I guess it makes implementation a bit more complicated because you're no longer just looking for a single range at a time. Alternatively, I guess I could create one dimension for each possible value, and then compute the union afterwards (but instantiating so many dimensions might be expensive?).

In case you haven't guessed yet, I'm seeing how easy it is to implement multi-faceted drilldown. :)

I've encountered a situation where Tesseract appears to behave contrary to the API documentation. Quotes:

Dimensions are stateful, recording the associated dimension-specific filters, if any.

Note: a grouping intersects the tesseract's current filters, except for the associated dimension's filter. Thus, group methods consider only records that satisfy every filter except this dimension's filter. So, if the tesseract of payments is filtered by type and total, then group by total only observes the filter by type.

In a small test page I wrote (http://pastebin.com/VXnJuK9p, test data is at http://pastebin.com/yYjp1pC9), the first two log messages give the expected results: the first yields all rows in the data set, and the second yields a single row that contains the token, "Jimmy". Groovy.

However, the group() result that's stored in the 'links' variable yields all three distinct links in the data set. This result appears to be inconsistent with the API documentation, since only one row satisfies the filter on the 'token' dimension.

Would anyone else find it useful to have the 'top' method return a single value by providing an extra accessor function in its arguments? So
function top (k,accessor) {
...
array.push(accessor(data[j]));
}
Or is there a way to emulate this efficiently on the output of the existing 'top'? Sorry, still beginner at javascript, just wondering if I can adapt crossfilter to my intended usage without forking it.

This is not a real issue report, I apologize. I just wanted to find out if anyone can help me to understand how I'm suppose to work with filters.

Can you check the following code?
http://jsfiddle.net/Pp8jd/

Why won't filterExact never have any effect at all on the content in the types group? I'm always getting the same groups with the same count value.

I need to be able to filter data and have the results reflect the changed filters in the application I'm planing to create. So I don't understand how just filtering dimension members would allow me to do this.

In the current stable release of Chrome filters seem to simply be dropping parts of the set when multiple filters are applied and reset. After much digging in my current project I found the same error appearing consistently in the canonical ASA data set example.

To reproduce simply set all the filters and reset them. The result is as as follows – the number of selected flights varies, but never matches the total:

37,407 of 231,083 flights selected.

If you continue to filter and reset the groupAll sum changes, sometimes rising, sometimes falling.

Firefox, Safari and Opera still has completely sane behaviour.

Is it only me?

Unpacking the file and loading the HTML on Chrome or FFox doesn't show the plot.

Any ideas? Do we need a special trick to make it running?

Request: filter.load(json) and filter.save() - allowing a pre-built filter (including dimensions / groups etc) to be loaded from JSON representation.

As state variables are not explicitly separated from local variables in the code, I find it hard to judge whether the this be a herculean task or whether the sheer size of the data would negate the benefits of pre-building (e.g. nightly node builds)

I'm looking for some help on how to implement a solution to the following problem, as I think it may involve integrating some undocumented pull requests (such as 33 and/or 36).

I have some data of several entities that each have multiple months of multiple metrics, like so:
id, yearmonth, value1, value2, value3...value 18 (one for each metric)

I would like to separately and selectively filterRange each metric value for a selected month and simultaneously crossfilter a chart that displays metric trends for all months. For example, the user might select yearmo="Jan 13" and a range on value1 & value7 of [80,100]. The trend chart should show the average value trend line for all entities over all months whose value1 & value7 for Jan 13 range from [80,100] but whose other month values are unfiltered for all metrics.

Following basic examples and the core API, I can see how I would filter a range on each metric but that would cause the trend chart dimension to interpret it as a range that applies to all of the months simultaneously. What I really need is for the range to apply only to the selected month. I don't want to (and don't think I can, within the limits of crossfilter) set up one dimension per month/metric combination, since I'm dealing with 18 metrics and 24 months (432 combinations).

Unless I'm totally over-thinking this, I suspect it will take some combination of the above-mentioned pull requests, but I'm not sure how to use them short of reading the source code, and I want to come up with the most performance-minded solution possible within the confines of the library. I'm also open to data layout suggestions, such as unpivoting it, if that helps.

Any suggestions would be very much appreciated.

Crossfilter is great, but I can't seem to get it to handle non-factorial data. I have some dimensions with missing values because there is no data point there. I tried to set the reduce functions to "skip over" counting those missing values, but I get negative bars and other weirdness:
...
...group.reduce(function(p,v){
return p + 1_!isNaN(v.myField);
},function(p,v){
return p - 1_!isNaN(v.myField);
},function(p,v){
return 0;
});

Is here something I'm missing? Or will this just not work? Has anyone done this before?
Thanks!

Ive been working on having positive and negative bar charts, i came across an issue with negative numbers.

It seems using reduceSum like this with negative numbers or a combination of negative and positive creates unexpected results.

Positive Only - works as intended:
http://pixeltradr.com/crossfilter/indexPayments2ColorsNamesPositive.html

Negative Only - not working as intended
Should be exactly the same as positive, but with red bars. Instead bars break out of the y domain range

• (manually extending the svg chart height with firebug shows how far the bar go.)
  http://pixeltradr.com/crossfilter/indexPayments2ColorsNamesNegative.html

Positive and Negative - not working as intended: Same as negative only
http://pixeltradr.com/crossfilter/indexPayments2ColorsNames.html

After looking at crossfilter.v1.js I cant say I'm any closer to solving this, any help would be appreciated

Any plans on adding tesseract to the node npm registry?

Right now multi-value grouping can be handled outside of crossfilter using underscore like this:

# Group by week date and target
groupedData = _.groupBy(data, (d) ->
  "#{d.week_date}#{groupSeparator}#{d[target] or 'None'}"
) 

It would be nice if crossfilter could do something similar internally instead. This would be help dynamic, user-driven selections.

I have a dataset that has about 5200 records. I am filtering on a value that has 1500 distinct values ranging from 0-4000. When I filter on the data it always cuts off at 999. Is there some artificial limit to 1000 groups?

What about using a traditional Map/Reduce API like CouchDB for instance? This could also solve the problem with filtering multivalued properties.

var countries = crossfilter([
  {name: "USA", languages: ["Spanish", "English", "Chinese", "French"], continent: "North America"},
  {name: "Canada", languages: ["German", "English"], continent: "North America"},
  {name: "UK", languages: ["English"], continent: "Europe"}
]);

function map(d) {
  d.languages.forEach(function(lang) {
    emit(lang, d);
  }
}

countriesByLanguage = countries.dimension(map);
countriesByLanguage.filter('English');

countriesByLanguage.top(Infinity) // => [USA, Canada, UK]

If you'd rather want to search for language+continent the map function would look like this:

function map(d) {
  d.languages.forEach(function(lang) {
    emit([lang,d.continent], d);
  }
}


countriesByLanguageAndContinent = countries.dimension(map);
countriesByLanguageAndContinent.filter(['English', 'North America']);

countriesByLanguage.top(Infinity) // => [USA, Canada]

Actually what you'd get is flexible indices that you can use for fast lookups.
Obviously this breaks the current API, but I just wanted to drop that thought. :)

What do you think?

Would there be any sensible way to filter a dimension of string variables using a regular expression?