Add Arrow output serialization. about arquero HOT 3 CLOSED

I've released a new package - arquero-arrow - that supports Arrow serialization of Arquero tables as well as arrays of JavaScript objects. This new package provides a simple interface to the apache-arrow JavaScript library (on which it depends), while also using higher-performance encoders for standard integer, float, date, boolean, and string dictionary types.

I also updated the Arquero and Apache Arrow notebook to demonstrate Arrow serialization!

cc @niviksha, @bmschmidt, @ericemc3

from arquero.

👍 would use for a slightly different case: it would let you more easily do arquero computations in a web worker, which is where I do most of my Arrow computation when not on Observable.

from arquero.

I've started a notebook for prototyping and testing Arrow serialization here:
https://observablehq.com/@jheer/arquero-to-arrow-serialization

Feedback and suggestions are welcome!

Also, @bmschmidt, in case you haven't seen it yet, you might be interested in https://github.com/uwdata/arquero-worker, which implements Arquero worker thread support. Right now the inter-thread communication is done via JSON string serialization, but the plan has always been to eventually augment that with a more efficient transfer options.

from arquero.