As seen here.
T.tidy( aaplMissing, T.select(["Date", "Close"]), T.complete({Date: T.fullSeqDate('Date', 'day', 1)}, {Close: 0}) )

The values in Close are not parsed to the new object.

it is convenient to say d => d.foo.bar for arrange instead of requiring functions to be comparators ((a,b)=>number)

This library is super exciting and possibly very helpful for some projects I work on!

However, I can't see from your documentation which open-source license you're using, which could prevent me from using this library at work.

Could you add more info on which open-source license you've decided to use?

Hi, Thanks for making this library. I've wanted it to exist for years. I've been trying to incorporate into some of my projects and I came across something surprising today.

Here's a REPL reproduction: https://svelte.dev/repl/3d3126f8ea994d3d866427cab0642e3b?version=3.38.2

I wanted to group a list of states based on a value, in this case count. I was getting a really weird output, though. After poking around a bit, I discovered that I fixed the problem by setting { addGroupKeys: false } in the group export. The issue seems to be that because my input data is a list of strings, the default behavior of adding a key onto the element is converting it to an object.

I'm not sure what the best solution is for it – maybe a change in the docs, a console warning or possibly a change in the default behavior so that it doesn't mutate the data by default. My expectation was definitely that it wouldn't mutate the original object, for what it's worth.

See the code example here: https://codesandbox.io/s/divine-cache-x3rzjj?file=/src/App.tsx

Basically doing this:

tidy(data, mutate({ ab: (d) => d.a * d.b }), groupBy("ab"));

Results in a Typescript error.

Note: It doesn't matter which argument is passed to groupBy, the error persists either way.

e.g. mutate({ foo: (d, i) => i })

I looked around at the code and saw that tidy calls the functions passed one by one on the whole array,
I think that switching to using Iterators and apply at least some of the functions per cell would speed up run time significantly.

In my view slice feels very javascripty. dplyr's implementation allows for multiple methods:

• slice(iris, c(1, 51, 101)) // single argument
• slice(iris, 1, 51, 101) // multiple arguments
• slice(iris, -1, -51, -101) // negation

Am I right slice in tidy.js is basically same as javascript?

x = [1,2,3,4,5,6,7,8,9,10];
tidy(x, Tidy.slice(2,5))
(3) [3, 4, 5]
tidy(x, Tidy.slice([2,5]))   // ?
(10) [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
tidy(x, Tidy.slice(-5))
(5) [6, 7, 8, 9, 10]

I feel javascript is crying out for an elegant implementation like dplyr's. Wonder if there might be appetite for a slice2 function to plug the gap?

Would you be interested in a PR adding a new summarizer that I've found helpful. It's the same as n but allows you to add a condition and only count items that meet that condition. So if you want to know how many elements in your groupBy had a certain value you can get that.

Simple JS implementation (Would convert to TypeScript)

export default function nWhere(conditional) {
  return function nWhereFn (list) {
    return list.filter(conditional).length;
  }
}

Usage

const data = [
  { str: 'foo', value: 3 },
  { str: 'foo', value: 1 },
  { str: 'bar', value: 3 },
  { str: 'bar', value: 1 },
  { str: 'bar', value: 7 },
];

tidy(data, summarize({
  foos: nWhere(d => d.str === 'foo'),
  bars: nWhere(d => d.str === 'bar'),
  count: n(),
})
// output:
[{ foos: 2, bars: 3, count: 5 }]

To make the playground more useful, we should be able to share URLs by making the content and inputs saved to the URL.

Requested in #6

Originally this was because all entriesObject does it supply a mapEntry for the entries export, but it's weird that it just doesn't work. So let's just use it if it is provided.

Currently groupBy automatically adds group keys back to objects after each function in the flow. This was primarily done to mitigate the fact that summarize (a very common groupBy operation) removes them. There has been some discussion in #34 around whether or not this should be default behavior.

It's a pretty big breaking change to switch to not adding them by default, so I'm not sure it will be worth it. However, it would improve the performance of groupBy and perhaps it is easier to reason about by not adding them (users can always explicitly keep them around when summarizing via first (e.g. summarize({ cyl: first('cyl') })).

An in-between would be to not add them in except for certain exports? It's likely when exporting by entries or object you don't want them added back in. I'm not sure what to do here, and am open to any ideas.

Example:

tidy(
  [{ a: 123, b: 345 }, { a: 452, b: 999}],
  leftJoin([{ a: 452, c: 456 }], { by: 'a' }),
  select(T.everything())
)
// output:
[{"a":123,"b":345},{"a":452,"b":999}]

// expected output: 
[{"a":123,"b":345,"c":undefined},{"a":452,"b":999,"c":456}]

first() and last() try running on an undefined element when there are no items, so they should be updated to just return undefined in that case.

hello, i have started using tidy and am finding it awesomely helpful, and am appreciating the many examples in the documentation. thank you!!

i think i have found a bug with valuesFillMap for pivotWider(), where it seems that the map is just ignored.

test case

tidy(
  [
    {n:2, c:'a', e:1001},{n:3, c:'a', e:1002},{n:7, c:'x', e:1003},
    {n:4, c:'b', e:1001},{n:2, c:'r', e:1002},{n:9, c:'y', e:1003},
    {n:6, c:'c', e:1001},{n:1, c:'z', e:1002},{n:1, c:'z', e:1003},
  ],
  pivotWider({
    namesFrom: 'c',
    valuesFrom: 'n',
    valuesFillMap: { a:0, b:0, c:0, r:0, s:0, t:0, x:0, y:0, z:0 },
  }),
)

results

[
  {a: 2, x: undefined, b: 4, r: undefined, y: undefined, c: 6, z: undefined, e: 1001},
  {a: 3, x: undefined, b: undefined, r: 2, y: undefined, c: undefined, z: 1, e: 1002},
  {a: undefined, x: 7, b: undefined, r: undefined, y: 9, c: undefined, z: 1, e: 1003},
]

• map values do not seem to be applied; widened keys are given value undefined
• map keys do not seem to be applied; s and t are missing from results

expected results

[
  {a: 2, x: 0, b: 4, r: 0, y: 0, c: 6, z: 0, s: 0, t: 0, e: 1001},
  {a: 3, x: 0, b: 0, r: 2, y: 0, c: 0, z: 1, s: 0, t: 0, e: 1002},
  {a: 0, x: 7, b: 0, r: 0, y: 9, c: 0, z: 1, s: 0, t: 0, e: 1003},
]

• map values applied to widened keys (as 0)
• map keys applied; every row has all columns (a, b, c, r, s, t, x, y, z, and e)

Needs to use valueOf when caching keys. Could switch to explicitly using internmap. Brought up in #46

Currently if you do select([]) or select([contains('foo')]) and contains returns no results, an error occurs. Instead, let's just have select do nothing if no keys are passed – the input data is output untouched.

From the changes made in #34, keys are now prepended on objects in groupBy but the docs still show them being appended – groupBy.entries() for example:

const data = [
  { str: 'a', ing: 'x', foo: 'G', value: 1 },
  { str: 'b', ing: 'x', foo: 'H', value: 100 },
  { str: 'b', ing: 'x', foo: 'K', value: 200 },
  { str: 'a', ing: 'y', foo: 'G', value: 2 },
  { str: 'a', ing: 'y', foo: 'H', value: 3 },
  { str: 'a', ing: 'y', foo: 'K', value: 4 },
  { str: 'b', ing: 'y', foo: 'G', value: 300 },
  { str: 'b', ing: 'z', foo: 'H', value: 400 },
  { str: 'a', ing: 'z', foo: 'K', value: 5 },
  { str: 'a', ing: 'z', foo: 'G', value: 6 },
]

tidy(
  data,
  groupBy('str', [
    summarize({ total: sum('value') })
  ], groupBy.entries())
)
// output:
[
  ["a", [{"total": 21, "str": "a"}]], 
  ["b", [{"total": 1000, "str": "b"}]]
]

But putting that into the playground gives me:

[
  ["a", [{"str": "a", "total": 21}]], 
  ["b", [{"str": "b", "total": 1000}]]
]

First of all thanks for this nice library, it's filling an important gap in the JS world.

When using typescript I'm getting the following error, any idea? It seems an issue with the grouped column parameter.

         groupBy('day', [
            summarize({ total_cost: mean('total_cost'),
                        total_revenue: mean('total_revenue'),
                        margin: mean('margin') })
         ]))

No overload matches this call.
  The last overload gave the following error.
    Argument of type 'string' is not assignable to parameter of type 'GK<object>'.ts(2769)
tidy.d.ts(1006, 18): The last overload is declared here.
```

It gets very annoying when the type inference is wrong. Perhaps we can mitigate this by stopping trying to be clever with keyof and other crazy generics and just make whatever comes out of tidy be opaque. At the end of tidy flows, users can cast their outputs to their expected types. Currently you have to fight the type system and guess where to override things and it is just crazy.

if you do for instance groupBy.object({ addGroupKeys: false }), addGroupKeys is ignored.

import { tidy, mutate, arrange, desc } from "@tidyjs/tidy" dosent work anymore.

Does anyone else run into the problem where you want to add a new column via mutate but you don't want that new key to be added at the very end? One common example is I have a bunch of state fips codes that are my first column and I add the full state name via a lookup with a mutate call. I'd like to have that full state name then be the second column so my spreadsheet is easier to read. I could do a select / pick call but I'd have to write out all of my columns and that is a bit verbose.

Perhaps mutate could be supplied an index and it inserts the key at that index? Open to other workarounds people have found for this...

Thanks so much - this library is awesome!

The very handy playground includes iris, but it's not clear how to use it. The input table is visible but output says Error: ReferenceError: iris is not defined.

Wonder if worth preserving state in e.g. URL hash string so examples could be shared this way?

A common use case for me is filtering possibly null values before mutating data. For example like this:

import { filter, mutate, tidy } from "@tidyjs/tidy";

type Data = { a: number | null };
const data: Array<Data> = [{ a: null }, { a: 1 }, { a: null }, { a: 2 }];

tidy(
  data,
  filter((x) => x.a !== null),
  mutate({
    b: (x) => 2 * x.a
  })
);

Of course typescript does not know that after filtering x.a can not be null so it complains: Object is possibly 'null'.
This is fully expected and I love that tidyjs has such great type inference (I hope you do not make it dumber as suggested in #14).

A solution to my problem would be a type guard (thanks to @phryneas for the suggestion):

tidy(
    data,
    filter((x): x is Data & { a: number } => x.a !== null),
    mutate({
        b: (x) => 2 * x.a
    })
);

However this requires an additional declaration for the filter function that allows for type guards (it can be added without interference with the existing functionality). I currently do this:

declare module "@tidyjs/tidy" {
  function filter<T extends object, O extends T>(
    filterFn: (item: T, index: number, array: T[]) => item is O
  ): TidyFn<T, O>;
}

I'll send a pull request that adds this declaration to the library itself.

Feature Request

Able to turn on/off debug depending when needed. For example: I want to suppress debug on production but show it on staging

For example:

const _DEBUG = false;
tidy(data, debug("info label", {limit: 15, show: _DEBUG})  );

I'm building a static app (no Node) and planned to run a local version of tidy.min.js (plus its d3 dependency), so I cloned the file at <script src="https://www.unpkg.com/@tidyjs/tidy/dist/umd/tidy.min.js"></script> and call locally. But encountered the error:

DevTools failed to load SourceMap: Could not load content for http://localhost:8888/js/tidy.min.js.map: 
HTTP error: status code 404, net::ERR_HTTP_RESPONSE_CODE_FAILURE

And it turns out there is a file at https://www.unpkg.com/@tidyjs/[email protected]/dist/umd/tidy.min.js.map which specifies a number of .ts files e.g:

Is tidy.min.js.map an actual dependency? I'm not a javascript dev by background so may be breaking some best practice by cloning as I have.