like bin bin count should be better than avg avg?

• We need good criteria for determine timeUnits that we should determine by default for a particular dataset

Blocked #95 -- need stats first

• Extract and test hasRequiredPropertyAsEnumSpec in satisfy of EncodingConstraintModel and SpecConstraintModel

Replicating Compass

Gen

• aggregate.test.ts
• encodings.test.ts

Run npm run cover and see coverage report -- add more tests for uncovered constraints

• constraint/encoding.test.ts @FelixCodes
• constraint/spec.test.ts – @FelixCodes

(for demo)

Scale

Background

• Look at description and changes of #27 to see the infrastructure for adding nested property (bin.maxbins) -- note that I might miss something in the description, but if that's the case, you'll notice problem as you debug.

1st step Scale.type

• add scale.type (one PR)
  • understand what scale.type means from Vega-Lite docs
  • Add stuff like in #27
  • spec constraints (add to spec.ts)
    • omitBarAreaForLogScale -- don't use bar and area mark for log scale.
  • encoding constraints
    • dataTypeMatchesScaleType -- look at
    • omitBinForLogScale (originally vega/compass#151)
  • Add Example Query to examples/
  • add test for enumerate,
  • add test forgenerate
  • add test for all new constraints

Scale.*

Repeat the process for other scale properties (one PR for each)

• add ones that are required by other tasks
  • type
    • clamp: Q, T
    • exponent: pow
    • round: Q, T
      • accept types of values depending on scale type
  • zero --> zero doesn't play well with [ ScaleType.ORDINAL, LOG, TIME, UTC]. I don't think I'm missing anything else...
    • #105
  • bandSize
    • #93
    • bandSize must be at least 0
  • range
    • #101
    • values must contain two or more values.
  • domain
  • round
  • clamp
    • must have continuous domain / continuous domain (quantitative and time types only)
  • nice
    • similar to clamp.. quantitatiev and time.
  • exponent
  • useRawDomain

--- LATER ---

• padding
  • works with channel.x, channel.y --> uses pixels
  • ??? padding (0, 1) for rangeBands ??? -- LATER

The rationale is that sometimes users might expect to see some visualizations but do not see it because they are pruned.

Outputting them with a label that they do not satisfy constraint could be useful both for debugging and for helping users understand.

• Channel, Cardinality
• Penalize over encoding

Test

• TxT
• TxQ
• QxT > Q

(See editor)

Right now we just say aggregate has no occlusion, while raw has occlusion -- that's not always correct.

Fields with too high cardinality takes up a lot of space and can be slow to render.

• add a flag isKeyLike (or some better name) to schema

We might want to consider a few options:

• distinguish between categories (low cardinality) and text (high cardinality) as they serve different purpose in data analysis anyway.
  • Check if the cardinality is above X% (50%?) of the overall data count and above minimum threshold (e.g., 40)

Maybe check if "if the cardinality is above ~80% of the overall data count" or some similar criteria

• Add a constraint that excludes fields with too high cardanality from being added automatically.

• determine input format for cardinality in the schema
• maxCardinalityForFacets
• maxCardinalityForColor
• maxCardinalityForShape
• minCardinalityForBin

Not sure if we should add the following

• maxCardinalityForAutoAddOrdinal #70
• alwaysAddHistogram
• consistentAutoQ -- if aggregate for all Q are "*" -- give all of them same level of aggregation. (already have omitRawContinuousFieldForAggregatePlot)

(Originally vega/compass#16)

Sometimes adding debugger breakpoint in mocha doesn't work.

• http://stackoverflow.com/questions/30023736/mocha-breakpoints-using-visual-studio-code

preferredTemporalAxis?: Channel;
preferredOrdinalAxis?: Channel;
preferredNominalAxis?: Channel;

• Should ordinal and nominal be grouped together?
• Should we group this by the output scale type? (If so, time field with timeUnit would be ordinal scale.)

Constraints

• Q
• N, O
• T
  • Varying TimeUnit
• NxN
• NxQ
• NxT
• TxT
• TxQ
• QxQ

Currently in EncodingQuery, it's

bin?: boolean | EnumSpec<boolean> | ShortEnumSpec;

However, bin can have parameter too and I don't want mixing up between boolean and object here.

So I'm thinking

bin?: BinQuery

with the following interface

interface BinQuery {
  enable: boolean | EnumSpec<boolean> | ShortEnumSpec;
  maxbins: number | EnumSpec<number> | ShortEnumSpec;
  ... // other params
}

Any thoughts? @domoritz

• Generate JSON Schema for CompassQL schema

Look at this line in Vega-Lite
https://github.com/vega/vega-lite/blob/master/package.json#L35

Do the same for Query.

• Add Tests to validate all examples

In Vega-Lite, we have a test that validates all example specs so that both its input and output validates JSON schema.

• Validates input CompassQL query (each example json files)
• For each example query, run the query method in query.ts and check the output. For each SpecQueryModel in the output convert them into Vega-Lite specs (call .toSpec()) and validates Vega-Lite output.)

Make sure that the example test is excluded from test coverage.
(See Vega-Lite's package.json)

• AggregateOnly
• RawOnly

• modify model build logic to merge exclude with `values.

(for completeness)

• For raw plots, don't put a field on detail (originally vega/compass#98)

• enumerate answers based on input CompassQL query
  • check if the constraint is enabled (in the option)
  • generate fields -- read from schema
• support two types of constraints
  • encoding constraint (constraint for one encoding mappings)
  • spec constraint (constraint that involves multiple encoding mappings or involves relationship between mark and encoding)
• determine order in a way that automatically adding count still works
  • noRepeatedField --> '*'
• Remember which field we assign for later reference

Missing Constraints

• channelsSupportRoles
• omitShapeWithBin (channel supports role?)
• omitShapeWithTimeDimension (channel supports role?)
• omitBarWithSize
• omitRawBar/Area

1. Prevent duplicate output if autoCount comes after channel in propertyPrecedence

Basically, whenever, autoCount is false, we shouldn't even assign it to a channel.

We have to either add Logic to prevent autoCount to come after channel in the propertyPrecedence
or make answerSet in generate really a set to prevent duplication

2. Prevent nested property output from coming before its parent

enumerator.test.ts

For each of these properties:

• aggregate
• timeUnit
• field
• type

1. Write a test that enumerate all valid values

• aggregate
• timeUnit
• field
• type

hint: turn config.verbose = true

2. Write a test that enumerate both valid and invalid values (and test that the output contains only valid values)

• aggregate
  • To see relevant constraints, look at constraints/{spec|encoding}.ts
    • look at properties of each constraint
    • look at a few ones that contain Property.AGGREGATE

(LATER)

1. Write a test that enumerate all valid values

• bin -- bin is the most complicated -- ping me to explain about it

2. Write a test that enumerate both valid and invalid values (and test that the output contains only valid values)

• To see relevant constraints, look at constraints/{spec|encoding}.ts
• timeUnit
• field
• type

Other Files

Run npm run cover and see coverage report -- add more tests for uncovered constraints

• constraint/encoding.test.ts @FelixCodes
• constraint/spec.test.ts – @FelixCodes
• nest.ts – @RileyChang
• query.ts – @RileyChang

• Stack
• Stack constraint (don't enumerate non-summing aggregate for stack)

• Consistent Variable Name
  • encodingQ => encQ
  • property => prop
• EnumSpecIndex.timeunit => timeUnit

cc: @ZeningQu

e.g.,

{
  "data": {
    "url": "data/cars.json"
  },
  "mark": "point",
  "encoding": {
    "row": {
      "field": "Cylinders",
      "type": "nominal"
    },
    "x": {
      "aggregate": "mean",
      "bin": false,
      "field": "Horsepower",
      "type": "quantitative"
    },
    "y": {
      "aggregate": "mean",
      "bin": false,
      "field": "Acceleration",
      "type": "quantitative"
    }
  }
}

(For temporal line, there might be too many points?)

Right now enumerator stuff are in generate.ts.
However, this makes generate.test.ts unduly long.

Therefore, we should extract enumerator.ts from generate.ts

x:Q, y:Q, color: O should be > y:O, x:Q, size: Q

Nested grouping is very important for understanding structure / debugging output results.
(I'm currently flooded by transposes of the visualizations.)

Therefore we need a good syntax for nested grouping.

Suppose I want to hierarchical grouping that first group by dataQueryKey then by encodingKey.

• For each subgroup (by encodingKey), I want to order the subgroup's items by rankFn1.
• For each group (by dataQueryKey), I want to order the group's items (which are subgroups based on encodingKey) by rankFn2.
• Finally, I want to order groups by rankFn3.

For example, rankingFn1 = rankingFn2 = "effectiveness". rankFn3 can be some data enumeration order. The ranking function will rank groups by calculating score for the top-item in each list.

Suppose

spec = {
    "data": {"url": "data/cars.json"},
    "mark": "?",
    "encodings": [
      {
        "channel": "?",
        "field": "Cylinders",
        "type": "ordinal"
      },{
        "channel": "?",
        "bin": "?",
        "aggregate": "?",
        "field": "Horsepower",
        "type": "quantitative"
      }
    ]
  }

Here are a few alternative queries:

a) Nested version

{
  spec: spec, 
  group/groupings: { 
    // This case, definitely start with top-level grouping key. 
    by: 'dataKey',
    // if we want one output for each group, we can replace this orderItemBy with chooseBy
    orderItemBy: 'rankingFn2' 
    subgroup/subgroupings: {
      by: 'encodingKey',
      orderItemBy: 'rankingFn1'     
    }
  }],
  orderBy: 'rankingFn3'   
}

b) Array-based

{
  spec: spec, 
  // should the first one be the top-level one or the subgroup one -- current it's the subgroup one
  group/groupings: [{ 

     groupBy: 'encodingKey',
     // if we want one output for each group, we can replace this orderItemBy with chooseBy
     orderItemBy: 'rankingFn1'  
  },{
     groupBy: 'dataKey',
     orderItemBy: 'rankingFn2'  
  }],
  orderBy: 'rankingFn3'   // or orderGroupBy?
}

@jheer @domoritz any preference for a. or b. (or other options) / minor wordings?

I am not married to of these yet. Other ideas are welcomed.
I'm leaning toward the nested version because it's seems clearer which one is the top-level grouping.

• 1D
• 2D
• Need to think what to add

Refactor constraints

Specs

• hasAppropriateGraphicTypeForMark
• omitRawBarLineArea
• omitRawTable

Spec constraint's satisfy can become more effective when requireAllProperties is true.
Currently we repeatedly checking if we have all the requires properties.

• Adjust interface to be similar to vega/vega-lite#1461
• Find //TODO: transform and implement each relevant part.

For example,

{
  "spec": {
    "data": {"url": "data/cars.json"},
    "mark": "?",
    "encodings": [
      {
        "channel": "?",
        "field": "Cylinders",
        "type": "nominal"
      },{
        "channel": "?",
        "field": "Origin",
        "type": "ordinal"
      },{
        "channel": "?",
        "bin": "?",
        "aggregate": "?",
        "field": "Acceleration",
        "type": "quantitative"
      }
    ]
  },
  "groupBy": "data",
  "config": {
    "autoAddCount": true
  }
}

has this group group: Cylinders,n|Origin,o|bin(Acceleration,q)|count(*,q) that contains a visualization like this one:

{
  "data": {
    "url": "data/cars.json"
  },
  "mark": "point",
  "encoding": {
    "y": {
      "field": "Cylinders",
      "type": "nominal"
    },
    "x": {
      "field": "Origin",
      "type": "ordinal"
    },
    "row": {
      "bin": true,
      "field": "Acceleration",
      "type": "quantitative"
    },
    "size": {
      "aggregate": "count",
      "field": "*",
      "type": "quantitative"
    }
  }
}

http://localhost:1234/?mode=compassql&spec=1d-T

In older Compass, we add a few hacks for recommending text table.

With the new label and tile, we need to revise how we deal with this.