Yes I agree. Its more explicit and altering the schema is not a big deal. Lets go with the object. I think it makes sense to close this issue and open a new issue describing the data structure we have decided on. @philipkueng could you do that?

from q-editor.

Concept for implementing labeled cells in data-table

@manuelroth and I discussed about the way you could implement this feature yesterday. We came to the conclusion that you have different approaches where and how to save the data and that it would be good to have everything written down to use it as a base for further discussion.

## Where to save the data
We thought about three different methods, each comes with some open questions that needs to be talked about first.

Values and meta-data in item

In this case you would alter the schema where the items-array will contain an object, not just a string. The object would contain a value-property and a meta-data property, containing which labeling the value has.

Ex.:

...
data: [
  [{
    value: 1
    options: {
      highlight: true
    }
  }],
  [{
    value: 2
    options: {
      highlight: false
    }
  }]
]
...

#### Pro

• Value and meta-data would be stored together

Contra

• Schema of all Q-Elements containing data-table would need some changes
• More complicated aspect of saving and reading data

### Meta data stored in properties
The schema would have a new property which would contain a shadow-array with the mapping of the meta-data to the data.
The question that comes up with this method is:
Can you implement a new property and using it as a shadow-element and not displaying it in the editor?

Ex.:

...
data: [
  ['header1', 'header2'],
  ['1', '2']
],
metaData: [
  [null, null]
  [{highlight: true}, null]
]
...

Pro/Contra

• Open questions about it

Meta-data stored in Q:options

To store the meta data in Q:options seems like the best method at first sight.
After investigating there's an open question as well:
According to the property allowTranspose in Q-Table, the option won't get stored in the item but in the schema. Is this thought trough?

Ex.:

...
"Q:options": {
  "allowTranspose": false,
  "metaData": [
    [{highlight: true}, null],
    [null, {highlight: true}],
  ]
}
...

### Define mirrored array in data-property
With this method, you split the data-object in 2 sectors:

• data
• meta-data

Ex.:

...
data :[
  [
    ['header1', 'header2'],
    [1,2]
  ],
  [
    [{highlight: true}, null],
    [null, null]
  ]
]
...

#### Pro

• Data and Meta-data is together

#### Contra

• Schema of all Q-Elements containing data-table would need some changes
• The differnet labels(highlight, annotation, footnote) need to be variable for each tool. if the tool has no labels defined, it would be possible that you'll send an empty 2nd array with every request.

Definition of the labels

Highlight (label values)

This label would visually highlight either a value, row, column or group of values in the table.
Could look on
json

highlight: {
  coord: {
    col: int,
    row: int
  }
}

###  Annotation / Footnote
This label allows to add a note to a certain value and would be defined as following:

annotation: {
  coord: {
    col: int,
    row: int
  },
  text: string
}

## How to safe the data

### Shadow-array of data
In this scenario, you create a shadow-array of the data-array. Each coordinate in the shadow-array points to it's value in the data-array. The bad thing about this is, if there are no highlights or annotations and lots of data, you send alot of null-data around.

metaData: [
    [{coord: {col: 1, row: 1}}, null],
    [null, null]
]

Coordinates of data

This method is the prefered one since you keep the data-transmition low and still submit all the important.

metaData: [
  highlight: [
    {coord: {col: 1, row: 1}},
    {coord: {col: 1, row: 2}}
  ],
  annotation: [
    {coord: {col: 2, row: 1}, text: "*1"}
  ]
]

from q-editor.

I would go with another variante of Values and meta-data in item

schema.json would look like this:

"data": {
  "title": "Daten",
  "type": "array",
  "Q:type": "table",
  "Q:options": {
    "metaData": {
      "property": "somePropertyName",
      "cells": {
        "schema": {
          "type": "object",
          "properties": {
            "highlight": {
              "type": "boolean"
            },
            "annotation": {
              "type": "string"
            }
          }
        } 
        "notOnColumn": 1,
        "notOnRow": 1
      },
      "rows": false,
      "columns": {
        "notOnColumn": 1,
        "schema": {
          "type": "object",
          "properties": {
            "highlight": {
              "type": "boolean"
            }
          }
        } 
      },
    }
  }
}

where Q:options.metadata.cells, Q:options.metadata.rows and Q:options.metadata.columns could be false or undefined to not allow any metadata or an object with more information where to allow/disallow this and a schema describing the the metaData.

schema-editor-table would then store the metaData on the item property defined in Q:options.metadata.property. In this case somePropertyName. This would result in item data like this:

{
  "data": [
    [
      "Day",
      "A"
    ],
    [
      "01.01.2018",
      "499"
    ],
    [
      "02.01.2018",
      "250"
    ]
  ],
  "somePropertyName": {
    cells: [
      [null, null],
      [
        null,
        {
          highlight: true,
          annotation: "some string"
        }
      ]
    ],
    columns: [
      null,
      {
        highlight: true
      }
    ]
  }
}

what do you think?

from q-editor.

I am not yet sure if I prefer the shadow-array storing format as described in the comment above or one that only stores the actual metadata with the coordinates. if we go with the coordinate system I'd suggest a slight change of the format from your suggestion @philipkueng to not have to transform the data coming from schema-editor with the given metaData.cells.schema.

this would then look something like this

{
  "data": [
    [
      "Day",
      "A"
    ],
    [
      "01.01.2018",
      "499"
    ],
    [
      "02.01.2018",
      "250"
    ]
  ],
  "somePropertyName": {
    cells: [
      {
        rowIndex: 1,
        colIndex: 1,
        data: {
          highlight: true,
          annotation: "some string"
        }
      }
    ],
    columns: [
      {
        colIndex: 1,
        data: {
          highlight: true
        }
      }
    ]
  }
}

Regarding the handling of the data: it's imho more simple to have a shadow-array. That way any metaData is right at hand without any looping over an array while looping through the data array.

Regarding the amount of data to be stored and sent from Q-editor to Q-server: I am not completely convinced that this is a problem. Even with 1000 rows and 3 columns this is not too much data, and it has never to be sent to readers clients.

I will think about this some more. Please do so as well.

from q-editor.

Some further explorations for variant 2 (storing only the coordinates instead of a complete array copy)

There are several options on the plate:

Variant A (brought up by @philipkueng)

metaData: [
  highlight: [
    {coord: {col: 1, row: 1}},
    {coord: {col: 1, row: 2}}
  ],
  annotation: [
    {coord: {col: 2, row: 1}, text: "*1"}
  ]
]

Variant B (brought up by @benib)

metaData: {
  cells: [
    {
      rowIndex: 1,
      colIndex: 1,
      data: {
        highlight: true,
        annotation: "some string"
      }
    }
  ],
  columns: [
    {
      colIndex: 1,
      data: {
        highlight: true
      }
    }
  ]
}

Variant A uses a different format to store the data depending on the type e.g. a boolean has only the coordinates stored whereas string uses the text property to store the data. We can also have array or object as the data type.
We could solve this by storing the data in property data like it's done in variant 2.
This doesn't completely solve the problem of having a property named annotation for cells as well as column selections.

Whereas variant B stores the data separated by the type of selection (cell, column, row). This makes sense if we describe the format of the metadata in separate schema per type of selection.
I think there are usecases where we want to allow for different metadata for cells than for columns. (e.g. we could supporting an annotation string for a cell, but not for a column in charts).

About the possibilites to access the data:

Probably one needs the metaData for the different selection types (cell, column, row) at different places in the code rendering the data (the tool).
One could write helper functions to get these based on rowIndex and colIndex while looping through the data array. For cells something like this:

// returning only the first occurence as we need to make sure in schema-editor-table to not have multiple entries per coordinates
function getCellMetadata(rowIndex, colIndex) {
  return metaData.cells.find(cellMetaData => {
    return cellMetaData.rowIndex === rowIndex && cellMetaData.colIndex === colIndex;
  })
}

On the problem of making sure to only have one metaData entry per cell/row/column

The problem of having to make sure that there are not multiple metadata entries for the same cell/row/column needs to be solved in schema-editor-table.
Variant 1 where we use a copy of the data array structure to store the metadata solves this problem in the datastructure itself, so it doesn't have to be handled in code. But it has the tradeoff of more data to be stored/transfered.

from q-editor.

I like the idea of only storing the metadata with the coordinates for cells/rows/columns which are actually annotated/highlighted. I think it doesn't add a huge overhead to make sure that there are not multiple metadata entries for each type.

I think if we want to allow different metadata based on the type (cell/column/row) then we should go for Variant B.

from q-editor.

After thinking about this some more:

As we do not have the complete item in schema-editor-table but just to property a that is defined as a table type, we cannot easily store the metaData in whatever property we like on the item without passing the complete item into schema-editor-table.

this was built like this to make it impossible to have side effects on the item inside of one schema-editor type.

I'd like to keep it that way.

this has the consequence that we need to store the metaData somewhere on the property a together with the data.

for example:

a: {
  table: [[],[],[]],
  meta: [
  ]
}

• The format in a.metaData should probably follow the Variant B explained above.
• We can remove the Q:options.metaData.property property and just hardcode them to table and meta in my opinion. I am willing to discuss the naming here.

This means that introducing Q:options.metaData on an existing tool using schema-editor-table is a breaking change and needs a migration. See https://nzzdev.github.io/Q-server/migrations.html about how to write the migration in the tool and how to run it.

We lack a system to automatically migrate the data of an item in case the schema doesn't match, so we should run this on prod in some low-profile time and with good preparation and orchestration.

/ @philipkueng @manuelroth

from q-editor.

• Just to be sure. You are proposing that the type of the data property should be changed from array to object. And has two properties (table and meta)?
• What do you think about keeping the data property type array and having the first item be the actual data (table) and second item be the metadata (meta). This has the benefit of being very easily extensible in the future as the schema doesn't need to be altered if more things get added to the array. The migration is also needed with this approach.

Example item:

{
  "data": [[["Header1", "Header2"], [1,2,3], [4,5,6]], {"cells": [], "columns": [], "rows": []}]
}

Schema:

"data": {
      "title": "Daten",
      "type": "array",
      "Q:type": "table",
      "Q:options": {
        "allowTranspose": false
      },
      "items": {
        "type": "array",
        "items": {
          "anyOf": [
            {
            	"type": "array"
          	},
           {
            	"type": "object"
          	}
          ]
        }
      },
      "minItems": 2
    }

from q-editor.

I propose to store it in an object yes.
Having an array could also be an option.

Although I like the variant with the object more as it is very explicit what is stored in what property.
I do not think that it makes extending it in the future any harder as one can add another property to the object in the schema without the need for another migration.

accessing the data in the code is also more explicit as data and metaData is then written like this in the code instead of accessing them by array index.

from q-editor.