A new flag --debug-field <field-name> should enable additional informations on a specific field :

• non masked value (missed)
• value masked incoherently (lowering coherent rate)
• pseudonym given to multiple value (lowering identifiant rate)

Disparity in data type (e.g. number vs string) or data structure (e.g. value vs array) should always trigger an error because data protection cannot be asserted.

Problem

Difficult to compute inter-stream coherence/identifiable rates when columns are built differently (by format or composition)

Example :

Stream A

A/real.jsonl

{"id":1,"name":"Clothilde","surname":"Renard"}
{"id":2,"name":"Andrée","surname":"Mathieu"}

A/masked.jsonl

{"id":1,"name":"John","surname":"Doe"}
{"id":2,"name":"Jade","surname":"Doe"}

Stream B

B/real.jsonl

{"id":1,"idutil":"1 - Clothilde Renard"}
{"id":2,"idutil":"2 - Andrée Mathieu"}

B/masked.jsonl

{"id":1,"idutil":"1 - John Doe"}
{"id":2,"idutil":"2 - John Doe"}

Solution

MIMO should be able to compute pre-values in the stream.

Configuration for Stream A

A/config.yaml

version: "1"
preprocess:
  - name: "idutil"
    value: "{{.id}} - {{.name}} {{.surname}}"
metrics:
  - name: "idutil"
    coherentWith: ["id"]

Configuration for Stream B

B/config.yaml

version: "1"
metrics:
  - name: "idutil"
    coherentWith: ["id"]

Command line

$ cat A/masked.jsonl | mimo --config A/config.yaml --persist localdb A/real.jsonl
$ cat B/masked.jsonl | mimo --config B/config.yaml --persist localdb B/real.jsonl

computation matrix (stored in localdb)

report.html

When an alias is set, contraints are disabled

The report is generated at the root workspace, insome context i need to change the outpur$t path and the file name
Indeed (because I m a maven man) all build information is write in * target * directory and this directory is in my * .gitignore *
Example :

> cat masked.jsonl | mimo --output target/ real.jsonl

> cat masked.jsonl | mimo -o target/ real.jsonl 

> cat masked.jsonl | mimo --output myHTML.html real.jsonl

> cat masked.jsonl | mimo -o target/myHTML.html real.jsonl 

The final * / * should define the stringis a directory (and create this path if not exist)

Does not seem to be working

As a user of MIMO, I want to validate

• a coherence rate between multiple real input fields vs the single pseudonym value
• a masking rate with or without taking into account empty values
• constraints on any rate like : equals to 100%, greater or equals to 95%, ...

Solution

Define a validation profile with a YAML configuration file:

mimo.yaml

metrics:
  - name: # name of the field to validate
    exclude: [ nil, "" ] # exclude nil and empty values from the masking rate computation (default: only nil values)
    coherentWith: ["name", "surname"] # list of fields from witch the coherent rate is computed (default: the current field)
    constraints:
      maskingRate:
        shouldEqualsTo: 1
      coherentRate:
        shouldBeGreaterThan: 0.95

$ mkfifo real.jsonl
$ lino pull bdd | tee real.jsonl | pimo | mimo --config mimo.yaml real.jsonl | lino push bdd

multmap should be an interface to switch from different implementations

A column could use an alias to be identified across multiple execution context

version: "1"
metrics:
  - name: "person.phone"
    alias: "phone_number"

{"batchs":[{"id":1,"accounts":[{"number":"A"},{"number":"B"}]}]}

version: "1"
metrics:
  - name: "batchs.[].accounts.[].number"
    coherentSource: "{{.current}} {{._1.id}}"

Path should be able to start from the current context of execution (inside an array structure for example)

• .current will reference the current object
• .root will reference the root object
• ._1 will reference the current first level object
• ._2 will reference the current second level object
• etc...

Actual usage

$ mkfifo real.jsonl
$ lino pull bdd | tee real.jsonl | pimo | mimo real.jsonl | lino push bdd

Wanted usage

$ # mimo starts the subprocess (pimo)
$ lino pull bdd | mimo pimo | lino push bdd

or

$ # mimo reads masked data from a file
$ lino pull bdd | tee real.jsonl | pimo | mimo --from-file real.jsonl | lino push bdd

MIMO should process object and array structures.

And all counters