A privacy plugin for metasyn, based on statistical disclosure control (SDC) rules of thumb as found in the following documents:
- The SDC handbook of the Secure Data group in the UK
- The Data Without Boundaries document Guidelines for output checking (pdf)
- Statistics Netherlands' output guidelines
While producing synthetic data with metasyn is already a great first step towards protecting privacy, it doesn't adhere to official standards. For example, fitting a uniform distribution will disclose the lowest and highest values in the dataset, which may be a privacy issue in sensitive data. This plugin solves these kinds of problems.
Currently, the disclosure control plugin is work in progress. Especially in light of this, we disclaim any responsibility as a result of using this plugin.
The easiest way to install the plugin is by using pip.
In your terminal (with the Python environment that you want to install to active) run the following command: pip install git+https://github.com/sodascience/metasyn-disclosure-control.git.
Alternatively, to install it within a Jupyter notebook: !pip install git+https://github.com/sodascience/metasyn-disclosure-control.git.
Basic usage for our built-in titanic dataset is as follows:
from metasyn import MetaFrame, demo_dataframe
from metasyncontrib.disclosure import DisclosurePrivacy
from metasyncontrib.disclosure.string import DisclosureFaker
df = demo_dataframe("titanic")
spec = [
{"name": "PassengerId", "distribution": {"unique": True}},
{"name": "Name", "distribution": DisclosureFaker("name")},
]
mf = MetaFrame.fit_dataframe(
df=df,
dist_providers=["builtin", "metasyn-disclosure"],
privacy=DisclosurePrivacy(),
var_specs=spec
)
mf.synthesize(5)shape: (5, 13)
┌─────────────┬────────────────────┬────────┬──────┬───┬────────────┬────────────┬─────────────────────┬────────┐
│ PassengerId ┆ Name ┆ Sex ┆ Age ┆ … ┆ Birthday ┆ Board time ┆ Married since ┆ all_NA │
│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ cat ┆ i64 ┆ ┆ date ┆ time ┆ datetime[μs] ┆ f32 │
╞═════════════╪════════════════════╪════════╪══════╪═══╪════════════╪════════════╪═════════════════════╪════════╡
│ 0 ┆ Benjamin Cox ┆ female ┆ 27 ┆ … ┆ 1931-12-01 ┆ 14:33:06 ┆ 2022-07-30 02:16:37 ┆ null │
│ 1 ┆ Mr. David Robinson ┆ female ┆ null ┆ … ┆ 1906-02-18 ┆ null ┆ 2022-08-03 13:09:19 ┆ null │
│ 2 ┆ Randy Mosley ┆ male ┆ 24 ┆ … ┆ 1933-01-06 ┆ 15:52:54 ┆ 2022-07-18 18:52:05 ┆ null │
│ 3 ┆ Vincent Maddox ┆ female ┆ 24 ┆ … ┆ 1937-02-10 ┆ 16:58:30 ┆ 2022-07-23 20:29:49 ┆ null │
│ 4 ┆ Kristin Holland ┆ male ┆ 17 ┆ … ┆ 1939-12-09 ┆ 18:07:45 ┆ 2022-08-05 02:41:51 ┆ null │
└─────────────┴────────────────────┴────────┴──────┴───┴────────────┴────────────┴─────────────────────┴────────┘
The rules of thumb, roughly, are:
- at least 10 units
- at least 10 degrees of freedom
- no group disclosure
- no dominance
For most distributions, we implemented micro-aggregation. This technique pre-averages a sorted version of the data, which then supplied to the original fitting mechanism. The idea is that during this pre-averaging step, we ensure that the rules of thumb are followed, so that the fitting method doesn't need to do anything in particular. While from a statistical point of view, we are losing more information than we probably need, it should ensure the safety of the data.
You can contribute to this metasyn plugin by giving feedback in the "Issues" tab, or by creating a pull request.
To create a pull request:
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
This is a project by the ODISSEI Social Data Science (SoDa) team. Do you have questions, suggestions, or remarks on the technical implementation? File an issue in the issue tracker or feel free to contact Raoul Schram or Erik-Jan van Kesteren.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent