The Data-Centric AI Community just launched a small community project to experiment with the NIST Challenge!

• Goal: To learn about Synthetic Data and how it can be used to prepare sensitive private data for public release!
• Dates: From April to July. You can also join at any time, follow the weekly plan, and post questions on our Discord.
• Where: 🤖-nist-challenge channel in our Discord Server
• Touch Points: We meet every Friday around 4 PM GTM on the 🧠-code-with-me channel to discuss the project.

🎯 The overall goal of the project is to explore synthetic data to prepare sensitive private data for public release.

📀 NIST has launched a benchmark of 3 datasets, MA, TX (Texas), and NATIONAL which you can use in the project.

📊 To provide an evaluation of the de-identified data against the target/real data, NIST has created the sdnist package that can be installed according to the instructions below.

💻 To create the de-identified data, we'll use ydata-synthetic package, explore different model settings and study the effect this has on the final results.

• Read the instructions and information about the challenge
• Learn about the benchmark data released -- The NIST Diverse Communities Data Excerpts
• Post questions and ideas on the 🤖-nist-challenge channel

• Learn about the basic aspects of Synthetic Data:
  • 10 Most Frequently Asked Questions About Synthetic Data
  • Synthetic Tabular Data Generation
  • Synthetic Data for Data Science Development
  • Applications of Synthetic Data - The role of Synthetic Data in Privacy Engineering
• Post questions and comments on the 🤖-nist-challenge channel
• Meet us on Friday (May 5) to discuss what you've learned (check the available slots on our 📅 Discord Calendar)

• Learn about the basic aspects of Data Profiling:

  • 📺 Auditing Data Quality with ydata-profiling: learn about what is data profiling, what common data quality issues we find in real-world domains (can you spot a few in the NIST datasets?), and how ydata-profiling can help you diagnose and overcome them
  • 📖 Awesome Data Science Tools to Master in 2023: Data Profiling Edition: learn more about data profiling and existing open source tools to understand your data to the fullest!
  • 📖 Auditing Data Quality with YData Profiling: an overview of ydata-profiling functionalities and how-to's

• Start profiling the NIST data:

  • Installydata-profiling (check the Installation Instructions below) and don't forget to star it, thank you! ⭐️
  • Choose one of the NIST datasets (MA, TX, or NATIONAL):
    • The datasets are available here
    • Run a Profile Report on your data (check the Installation Instructions below)
    • Create an excel file to register your learnings. Suggestion for the columns: Feature Name | Data Type (Numeric/Categorical) | Missing Values (Y/N) | Notes/Observations. Your observations should be based on the profiling report, but also on the description of the features provided

• Post questions and comments on the 🤖-nist-challenge channel.

• Meet us on Friday (May 12) to discuss what you've learned (check the available slots on our 📅 Discord Calendar). Don't forget to bring your excel file with the data description and your profiling report!

• Investigate ydata-synthetic and some of the models used to Generate Synthetic Data:
  • 📖 GANs for Synthetic Data Generation
  • 📖 10 Most Asked Questions in ydata-synthetic
  • 📖 Synthetic Data Generation with Gaussian Mixture Models
  • 📖 Generating Synthetic Tabular Data with GANs - Part 1
  • 📖 Generating Synthetic Tabular Data with GANs - Part 2
  • 📖 How to Generate Synthetic Tabular Data
• Start experimenting with ydata-synthetic (check the Installation Instructions below and don't forget to star it, thank you! ⭐️). If you prefer a UI experience, you can also leverage the Streamlit App in version 1.0.0:
  • 📺 How to "pip install ydata-synthetic" without errors!
  • 📺 Install ydata-synthetic in 5 min
  • 📖 How to Generate Real-World Synthetic Data with CTGAN
  • 📺 How to Generate Synthetic Data with ydata-synthetic's Streamlit app
• Compare your synthetic data with the real data using the .compare() functionality of ydata-profiling:
  • 📖 How to compare 2 datasets with ydata-profiling. What are the obtained results? Are there any aspects that you can improve?
• Post questions and comments on the 🤖-nist-challenge channel! You can upload your profiling reports the the channel so that we can discuss changes and improvements.

conda create --name synth-env python=3.10

conda activate synth-env

pip install ydata-synthetic

conda deactivate synth-env

conda create -n synth-env python=3.10
conda activate synth-env
pip install ydata-profiling==4.1.2

import pandas as pd
from pandas_profiling import ProfileReport

# Read the data from a csv file (NIST "MA" data in the example)
df = pd.read_csv("ma2019.csv")

# Generate the data profiling report 
original_report = ProfileReport(df, title='Original Data')
original_report.to_file("original_report.html")

conda activate synth-env
pip install ydata-synthetic==1.1.0

# Load data
data = fetch_data('adult')
num_cols = ['age', 'fnlwgt', 'capital-gain', 'capital-loss', 'hours-per-week']
cat_cols = ['workclass','education', 'education-num', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country', 'target']

# Defining the training and model parameters
batch_size = 500
epochs = 500+1
learning_rate = 2e-4
beta_1 = 0.5
beta_2 = 0.9

# Create and train the model
ctgan_args = ModelParameters(batch_size=batch_size,
                             lr=learning_rate,
                             betas=(beta_1, beta_2))

train_args = TrainParameters(epochs=epochs)

synth = RegularSynthesizer(modelname='ctgan', model_parameters=ctgan_args)
synth.fit(data=data, train_arguments=train_args, num_cols=num_cols, cat_cols=cat_cols)

# Generate new samples
synth_data = synth.sample(1000)

print(synth_data)