Data Explorer

The various functions lets us undertstand the data, it's datatypes and describe the features. We can get basic details about data as well as advanced descriptive statistcs. We can check if any null values are present, if yes we have the functionality to fill them using appropriate logic. Another automation method lets us check for duplicates and lets us remove them if desired.pandas-profiling generates profile reports from a pandas DataFrame. pandas-profiling extends pandas DataFrame with df.profile_report(), which automatically generates a standardized univariate and multivariate report for data understanding. We are proviede with an option to download the pandas profile report.

For each column, the following information (whenever relevant for the column type) is presented in an interactive HTML report:

• Type inference: detect the types of columns in a DataFrame
• Essentials: type, unique values, indication of missing values
• Quantile statistics: minimum value, Q1, median, Q3, maximum, range, interquartile range
• Descriptive statistics: mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness
• Most frequent and extreme values
• Histograms: categorical and numerical
• Correlations: high correlation warnings, based on different correlation metrics (Spearman, Pearson, Kendall, Cramér’s V, Phik)
• Missing values: through counts, matrix, heatmap and dendrograms
• Duplicate rows: list of the most common duplicated rows
• Text analysis: most common categories (uppercase, lowercase, separator), scripts (Latin, Cyrillic) and blocks (ASCII, Cyrilic)
• File and Image analysis: file sizes, creation dates, dimensions, indication of truncated images and existance of EXIF metadata

To recreate this web app on your own computer, do the following.

Firstly, we will create a conda environment called dex

conda create -n dex python=3.7.9

Secondly, we will login to the eda environment

conda activate dex

Download requirements.txt file

wget https://raw.githubusercontent.com/gmayuriiii/data-explorer/main/requirements.txt

Pip install libraries

pip install -r requirements.txt

Download and unzip contents from

streamlit run dataexplorer.py