I got confusing ranking results from the sample code about py-ecomplexity HOT 4 CLOSED

Thanks for the comment. The sample code uses only 2 digit HS codes (just for syntax demonstration, since the 4 digit dataset takes much longer to download), which is too coarse to generate reasonable ECI and PCI values. You should get results much closer to the Atlas if you use 4 digit HS codes instead:

from ecomplexity import ecomplexity
from ecomplexity import proximity

# Import trade data from CID Atlas
data_url = "https://intl-atlas-downloads.s3.amazonaws.com/country_hsproduct4digit_year.csv.zip"
data = pd.read_csv(data_url, compression="zip", low_memory=False)
data = data[['year','location_code','hs_product_code','export_value']]

# Calculate complexity
trade_cols = {'time':'year', 'loc':'location_code', 'prod':'hs_product_code', 'val':'export_value'}
cdata = ecomplexity(data, trade_cols)

from py-ecomplexity.

Thank you for your quick response!

I have tried the 4 digit data but the results were still confusing. Take the 2016 ECIs as an example, UMI (United States Minor Outlying Islands), WLF (Wallis and Futuna), NIU (Niue), and many other small countries still possess an ECI much higher than USA, China, etc, even after data filtering (Top-100).

Is this normal?

Thanks again.

from py-ecomplexity.

Ah, I see. We do have an underlying data cleaning process that removes certain products and countries (including very small countries), which is why the countries you mentioned don't feature on the Atlas rankings page that you linked to. There's a little bit more detail on the data cleaning here. If you want the full cleaning process, I'd suggest you reach out to the authors of the cleaning process (Sebastian Bustos / Muhammed Yildrim), who might be able to send you a copy of the process we use to clean UN COMTRADE data before using it for complexity calculations.

from py-ecomplexity.

Thank you. I thought this algorithm should be more robust to the data. Is it possible to share a one-year copy of the processed data? So I can get a sense of the data cleaning process I should design for my task.

from py-ecomplexity.