Why are the eci computed with ecomplexity different from their given values? about py-ecomplexity HOT 5 CLOSED

Ah, this issue is much simpler than I expected. I didn't notice earlier, but you're using 2-digit SITC codes. The calculations for ECI in the Atlas are based on 4-digit SITC codes. This explains most of the discrepancy that you're seeing. Any remaining differences are due to the fact that on the Atlas, we restrict the set of countries to ones with reliable data for ECI calculations. Learn more about Atlas data here

Closing as not an issue.

from py-ecomplexity.

I did mention earlier that there would be some remaining differences. Atlas data uses a set of processes that clean the trade data based on bilateral reporting, and restrict the set of countries used for ECI calculations. More here. If you have specific questions about the Atlas data itself, I would suggest that you request help from the Atlas team here.

from py-ecomplexity.

Thanks for bringing this to our attention. We'll look into this.

Here is code to reproduce the issue:

import pandas as pd
import numpy as np
import seaborn as sns
from ecomplexity import ecomplexity

# Import trade data from CID Atlas
data_url = "https://intl-atlas-downloads.s3.amazonaws.com/country_sitcproduct2digit_year.csv.zip"
data = pd.read_csv(data_url, compression="zip", low_memory=False)
data.head()

# Calculate eci's
data = data[['year','location_code','sitc_product_code','export_value','sitc_eci']]
data = data[data.year==data.year.max()]
trade_cols = {'time':'year', 'loc':'location_code', 'prod':'sitc_product_code', 'val':'export_value'}
cdata = ecomplexity(data, trade_cols)
cdata = cdata[['location_code','eci','sitc_eci']].drop_duplicates(['location_code'])

# Compare against Atlas
sns.scatterplot('sitc_eci','eci', data=cdata)

from py-ecomplexity.

@shreyasgm
Thank you for your generous and quick reply. I have just tried the 4-digit SITC dataset using your python script as follows.

import pandas as pd
import numpy as np
import seaborn as sns
from ecomplexity import ecomplexity

# Import trade data from CID Atlas
data = pd.read_csv('../data/country_partner_sitcproduct4digit_year_2017.zip', compression="zip", low_memory=False)
data.head()

# Calculate eci's
data = data[['year','location_code','sitc_product_code','export_value','sitc_eci']]
data = data[data.year==data.year.max()]
trade_cols = {'time':'year', 'loc':'location_code', 'prod':'sitc_product_code', 'val':'export_value'}
cdata = ecomplexity(data, trade_cols)
cdata = cdata[['location_code','eci','sitc_eci']].drop_duplicates(['location_code'])

# Compare against Atlas
# To download gdp data from wordbank 
# https://data.worldbank.org/indicator/ny.gdp.pcap.cd?most_recent_year_desc=true
df_gdp = pd.read_csv('./GDP/API_NY.GDP.PCAP.CD_DS2_en_csv_v2_41089.csv', sep = '\t')
year =2017
df_g = df_gdp[['Country Code', str(year)]]
gdp_dict =dict(zip(df_g['Country Code'],df_g[str(year)]))
cdata['gdp'] = [gdp_dict[name] if name in gdp_dict else 0.0 for name in cdata['location_code']]

# Compare against Atlas
df = cdata[cdata['gdp'] >0]
sns.scatterplot('eci','gdp', data=df, label = 'eci computed with ecomplexity')
sns.scatterplot('sitc_eci','gdp', data=df, label = 'sitc_eci given by Hidalgo')
plt.yscale('log')
plt.legend()
plt.xlabel('ECI', fontsize = 15)
plt.ylabel('GDP', fontsize = 15)
plt.title('2017');

I think this problem has been largely solved following your suggestion. Thank you for your comments.

from py-ecomplexity.

@shreyasgm Thank you for your kindly reply. I have benefited a lot from your insightful and comprehensive answers which have well addressed my question. The idea of ECI is very inspiring and it could be applied to many other areas. Thanks again for your maintaining of this wonderful Python package. Best regards. C.J.

from py-ecomplexity.