Ayanwoye Gideon Ayandele – [email protected]

1. Project Description.
2. Web Scraping (ycombinator_scraper.ipynb/ycombinator_scraper.py)
3. Data Wrangling and Exploration (EDA_ycombinator.ipynb)
4. Analysis Summary
5. Details of Charts
6. References

The motivation for this project is to achieve a very basic end-to-end data engineering project by collecting/scraping, wrangling, cleaning and analysing/visualizing companies' information listed on https://ycombinator.com/companies.

The project main objectives were:

1. Perform web scraping
2. Do data wrangling (gathering, assessing and cleaning) on the crawled data.
3. Store, analyze, and visualize the wrangled data.
4. Reporting on:
   • data wrangling efforts.
   • data analysis and visualizations

The project was divided into two parts:

1. Web Scraping (ycombinator_scraper.ipynb/ycombinator_scraper.py)
2. Data Wrangling and Exploration (EDA_ycombinator.ipynb)

The dependencies and third party libraries for the scraper include:

• Selenium
• BeautifulSoup
• requests
• numpy
• pandas

1. I scraped data pertaining to all 1000 companies listed on https://ycombinator.com/companies, which are:

• The listed company names
• The company's ycombinator page url
• The company location
• The company short description (Description head) using the selenium library since the page is dynamic.

2. I then went through the scraped company's ycombinator page url using requests library since the pages are static, and grab many other informations (company's description, year founded, team size, company page url, social media urls, management details) as they appear for each company.

3. At the end, I created a CSV file in the following format:

4. The scraper runs for approxiamtely 1.5 minute with multithreading and approximately 7 minutes when NOT multithreaded

The dependencies and third party libraries for the EDA include:

• numpy
• pandas
• matplotlib
• seaborn

The summary from the data assessment and cleaning were that:

• There were cases of duplicated company names (Nash, Atlas and Streak) which appeared twice but had their characteristics to be different from the duplicate, it was then concluded to neglect the issue.

2. Missing data were represented with NaN which would not be imputed or removed as they represented charateristics that were not for the particular company
3. New variable showing the Country_Of_Origin of the company was extracted from the Company_Location column and, another variable Number_Of_Founders was also extracted from Active_Founder1 through to Active_Founder6

Using both Univariate and Bivariate analysis:

• The most represented country of all is the USA which counts 654 of the total 1000 companies. It is followed by India, Canada, UK, Nigeria and Indonesia

• It could be seen that the more recent a company is founded, the likely it is to be funded/listed by ycombinator

• The Team size distribution is highly right-skewed with a really long tail that it was very difficult to view the plot. I had to resolve into binning of size 100 and also set the plot's x_axis limit to 3000. Most teamsize is between 2-4

• Most number of founders is 2 followed by 1 and 3

• No interesting relationship between country of origin and team size, number of founder and year founded. Also

• There is a weak, negative linear correlation between Number_Of_Founder and team size.

• Most represented country (Country_Of_Origin) on ycombinator: The most represented country of all is the USA which counts 654 of the total 1000 companies. It is followed by India, Canada, UK, Nigeria and Indonesia:

• The distribution of the Year founded of the companies: It could be seen that the more recent a company is founded, the likely it is to be funded/listed by ycombinator:

• The distribution of the team size of the companies: The Team size distribution is highly right-skewed with a really long tail that it was very difficult to view the plot. I had to resolve into binning of size 100 and also set the plot's x_axis limit to 3000. Most teamsize is between 2-4:

• The distribution of the Number_Of_Founder of the companies: Most number of founders is 2 followed by 1 and 3:

There is no interesting relationship between country of origin and team size, number of founder and year founded. Also there is a weak, negative linear correlation between Number_Of_Founder and team size.

• The relationship between Country_Of_Origin, and Year founded.:

• The relationship between Country_Of_Origin, and team size:

• The relationship between Country_Of_Origin, and Number_Of_Founder:

• The relationship between Number_Of_Founder, and team size:

• ycombinator
• How to make infinite scrolling with selenium