Welcome to Tech Spotlight. The goal of this project is to search through relevant job postings and find the frequency of technologies associated with those jobs.
This is achieved in two parts: a scraper that scrapes data off Indeed.com with four queries, job title, location, age of posting, and a number of jobs to scrape. This produces a CSV file which can be processed within a Kaggle notebook, that tells the story about what technologies, frameworks, and libraries were found and how frequently they were posted.
Enabling and user to get hyper focused insights about the job market and they are interested in, as it pertains to software development
Example of output after csv file processed by kaggle notebook
Team Spotlight
Possible known bug with Windows Native, in writing files.
Term Selection is not all inclusive. (Feel free to add more terms to the Tech_term list in the datasets folder to further customize your search!)
Make sure you're running python 3.10
This data is intended for personal research purpose and not as a single source of truth.
We cannot account for the validity of the data.
This is a learning project and is not 100% flushed out.
git clone https://github.com/regex-rejects/tech-spotlight.git
Create visual python environment (venv)
example:
python3 -m venv .venv (mac/wsl)
py -m venv .venv (windows native)
activate virtual environment
example:
source .venv/bin/activate (mac/wsl)
.\.venv\Scripts\activate (windows native)
pip install -r requirements.txt
Run web scraper:
python tech_spotlight/scraper.py
This is the kaggle notebook we produced from datasets we scraped during our project. It includes insights based on our interpretations of the data.
Indeed Past 7 Day 900 Listing Term Data 18 May 2022
Fork this notebook and upload your csv
We welcome you to use this tool to create your own datasets tailored to your specific interest in the market.
Upload output csv file to new notebook and use above code as reference
Version 1.0
- numpy
- pandas
- seaborn
- matplotlib
- requests
- urllib
- bs4
- time
- random
- sys
- csv
- JB Tellez
- David Hecker
- Adam Owada
- Aaron Imbrock
- Joseph Streifel
- Chloe Nott