Tony Regalado
Build a web scraper that can automate the process of manually using the site.
- Scrape a Wikipedia page and record which passages need citations.
- E.g. History of Mexico has few "citations needed" (5).
- Web scraper should report the number of citations needed.
- Web scraper should identify those cases AND include the relevant passage.
- E.g. Citation needed for "lorem spam and impsum eggs".
- Consider the "relevant passage" to be the parent element that contains the passage, often a paragraph element.
- Count function must be named get_citations_needed_count.
- get_citations_needed takes in a url and returns an integer.
- Report function must be named get_citations_needed_report.
- get_citations_needed_report takes in a url and returns a string.
- the string should be formatted with each citation needed on own line, in order found.
- Module must be named scraper.py.
- [o] Organize the needed citations by section (i.e. the parent heading tag).
- Name additional function get_citations_needed_by_section.
- [o] Automatically do a citation scan for any links from the main section of wikipedia page.
- Verify that the correct count of citations needed is calculated.
- Verify that preceding passage
- $poetry new web-scraper
- Use the folder created by Poetry as the root of your project's git repository
- Refer to Lab Submission Instructions for detailed instructions.