arthur-lozano / web-scraper-1 Goto Github PK

Tony Regalado

Pull Request

Build a web scraper that can automate the process of manually using the site.

• Scrape a Wikipedia page and record which passages need citations.
  • E.g. History of Mexico has few "citations needed" (5).
• Web scraper should report the number of citations needed.
• Web scraper should identify those cases AND include the relevant passage.
  • E.g. Citation needed for "lorem spam and impsum eggs".
  • Consider the "relevant passage" to be the parent element that contains the passage, often a paragraph element.

• Count function must be named get_citations_needed_count.
  • get_citations_needed takes in a url and returns an integer.
• Report function must be named get_citations_needed_report.
  • get_citations_needed_report takes in a url and returns a string.
  • the string should be formatted with each citation needed on own line, in order found.
• Module must be named scraper.py.

• [o] Organize the needed citations by section (i.e. the parent heading tag).
  • Name additional function get_citations_needed_by_section.
• [o] Automatically do a citation scan for any links from the main section of wikipedia page.

• Verify that the correct count of citations needed is calculated.
• Verify that preceding passage

• Crummy
• Youtube

• $poetry new web-scraper
• Use the folder created by Poetry as the root of your project's git repository
• Refer to Lab Submission Instructions for detailed instructions.