Video Demo: https://youtu.be/1uwH6bPe1wo
This application is a sophisticated web scraper specifically designed to extract and aggregate information from the National UFO Reporting Center (NUFORC) website. It efficiently handles asynchronous HTTP requests, capable of processing up to 1000 pages simultaneously, which is the upper limit before the website's throttling mechanisms activate. The core of this application lies in its ability to store and manage data in a JSON format, making it easily accessible and manipulable by future programs. Each entry in the JSON file contains detailed information about UFO sightings, including the date, time, location, a narrative description of the event, the shape of the observed craft, accompanying images, and a boolean value representing whether or not there are videos present in the entry.
A notable feature of this scraper is its handling of video content. As the videos related to UFO sightings are hosted on third-party platforms, they are not directly accessible by the scraper. To address this, a dedicated component, vid_scraper.py, is employed. This module is responsible for identifying sightings that include video content and tracking their associated ID numbers, thus maintaining a comprehensive record of which sightings include a video to accompany their description.
In addition to data collection, the application includes an advanced parsing and data presentation tool, logger.py. This module offers a variety of functions to display the compiled data in user-friendly formats. Users can view statistics such as the number of sightings reported per country, the frequency of different craft shapes observed, or generate a list of all sightings in a specified geographic location. These insights are invaluable for researchers and enthusiasts alike, offering a unique perspective on UFO sighting trends and patterns.
Furthermore, logger.py is designed with ease of use in mind, featuring a command-line interface with predetermined arguments. This allows users to quickly and efficiently access the specific data they are interested in, without the need for complex queries or database knowledge.
Overall, this application represents a powerful tool for the collection, analysis, and presentation of UFO sighting data. Its ability to handle large volumes of data, combined with its user-friendly interface and comprehensive data presentation capabilities, makes it an invaluable resource for anyone interested in the study and exploration of unidentified flying objects.