Giter VIP home page Giter VIP logo

ufo-map's Introduction

UFO DATA WEB SCRAPER

Description:

This application is a sophisticated web scraper specifically designed to extract and aggregate information from the National UFO Reporting Center (NUFORC) website. It efficiently handles asynchronous HTTP requests, capable of processing up to 1000 pages simultaneously, which is the upper limit before the website's throttling mechanisms activate. The core of this application lies in its ability to store and manage data in a JSON format, making it easily accessible and manipulable by future programs. Each entry in the JSON file contains detailed information about UFO sightings, including the date, time, location, a narrative description of the event, the shape of the observed craft, accompanying images, and a boolean value representing whether or not there are videos present in the entry.

A notable feature of this scraper is its handling of video content. As the videos related to UFO sightings are hosted on third-party platforms, they are not directly accessible by the scraper. To address this, a dedicated component, vid_scraper.py, is employed. This module is responsible for identifying sightings that include video content and tracking their associated ID numbers, thus maintaining a comprehensive record of which sightings include a video to accompany their description.

In addition to data collection, the application includes an advanced parsing and data presentation tool, logger.py. This module offers a variety of functions to display the compiled data in user-friendly formats. Users can view statistics such as the number of sightings reported per country, the frequency of different craft shapes observed, or generate a list of all sightings in a specified geographic location. These insights are invaluable for researchers and enthusiasts alike, offering a unique perspective on UFO sighting trends and patterns.

Furthermore, logger.py is designed with ease of use in mind, featuring a command-line interface with predetermined arguments. This allows users to quickly and efficiently access the specific data they are interested in, without the need for complex queries or database knowledge.

Overall, this application represents a powerful tool for the collection, analysis, and presentation of UFO sighting data. Its ability to handle large volumes of data, combined with its user-friendly interface and comprehensive data presentation capabilities, makes it an invaluable resource for anyone interested in the study and exploration of unidentified flying objects.

ufo-map's People

Contributors

louispino avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.