scrape's Introduction

Scrape

CLI utility to scrape emails from websites

Features

Asynchronous scraping
Recursive link follow
External link follow
Cloudflare email obfuscation decoding
Client side rendered pages support through headless chromium load awaits
Simple, grepable output

Install

MacOS:

brew tap lawzava/scrape https://github.com/lawzava/scrape
brew install scrape

Linux:
```
sudo snap install scrape
```

Usage

Sample call:

scrape -w https://lawzava.com

Depends on chromium or google-chrome being available in path if --js is used

Parameters:

      --async             Scrape website pages asynchronously (default true)
      --debug             Print debug logs
  -d, --depth int         Max depth to follow when scraping recursively (default 3)
      --follow-external   Follow external 3rd party links within website
  -h, --help              help for scrape
      --js                Enables EnableJavascript execution await
      --output string     Output type to use (default 'plain', supported: 'csv', 'json') (default "plain")
      --output-with-url   Adds URL to output with each email
      --recursively       Scrape website recursively (default true)
      --timeout int       If > 0, specify a timeout (seconds) for js execution await
  -w, --website string    Website to scrape (default "https://lawzava.com")

Note about scraper package

For those that are looking for scraper package - this repository was intended as a cli-use only thus the scraper package was moved to lawzava/emailscraper. The scrape utility will be maintained as a CLI implementation of emailscraper package.

Recommend Projects