Giter VIP home page Giter VIP logo

scrawler's Introduction

Social networks crawler

GitHub release (latest by date) GitHub license FAQ GUIDE How to support

A program to download photo and video from any site (e.g. Reddit, Twitter, Instagram).

Do you like this program? Consider adding to my coffee fund by making a donation to show your support. ๐Ÿ˜Š

ko-fi

Bitcoin: bitcoin:BC1Q0NH839FT5TA44DD7L7RLR97XDQAG9V8D6N7XET

Main window Channels window

What can program do:

  • Download pictures and videos from users' profiles and subreddits:
    • Reddit images;
    • Reddit galleries of images;
    • Redgifs hosted videos (https://www.redgifs.com/);
    • Reddit hosted videos (downloading Reddit hosted video is going through ffmpeg (ffmpeg only works with the x64 program));
    • Twitter images and videos;
    • Instagram images and videos.
    • Imgur images, galleries and videos
    • Gfycat videos
    • Other supported sites
  • Parse channel and view data.
  • Download saved Reddit and Instagram posts.
  • Add users from parsed channel.
  • Advanced user management.
  • Labeling users.
  • Adding users to favorites and temporary.
  • Filter exists users by label or group.
  • Selection of media types you want to download (images only, videos only, both)
  • Download a special video, image or gallery
  • Making collections (grouping users into collections)
  • Specifying a user folder (for downloading data to another location)
  • Changing user icons
  • Changing view modes
  • ...and many others...

Supported sites

  • Reddit
  • Twitter
  • Instagram
  • RedGifs
  • Imgur
  • Gfycat
  • LPSG
  • XVIDEOS
  • Other sites

How does it works:

First, the program downloads the full profile. After the program downloads only new posts. The program remembers downloaded posts.

Reddit

The program parses all user posts, obtain MD5 images hash and compares them with existing ones to remove duplicates. Then the media will be downloaded.

Other sites

The program parses all user posts and compares file names with existing ones to remove duplicates. Then the media will be downloaded.

You can read about Instagram restrictions here

How to request a new site

Read here about

Requirements

  • Windows 7, 8, 9, 10, 11 with NET Framework 4.6.1 or higher (v4.6.1 must be installed). You can check version compatibility with this tool.
  • Authorization cookies and tokens for Twitter (if you want to download data from Twitter)
  • Authorization cookies and Hash for Instagram (if you want to download data from Instagram), Hash 2 for saved Instagram posts, Instagram stories authorization headers for Stories and Tagged data
  • ffmpeg library for downloading videos hosted on Reddit (you can download it from the official repo or from my first release). ffmpeg only works with the x64 version of the program.

Guide

Full guide you can find here

Installation

Just unzip the program archive to any folder, copy the file ffmpeg.exe into it and enjoy. ๐Ÿ˜Š

Don't put program in the Program Files system folder (this is portable program and program settings are stored in the program folder)

Updating

Just download latest version and unpack it into the program folder. Before starting a new version, I recommend making a backup copy of the program settings folder.

How to build from source

  1. Delete the "PersonalUtilities" project from the solution.
  2. Add the latest version of the "PersonalUtilities.dll" library (from the latest release).
  3. Import PersonalUtilities.Functions for the whole project.

How to make a plugin

Read about how to make plugin here.

How to support

Read more about how to support the program here.

Settings and usage

The program has an intuitive interface.

You need to set up authorization for Twitter and Instagram:

Just add a user profile and click the Start downloading button.

You can add users by patterns:

Read more about adding users and subreddits here

Add user

Using program as just video downloader

Create a shortcut for the program. Open shortcut properties. In the Shortcut tab, in the Target field, just add the letter v at the end across the space.

Example: D:\Programs\SCrawler\SCrawler.exe v

Separate video downloader

Contact me

matrix

scrawler's People

Contributors

aandyprogram avatar unknown81311 avatar

Stargazers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.