Giter VIP home page Giter VIP logo

sayoojsamuel / dragonload Goto Github PK

View Code? Open in Web Editor NEW
9.0 1.0 0.0 7.87 MB

Dragonload is a distributed download engine to distribute large scale downloads on different machines to better utilize high-speed intranet connected to low-speed internet.

Home Page: https://docs.google.com/presentation/d/e/2PACX-1vS9smlaQgnTwtKdhc6ciMOCYsUYzxrgwLmG2dppoD2cfBtGzZt7CrB16oTorEkZDb8Xlp7OCptULuMM/pub?start=true&loop=true&delayms=3000

License: MIT License

Python 98.81% Shell 1.19%
download-manager distributed-systems protocol file-sharing fastdownload

dragonload's Introduction

Dragonload Distributed Download Manager

Version 1.0 Python 3.8 MIT License sayoojsamuel Twitter

Dragonload is a state of the art Distributed Download Manager written in Python to distribute large scale downloads on different machines to better utilize high-speed intranet connected to low-speed internet.

Developed by Team Madcaps

Download

Head over to out Source Repo and clone the project

git clone github.com/sayoojsamuel/dragonload
cd dragonload

Installation

python setup.py install

Developer Mode

python setup.py develop

Sample Config File

Download Engine Sequence Diagram

Web App User Engagements

Table of Contents

  1. dragonload Distributed Downlaod Manager
    1. Tasks
    2. Tesing
    3. Protocol
      1. Splitfire Protocol
      2. ChainRain Protocol
      3. Dragonvault Protocol
    4. Ideas
      1. Simultaneous running of Splitfile and ChainRain in pipeline
      2. Add MultiThreaded Download for individual parts
      3. Authenticated Private Rooms
      4. curses based application
      5. Use click and pyinquirer, prompt-toolkit
      6. Use grpc for client, server, and all comms
    5. Necessary Reads
      1. Pathlib: Article to make os independent paths
      2. pycui: alternative to ncurses
    6. ISSUES:
    7. ENHANCEMENTS:

Project Board

Tasks

  • [-] Design a robust backend
  • [-] Manage the protocol
  • Create a config directory in POSIX and MAC machines. This about NS machines later. Config directory is necessary to store the db and the config file. DB is necessary for the downloading to happen. Also fix the directory where the downloading happens. Parital downloads in the temp directory or the main download directory.
  • Test the platform
  • Try to convert the commands to python only; platform independance

Tesing

  • [-] Check for partial download
  • Test the merge functionality

Protocol

Splitfire Protocol

This protocol is responsible for managing all the interested parties into one pool.

  1. Objective

    • Create Pool for interested Parties; N party proto
    • Check the file size;
    • Split the fileSize by a multiple of N
    • Collect user IP’s
    • Generate the download script. Notify the CNC after each successful download; Realtime progress for the pool
    • Schedule Download??? Take care of the sleep
    • Manage Parties who leave in between the protocol
    • Split part of the remaining itemsList to the newcomming member; Dynamic Pool Model

ChainRain Protocol

This protocol is for efficiently sharing the files after partly download among the decorated users.

  1. Objectives

    • Verify if parties are still active during the process
    • Check connection with the CNC
    • Share files parallely, efficiently and equally among the users. (check for corruptions)
    • Test if local hosting is proper
    • Share in a binary Tree manner - Will it work out?
    • Merge the parts as and when they arrive. Notify other about the status of the proces

Dragonvault Protocol

This is the main server application, which will handle the whole download manager. This can be implemented in two ways:

  • A centralized main server
  • A requirement based

Ideas

[ ] Simultaneous running of Splitfile and ChainRain in pipeline

[ ] Add MultiThreaded Download for individual parts

[ ] Authenticated Private Rooms

[X] curses based application

[ ] Use click and pyinquirer, prompt-toolkit

[ ] Use grpc for client, server, and all comms

  1. Create robust framework, and build curses app, web app on top

Necessary Reads

Pathlib: Article to make os independent paths

pycui: alternative to ncurses

ISSUES:

  • Fix the port issue. Update the server with the users port address in addition to the IP address. Code everything in main file.

ENHANCEMENTS:

  • Use python dataclasses
  • Make a custom URI scheme (dragon://)
  • Gossip framework for status sharing??

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.