Giter VIP home page Giter VIP logo

node-mt-files-downloader's Introduction

Multi Threaded Files Downloader

NPM

NOT VERY STABLE: This module is not very stable, and is in need of a big refactor to work with the latest mt-downloader version. I will try to do this when I have the time, but unfotrunately other projects are taking a lot of time from me right now. Thanks for your understanding.

This module wrap the mt-downloader module and let you :

  • Manage multiple downloads
  • Get stats (speed, eta, completed, etc)
  • Auto-retry (continue) a download in case of error (ie. network error)
  • Manually resume a download from partial file
  • Stop and resume downloads
  • Get notified by events when a download start, fail, retry, stopped, destroyed or complete

Install

npm install mt-files-downloader

Usage

Require the module :

var Downloader = require('mt-files-downloader');

Create a new Downloader instance :

var downloader = new Downloader();

Create a new download :

var dl = downloader.download('FILE_URL', 'FILE_SAVE_PATH');

Start the download :

dl.start();

Examples

You can find complete examples in the examples/ folder :

Events

You can then listen to those events :

  • dl.on('start', function(dl) { ... });
  • dl.on('error', function(dl) { ... });
  • dl.on('end', function(dl) { ... });
  • dl.on('stopped', function(dl) { ... });
  • dl.on('destroyed', function(dl) { ... });
  • dl.on('retry', function(dl) { ... });

Downloader object

Methods

  • download(URL, FILE_SAVE_PATH, [options])
    • URL : URL of the file to download
    • FILE_SAVE_PATH : where to save the file (including filename !)
    • options : optional, passed directly to Download object
  • resumeDownload(filePath) : create a new download by resuming from an existing file
  • getDownloads() : get the list of downloads in manager
  • getDownloadByUrl(url) : get a specified download by URL
  • getDownloadByFilePath(filePath) : get a specified download by file path
  • removeDownloadByFilePath(filePath) : remove a specified download by file path. It does not destroy it, just remove from download manager ! Call download.destroy() before if you want to completely remove it.

Formatters methods

The Downloader object exposes some formatters for the stats as static methods :

  • Downloader.Formatters.speed(speed)
  • Downloader.Formatters.elapsedTime(seconds)
  • Downloader.Formatters.remainingTime(seconds)

Download object

Properties

  • status :
    • -3 = destroyed
    • -2 = stopped
    • -1 = error
    • 0 = not started
    • 1 = started (downloading)
    • 2 = error, retrying
    • 3 = finished
  • url
  • filePath
  • options
  • meta

Methods

  • setUrl(url) : set the download URL
  • setFilePath(path) : set the download file save path
  • setOptions(options) : set the download options
    • threadsCount: Default: 2, Set the total number of download threads
    • method: Default: GET, HTTP method
    • port: Default: 80, HTTP port
    • timeout: Default: 5000, If no data is received, the download times out (milliseconds)
    • range: Default: 0-100, Control the part of file that needs to be downloaded.
  • setRetryOptions(options) : set the retry options
    • maxRetries: Default 5, max number of retries before considering the download as failed
    • retryInterval: Default 2000, interval (milliseconds) between each retry
  • setMeta(meta) : set download metadata
  • setStatus(status) : set download status
  • setError(error) : set error message for download
  • getStats() : compute and get stats for the download
  • start() : start download
  • resume() : resume download
  • stop() : stop the download, keep the files
  • destroy() : stop the download, remove files

TODO

  • Validate data (setters)
  • Add tests

Licence

The MIT License (MIT)

Copyright (c) 2015 Leeroy Brun

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

node-mt-files-downloader's People

Contributors

leeroybrun avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

node-mt-files-downloader's Issues

Serve for download

Is there any option to server file to down at client side, not in server

Error using with Electron JS

I don't if this package is compatible with Electron js or not. But I give it a try anyway.
And I got this error:

Uncaught Exception:
Error: Uncaught, unspecified "error" event. ([object Object])
    at Download.emit (events.js:163:17)
    at options.onEnd (/....../node_modules/mt-files-downloader/lib/Download.js:308:18)
    at taskCallback (/....../node_modules/mt-downloader/node_modules/async/lib/async.js:443:21)
    at /....../node_modules/mt-downloader/lib/core/DownloadReaderAsyncTask.js:36:4
    at FSReqWrap.wrapper [as oncomplete] (fs.js:675:17)

Any ideal why?

After download file is complete the file is "LOCKED"

When the download finishes i am unable to do files action such as move/del the downloaded file(s).
I assume somewhere when the download is finished the file handler is not released. only if i close the program that runs the downloader then i can do the specified actions.

please elaborate.

P.S thanks for this great lib!

Cannot access file when finished download

Hi all,
I have an issue about download 7z file:
My option:

        dl.setOptions({
            threadsCount: 5,    // Default: 2, Set the total number of download threads
            method: 'GET',      // Default: GET, HTTP method
            port: 80,           // Default: 80, HTTP port
            timeout: 10000,     // Default: 5000, If no data is received, the download times out (milliseconds)
            range: '0-100',     // Default: 0-100, Control the part of file that needs to be downloaded.
        });

Handle event:

        dl.on('error', function () {
            logger.error(`Download error: ${dl.error}`);
            reject(
                {
                    error: dl.error,
                    return: null
                }
            );
        });
        dl.on('end', function () {
            logger.info(`Download done: ${local_file}`);
            resolve(`Remote: ${remote_file}. Local: ${local_file}`);
        });

Then i call start download file.
This file type is 7z, i will extract it when finished, so when resolve statement of promise, i call extractor tool. But i got an exception:

System ERROR:
The process cannot access the file because it is being used by another process.

I think downloader not yet finished at "end" event. Check my local folder, the downloaded file is corrupted.
Have you solution for this?
Thanks for reading.

Raise error on 404 return code

Hello,
mt-files-downloader hapilly downloads error pages instead of content. For example if server returns 404 .
I think error should be raised instead.

Does it support REST?

Hi, just want to ask if it support REST? I have a Java Server where it can stream files.. but it seems that if I use this module.. the output file is always 5KB only :D

Can i call this down loader from a java application and download the File.

Am having a Spring - Java Application which is running in the Linux Server. But the user will be accessing this application from the client machine which may be windows using a browser. Now the file will be downloaded to the users PC.
How can I make that happen in a faster method ? can I use this?

CLI version?

Is there a way to do this via the command line?

Add tests

Add tests to ensure everything is working when doing changes or updating dependencies.

FileHandlerAsyncTask.js - Error: EMFILE: too many open files

I'm getting an error that too many files have been opened. This happens when I try to download many files. I was downloading the files in some queue so when some files are downloaded I'm getting next one. The thing is that after downloading 912 files this error occurs. Main question is there any way to force to close the file once it has been downloaded, or should I destroy the "Downloader" or "download" somehow ?, if so how to ?

Corrupted file downloads

Could you please help me trace the issue:

If I download a file without resuming in any way, it works pretty much 100%

However, the moment the file is resumed, its luck of the draw with most of the time the file throwing an error

The first time I received this error:
DEMO - Resuming download...
EVENT - Download 1 error !
[Error: Invalid file path: c:/test/10MBtest.bin.mtd]
Download 1 error : Error: Invalid file path: c:/test/10MBtest.bin.mtd

The file was there, don't know why it was locked out.

The second time is the main issue I am facing:
EVENT - Download 1 error !
{ [Error: EPERM, ftruncate] errno: -4048, code: 'EPERM' }
Download 1 error : Error: EPERM, ftruncate

The file can balloon from 4-20 x its original size.

Here's my test code, the file name can be set to 100MBtest.bin if your speed is to fast to receive the issue:

var os   = require('os');
var path = require('path');
var Downloader = require('mt-files-downloader');


// Create new downloader
var downloader = new Downloader();

var fileUrl = 'http://speedtest.reliableservers.com/10MBtest.bin';
//var fileSavePath = path.join(os.tmpdir(), 'mtFileDlTest1.zip');
var fileSavePath = 'c:/test/10MBtest.bin';
console.log('File will be downloaded from '+ fileUrl +' to '+ fileSavePath);

// Start download
var dl = downloader.download(fileUrl, fileSavePath);

// Import generic examples for handling events and printing stats
require('./_handleEvents')(dl);
require('./_printStats')(dl);

dl.start();

dl.on('start', function() {
    console.log('Download started with '+ ((dl.meta.threads) ? dl.meta.threads.length : 0) +' threads.')
});

// Wait 10s before stopping and then 4s before resuming download
cycle();

function cycle() {
    setTimeout(function() {
        console.log('DEMO - Stopping download... will be resumed in 2s');

        dl.stop();

        setTimeout(function() {
            console.log('DEMO - Resuming download...');
            dl.resume();
            cycle();
        }, 2000);
    }, 5000);
}

The only other changes I made to your base code is to set the default threads to 6. I had them at 16 but found it replicated at lower numbers as well.

Downloaded files displays "You don't have permission to view this file" on Windows 10

Hi,

I am trying to achieve an image download manager via electron. I have implemented the mt-files-downloader and attempted a simple implementation. The file downloads fine but displays the text "it looks like you don't have permission to view this file. Check the permissions and try again". When I copy the file to another Drive, the file opens fine.

My previous implementation with "request" module worked fine and I could open the files at the same location.

Any suggestion to which direction I should take to figure out the problem?

Error: The .mtd file is corrupt. Start a new download??

Hi,

I'm getting this error while running the sample download scripts. I saw an .mtd file is creating on my destination directory. After few seconds it is failing. The error is

Error: The .mtd file is corrupt. Start a new download.
at module.exports (/home/testmachine/node_modules/mt-downloader/lib/Exceptions.js:83:33)
at /home/testmachine/node_modules/mt-downloader/lib/core/DownloadReaderAsyncTask.js:36:13
at FSReqWrap.wrapper [as oncomplete] (fs.js:602:17)

Resume does not work in Windows

when i resume a download, at the 99.9% get this error:
Error: EPERM: operation not permitted, ftruncate

do u know what is the problem? how can i fix this !?!?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.