Giter VIP home page Giter VIP logo

node-downloader-helper's People

Contributors

chaphasilor avatar ezwelty avatar fossabot avatar geopic avatar gitter-badger avatar half-shot avatar hgouveia avatar leohubert avatar mumuyu66 avatar nathanbarrett avatar remyjeancolas avatar robinschneider avatar ry0tak avatar shenlanchenwei avatar shroxd avatar splitgemini avatar srbrahma avatar thurasw avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-downloader-helper's Issues

Download has incomplete status if 'content-length' is not present

If for some reasons target server will not return 'content-length' header for downloaded file, download info object will always have incomplete: true property, while file will be downloaded and written to filesystem. (https://github.com/hgouveia/node-downloader-helper/blob/master/src/index.js#L514)

How to reproduce:

let result
const url = 'https://www.learningcontainer.com/wp-content/uploads/2020/07/Sample-JPEG-Image-File-Download-scaled.jpg'
const dl = new DownloaderHelper(url, __dirname)

dl.on('end', (downloadInfo) => {
  result = downloadInfo
})

await dl.start()

console.log(result)

// Output: 
// {
//  fileName: 'Sample-JPEG-Image-File-Download-scaled.jpg',
//  filePath: '/my/dir/path/Sample-JPEG-Image-File-Download-scaled (2).jpg',
//  totalSize: NaN,
//  incomplete: true,
//  onDiskSize: 221472,
//  downloadedSize: 221472
// }

I think, that this behavior is wrong. 'content-length' header check should be optional because not every server will respond with this header.

Error when calling pause or stop method

When I call the pause() function using a download object, the following error occurs:

image

what is the correct way to call the function just passing the object?

Crashing instead of catching errors.

Hello, When I attempt to download a file that doesn't exist I get the error Error: Response status was 404 \n at ClientRequest.<anonymous> and then the project crashes.
I have the dl.on("error",) code. But it doesn't seem to be doing anything.

dl.on("error", (error) => {}); 

I'm not sure if I'm doing something wrong.

Not respecting `RemoveOnStop` config flag.

Sporadically the library goes into a state where Stopping the download would trigger the Remove action on the Downloaded file even if the library is instantiated with the RemoveOnStop: false flag.

Here is my instantiation code

    this.helper = new NodeDownloadHelper(metadata.source,
      metadata.destination,
      {
        forceResume: false,
        override: false,
        removeOnFail: false,
        removeOnStop: false,
        retry: false,
        httpRequestOptions: {
          timeout: Config.DOWNLOADS.TIMEOUT,
        },
        httpsRequestOptions: {
          timeout: Config.DOWNLOADS.TIMEOUT,
        },
      });

Logs

{"moduleName":"Node Downloader for af293d87-a90d-490d-ad27-4ee2905aae02","level":"info","message":"[ 'Download stopped' ]","timestamp":"2021-04-20T10:48:35.219Z"}


{"moduleName":"Node Downloader for af293d87-a90d-490d-ad27-4ee2905aae02","level":"info","message":"[ 'Download stopped' ]","timestamp":"2021-04-20T10:48:35.219Z"}
{"level":"error","message":"Unhandled promise rejection, reason: ENOENT: no such file or directory, stat '/var/folders/gm/713hkdtn60j5kpkht_fh1yph0000gp/T/unityhub-e57c7420-a1c5-11eb-9147-0d5d8a32fca5/UnitySetup-AppleTV-Support-for-Editor-2018.4.33f1.pkg'\n\tstack Error: ENOENT: no such file or directory, stat '/var/folders/gm/713hkdtn60j5kpkht_fh1yph0000gp/T/unityhub-e57c7420-a1c5-11eb-9147-0d5d8a32fca5/UnitySetup-AppleTV-Support-for-Editor-2018.4.33f1.pkg'\n    at statSync (fs.js:1020:3)\n    at Object.e.statSync (electron/js2c/asar_bundle.js:5:4264)\n    at b.d (/Users/antoine.lheureux/dev/unity/hub.uw-hub/dist/mac/Unity Hub.app/Contents/Resources/app.asar/node_modules/node-downloader-helper/dist/index.js:1:13483)\n    at WriteStream.<anonymous> (/Users/antoine.lheureux/dev/unity/hub.uw-hub/dist/mac/Unity Hub.app/Contents/Resources/app.asar/node_modules/node-downloader-helper/dist/index.js:1:9440)\n    at WriteStream.emit (events.js:327:22)\n    at WriteStream.EventEmitter.emit (domain.js:483:12)\n    at internal/fs/streams.js:247:14\n    at /Users/antoine.lheureux/dev/unity/hub.uw-hub/dist/mac/Unity Hub.app/Contents/Resources/app.asar/node_modules/graceful-fs/graceful-fs.js:61:14\n    at FSReqCallback.oncomplete (fs.js:160:23)","timestamp":"2021-04-20T10:48:35.225Z"}

Exception EFBIG

Thanks for this package. It works really well.

I try to download a file of size 2.67 GB. It gives error EFBIF: File too larget at around 84.5%. Is there a file size limitation and can I change it?

My node app runs on raspberry bi/raspbian jesse.

Add a return on end event for renamed file

When the override attribute is set to false and a file is downloaded, if there is already a file of that name, it creates a new one with a number attached. This naming convention is handled by the downloader, so there isn't a straightforward way the capture this renamed file or even know the file has been renamed. On the end event, there should be a returned object or value that indicates that the file has been renamed.

Missing extension when trying to download webpages

I just tried 'downloading' https://google.com using the following command:

new DownloaderHelper('https://google.com', '/home/downloads', {
      fileName: {name: 'test', ext: false},
      ...
    });

This will download the page, but the filename will be test., without the html part. On windows, this will cause a corrupt file which can't be renamed or deleted.

I'm guessing this is because of ext: false and the fact that there normally is no extension when loading a webpage.
However, this should not happen, at least not with the trailing dot.

When the file is only downloaded a part, but the URL expires, we need to update the URL to resume the download.

from @lyswhut in #15

When the URL expires, but the file is only partially downloaded, we need to refresh the URL to resume the download, instead of creating a new instance, what do you think?
They all belong to the "Resume Download" feature.
When you resume downloading, you can choose whether to update the URL.

So basically the idea will be to add something like dl.resume(newUrl); and this should update the URL

or probably be a fallback/mirror URL list

const urlMirror = [
 'http://...',
 'http://...',
 'http://...',
];
const dl = new DownloadHelper(urlMirror,__dirname);

Linked #13

Error event is emitted before every try even if download finished successfully

Hey, thank you for this package, it helps.

However, I found some confusing behavior that I believe is a bug. error event is emitted after every unsuccessful try even if download was successful.

In my understanding of EventEmitter errors, error event is (usually) a fatal one: it can't be recovered and you need to restart everything from scratch.

For example:

const { DownloaderHelper } = require("node-downloader-helper")

(async () => {
  const dl = new DownloaderHelper(
    "https://example.com/",
    ".", {
      fileName: "anything",
      retry: { maxRetries: 3, delay: 50 },
    }
  );
  dl.on("retry", console.debug);
  dl.on("error", console.error);
  dl.on("end", console.info);
  await dl.start();
})()

Suppose that network error occurs for first two times, but on third try file downloads successfully.

Expected behavior:

error event isn't fired if download was successful after all.

DEBUG: 1 { maxRetries: 3, delay: 50 }
DEBUG: 2 { maxRetries: 3, delay: 50 }
INFO:  {  fileName: 'anything', … , downloadedSize: 1256 }

Actual behavior:

error event is emitted before every try.

ERROR: Error: ECONNRESET
DEBUG: 1 { maxRetries: 3, delay: 50 }
ERROR: Error: ECONNRESET
DEBUG: 2 { maxRetries: 3, delay: 50 }
INFO:  {  fileName: 'anything', … , downloadedSize: 1256 }

Custom throttle timeout

The current throttle time is 1000ms, it would be great if we could configure that to be *ms in the downloader options since 1 second might be long for some usecases

Filename unexpectedly changed after download

When downloading a file with a dot prefix the actual file that is saved doesn't include the dot. Is there a way to disable this behavior? For my case, I need 1:1 the file names from the server on the local Disc.

Demo Code:

const { DownloaderHelper } = require('node-downloader-helper');

const dl = new DownloaderHelper(
    'https://files.taincer.com/testdata/.changelog.txt',
    __dirname,
    {
        override: true,
        fileName: ".changelog.txt"
    }
);

dl.on('end', () => console.log('Download Completed'))
dl.start();

Hi, how can I set the progress event fired at 100 ms?

Thank you very much for the package, it helps me a lot.
But I have a problem using it.

Because my app is download very small files, usually below 3 MB.
If the progress event fired 1 seconds a time, user can only see 2 or 3 numbers before it finished. It is really bad for user experience.

So, can I change the time interval? where to do it? Thanks.

Download finishes before reaching 100%

When download reaches 2%, status goes to FINISHED and end event is fired.
Server has accept-ranges: bytes and it's an nginx acting as CDN with Cloudflare layer over it.
Also it only happens over Windows. On linux it works fine.

I had to make this hack:

let download = new DownloaderHelper(url, destination);
let lastStat

download.on('progress', stats => {
        console.log(stats.progress);
        lastStat = stats
    })
    .on('stateChanged', state => {
        if (state == "FINISHED") {
            console.log("status changed", lastStat);
            if (lastStat.progress === 100) {
                resolve()
            } else {
                download.resume()
            }
        }
    })
    .start()

That way it keeps going until reaches 100%.

download Fail

When the file name of the downloaded file is too long, the file download will fail.

Event progress seems to spam the console.

It was working fine until this afternoon, then tonight when tried it again it seem to spammed the console.
Code to reproduce.

const { DownloaderHelper } = require('node-downloader-helper');
const ProgressBar = require('progress');
const chalk = require('chalk');

/**
 * Downloads file from the internet
 * @param {string} url - URL to download from.
 * @param {string} path - Path where the file should be saved.
 */
module.exports.download = (url, path) => {
	// eslint-disable-next-line no-useless-escape
	const fileNameRegex = /(?:[^/][\d\w\.]+)$(?<=\.\w{3,4})/gi;
	const fileName = fileNameRegex.exec(url);
	const dl = new DownloaderHelper(url, path);
	dl.on('progress', stats => {
		const bar = new ProgressBar(`${chalk.bgRed('Downloading')} ${chalk.blue(fileName)}
        [:bar] :percent ${byteHelper(stats.downloaded)}/${byteHelper(stats.total)} ${byteHelper(stats.speed)}`, {
			complete: '=',
			incomplete: ' ',
			width: 20,
			total: stats.total,
		});
		bar.tick(stats.downloaded);
	});
	dl.on('error', err => console.error(err));
	dl.start();
	dl.on('end', () => console.log(`Downloaded ${fileName}`));
};
/**
 * Converts bytes into readable values.
 * @param {number} value - Bytes you want to convert
 */
const byteHelper = (value) => {
	if (value === 0) {
		return '0 b';
	}
	const units = ['b', 'kB', 'MB', 'GB', 'TB'];
	const number = Math.floor(Math.log(value) / Math.log(1024));
	return `${(value / Math.pow(1024, Math.floor(number))).toFixed(1)} ${units[number]}`;
};

I even check if the bar is not making any issues as such , and i was write the bar as nothing to do with it I logged it by doing
console.log(${byteHelper(stats.downloaded)}/${byteHelper(stats.total)});
And it spammed the console.
Images:
with progress bar I made
Image
without progress bar
image
code where i execute the download function

const { download } = require('./download');
const fs = require('fs');
const { port, password } = require('../config.js').config.lavalink;
let baseConfig = fs.readFileSync('./lavalinkconfig.txt', 'utf8');

baseConfig = baseConfig
	.replace('{{port}}', port)
	.replace('{{password}}', password);

fs.writeFileSync('../Lavalink/application.yml', baseConfig);

download('https://github.com/Frederikam/Lavalink/releases/latest/download/Lavalink.jar', '../Lavalink/');

If you need more info, let me know if i managed to fix it will drop a comment how i managed to fix it and close the issue. Happy Hacking!

Ignore invalid https certificates?

Today I received the following error while trying to download a file:

events.js:187
      throw er; // Unhandled 'error' event
      ^

Error: unable to verify the first certificate
    at TLSSocket.onConnectSecure (_tls_wrap.js:1321:34)
    at TLSSocket.emit (events.js:210:5)
    at TLSSocket._finishInit (_tls_wrap.js:794:8)
    at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:608:12)
Emitted 'error' event on b instance at:
    at ClientRequest.<anonymous> (C:\Users\Chaphasilor\AppData\Roaming\npm-cache\_npx\37684\node_modules\node-downloader-helper\dist\index.js:1:9035)
    at ClientRequest.emit (events.js:210:5)
    at TLSSocket.socketErrorListener (_http_client.js:406:9)
    at TLSSocket.emit (events.js:210:5)
    at emitErrorNT (internal/streams/destroy.js:92:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
}

It seems like there's an issue with the ssl certificate, but when I visit the website the browser says the cert is valid.
I also tried downloading the files with iDownload Manager from the Microsoft Store, no issues there.

Any ideas on what could cause this and how to work around it? I'd be happy if there was a way to 'ignore' invalid certs (given that HSTS isn't used), but I'm not even sure if this isn't just a strange bug?

Bandwidth-problem on bigger videos?

Hey there, first of all thank for for this nice plugin. I have it installed within an node.js Electron application.

I have the problem, that if i am trying to download bigger videos (like >20 MB maybe?) the plugin starts nicely, but very quickly, performance drops down. I just measured the MB/s for the first 6 seconds:

image

Similar behaviour occurs with npm request plugin, but works flawlessly as plain browser download. Any chance to solve this by different config etc?

If not, maybe its solvable via "hack": Funnily If i pause the download every 3 seconds and resume it back, it quickly downloads 7MB-bunches step by step, but unfortunately, output video is malformed afterwards.

Download after connection interrupted

There is a way to auto resume downloading file after connection is interrupted ? I need a timeout option when connection is interrupted more than timeout so an error is triggered and if connection is established again so download will resume downloading.

Catch file deleted from computer error

When I delete a file while it's still downloading, how do I catch the error so I can handle it? I don't get anything in on('error') and I have no clue where else it could come from

CLI mode doesn't work since v1.0.14

To recreate:

  1. Use Node 14
  2. Install node-downloader-helper > 1.0.13 (npm i node-downloader-helper -g)
  3. Run ndh . <any url>
  4. Get the following error message:
internal/modules/cjs/loader.js:883
  throw err;
  ^

Error: Cannot find module '../example/helpers'
Require stack:
- ...\npm\node_modules\node-downloader-helper\bin\ndh
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
    at Function.Module._load (internal/modules/cjs/loader.js:725:27)
    at Module.require (internal/modules/cjs/loader.js:952:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (...\npm\node_modules\node-downloader-helper\bin\ndh:8:5)
    at Module._compile (internal/modules/cjs/loader.js:1063:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
    at Module.load (internal/modules/cjs/loader.js:928:32)
    at Function.Module._load (internal/modules/cjs/loader.js:769:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '...\\npm\\node_modules\\node-downloader-helper\\bin\\ndh'
  ]
}

Installing [email protected] fixed the issue.

Resuming download after restart results in actual file size being incorrect causing corruption

Hello,

I've got some base code setup using what I found in issue 13. However it seems that once the download "resumes" or starts again it doesn't realize the byte size the file already is and just re-downloads the file to the same location/file causing overlap and mismatched byte size. Is the package supposed to detect this once it resumes the download?

"DownloadFile": function(url,FullFilePath, FileName) {

    const dl = new DownloaderHelper(url, FullFilePath);

    if (fs.existsSync(FullFilePath + FileName)) {

      functions.getTotalFileSize(url).then(totalFileSize => {

        var stats = fs.statSync(FullFilePath + FileName)

        if (stats["size"] < totalFileSize && stats["size"] != totalFileSize) {

          dl.__total = totalFileSize; //<--- Workaround
          dl.__filePath = FullFilePath + FileName; //<--- Workaround
          dl.__isResumable = true; //<--- Workaround
          // Old Rate: .10MB
          //dl.pipe( new Throttle({ rate: 250000 }) );

          console.log("File found, download not complete, resuming...");
          dl.on('error', (err) => {
            dl.resume(); console.log('Download failed, Attempting Retry');
          }).on('timeout', () => {
            dl.resume(); console.log('Download timed out, Attempting Retry');
          }).on('end', () => {
            console.log('Download Completed');
          }).resume();

        }else{

          console.log("File download is already complete, file is in tact.");

        }

      });

    }else{

      console.log("File not found, starting download...");
      dl.on('error', (err) => {
        dl.resume(); console.log('Download failed, Attempting Retry');
      }).on('timeout', () => {
        dl.resume(); console.log('Download timed out, Attempting Retry');
      }).on('end', () => {
        console.log('Download Completed');
      }).start();

    }

  },

  "getTotalFileSize": function(_url) {
      const urlParse = URLFN.parse(_url);
      const options = {
          protocol: urlParse.protocol,
          host: urlParse.hostname,
          port: urlParse.port,
          path: urlParse.path,
          method: 'HEAD'
      };

      return new Promise((resolve, reject) => {
        const req = http.request(options, (res) => {
            const { statusCode } = res;

            if (statusCode !== 200) {
                reject(new Error(`Request Failed.Status Code: ${statusCode} `));
            }

            console.log(res.headers['content-length']);
            resolve(parseInt(res.headers['content-length']));
        });

        req.end();
    });
  }

No status change when cut off the network while downloading

"node-downloader-helper": "^1.0.11"

I try to follow every event, but nothing happen when network cut off and download actrually failed.
image

Event hang on like pictures above. And nothing happened neither while network back up online.

dl.on('error', err => {
console.log('download error', err)
})
dl.on('stop', err => {
console.log('download stop', err)
})
dl.on('timeout', err => {
console.log('download timeout', err)
})
dl.on('retry', err => {
console.log('download retry', err)
})
dl.on('stateChanged', stats => {
console.log('state changed', stats)
})

Using port number doesn't work

If you specify to download from a url with IP:PORT it will crash with

{ Error: getaddrinfo ENOTFOUND 34.235.190.68:3000 34.235.190.68:3000:3000
at errnoException (dns.js:53:5)
at onlookup (dns.js:95:5)
code: 'ENOTFOUND',
errno: 'ENOTFOUND',
syscall: 'getaddrinfo',
hostname: '34.235.190.68:3000',
host: '34.235.190.68:3000',
port: '3000' }

Downloads "freezing" for no obvious reason?

I've now experienced multiple times that while downloading a file, once in a while a download will just freeze and not continue to download. It doesn't fail, throws no error, doesn't change state, it just stops downloading. The dl object either stops emitting progress events or always reports the same stat, downloaded, progress, speed all stay the same.
In my application I have 5 download "slots" and if I download many files, those slots keep freezing on after the other, until every slot is filled with a frozen download and nothing gets downloaded anymore.

From my experience the downloads will stay frozen indefinitely, until I manually pause and then resume them (that is, if the download URL is still working and resume is supported).

I have no idea why this is happening or what exactly is causing this, so I can't tell you how to recreate the behavior. But I am pretty sure that the issue is with node-downloader-helper and not my app (the downloads really are frozen and don't just continue downloading in the background without the stats showing up).
The download also don't get slower before freezing (which could indicate server-side throttling), the last reported speed is similar to previous/average speeds.

Any idea why this could happen?

dl.pause() and dl.stop() not working properly on Node 16

When using the package with Node 16, calling dl.pause() or dl.stop() will result in the http response emitting an error event, which in turn causes dl to emit an error event as well. With dl.stop() this isn't a big deal (although still unexpected), but with dl.pause() this results in a retry, which means that after waiting for retry.delay, the download will be resumed instead of staying paused.

After "some" investigating, I've found out the following:

  • the original error is emitted by the http response
  • the reason is emitted seems to be that the http request is being canceled, which leads to the response socket to be destroyed
  • the error can be avoided by canceling the response before canceling the request, e.g. by calling response.destroy()
  • request.abort(), which is currently being used is deprecated and should be replaced with request.destroy() (which does the same)

Here's some code for recreating the issue:

Error when calling `dl.pause()` or `dl.stop()`
const { DownloaderHelper } = require('./dist')

const dl = new DownloaderHelper('http://samples.mplayerhq.hu/4khdr/LaLaLand_cafe_4K.mkv', __dirname, {
    fileName: 'test',
});

dl.start().catch(err => {
    console.error(`Download failed:`, err)
})

dl.on(`error`, error => {
    console.warn(`error:`, error)
})

dl.on(`download`, () => {
    setTimeout(async () => {
        let success = await dl.pause()
        // let success = await dl.stop() // uncomment me
        if (success) {
            console.log(`PAUSED`)
        } else {
            console.log(`FAILED`)
        }
    }, 2500)
})
dl.on(`retry`, (attempt, retryOpts) => {
    console.log(`RETRYING`)
    console.log(`attempt:`, attempt)
    console.log(`retryOpts:`, retryOpts)
})
dl.on(`progress.throttled`, progress => {
    console.log(`progress:`, progress)
})

process.stdin.on(`data`, data => {
    console.log(`data:`, data)
})
Error (and fix) for the `http` module
const http = require(`http`);
const fs = require(`fs`);

const url = "http://samples.mplayerhq.hu/4khdr/LaLaLand_cafe_4K.mkv";

let urlObj = new URL(url)

const requestOptions = {
  protocol: urlObj.protocol,
  host: urlObj.hostname,
  port: urlObj.port,
  path: urlObj.pathname,
  method: `GET`,
};

let response
const request = http.request(requestOptions, res => {
  response = res;
  console.log(`response received`)
  console.log(`statusCode: ${res.statusCode}`)
  if (res.statusCode == 200) {
    res.on(`data`, () => process.stdout.write(`.`))
    res.on(`error`, err => console.error(`RESPONSE ERROR:`, err))
    const fileStream = fs.createWriteStream(`test.file`, {});
    res.pipe(fileStream)
  } else {
    console.error(`Bad response`);
  }
});

request.on('error', error => {
  console.error(`request error:`, error)
})

request.end()

setTimeout(() => {

  // cancel response *before* canceling request
  // response.destroy(); //!!! uncomment to fix error
  
  request.destroy() // or `request.abort()`
}, 2500)

It should be a fairly easy fix, but I'd prefer not to contribute it myself because I'm not super-familiar with how the events and request/reponse streams should be handled.
If you really don't have the time to fix it yourself, it would be very appreciated if you could give me a rough outline of how and where I should change things :)

Implement Typescript definition

Based on the effort already made on https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/types/node-downloader-helper/index.d.ts

This would need to be placed probably inside ./types/index.d.ts and defined inside the package.json as "types": "./types/index.d.ts",

Hello @RemyJeancolas I noticed that you did the definition in DefinitelyTyped, I would like to request if you could consider implementing this inside this repository, so I could to contribute to updating it more frequently

How to prevent renaming?

If you download multiple files, you will add (1), (2) such strings later.
But the demand is to cover them instead of renaming
This library has related APIs that can prevent this?

Filenames parsed from the `Content-Disposition` headers are incorrect.

Hi there. We've recently identified an issue with this library when using the downloader alongside the JFrog Artifactory platform. This doesn't affect only this platform -- it could affect many platforms that use the aforementioned header.

If no fileName option is provided to node-downloader-helper, the helper will attempt to parse the file name from the Content-Disposition header. If the header is present, which it is for files downloaded from Artifactory, it can contain more than what the script expects.

This gives the file a mangled name. Incidentally, this can often pass on macOS/Unix, but will cause the download to fail on Windows.

As you can see in this function:

__getFileNameFromHeaders(headers, response) {
let fileName = '';
// Get Filename
if (headers.hasOwnProperty('content-disposition') &&
headers['content-disposition'].indexOf('filename=') > -1) {
fileName = headers['content-disposition'];
fileName = fileName.trim();
fileName = fileName.substr(fileName.indexOf('filename=') + 9);
fileName = fileName.replace(new RegExp('"', 'g'), '');
fileName = fileName.replace(/[/\\]/g, '');
} else {
if (path.basename(URL.parse(this.requestURL).pathname).length > 0) {
fileName = path.basename(URL.parse(this.requestURL).pathname);
} else {
fileName = `${URL.parse(this.requestURL).hostname}.html`;
}
}
return (
(this.__opts.fileName)
? this.__getFileNameFromOpts(fileName, response)
: fileName.replace(/\.*$/, '') // remove any potential trailing '.' (just to be sure)
)
}

on this line:

fileName = fileName.substr(fileName.indexOf('filename=') + 9);

The entirety of the header is included in this substring. There can be multiple properties in this header, as per spec. I think this header parsing needs to be done a little differently, perhaps with some sort of token splitting.

I have an example script I've been toying with that might have a potential solution:

const headers = {
  'content-disposition': `attachment; filename="Setup64.exe"; filename*=UTF-8''Setup64.exe`
};

const parseHeaderValue = (headerValue) => {
  let headerDictionary = headerValue.split(';');
  headerDictionary = headerDictionary.reduce((acc, val) => {
    const splitVal = val.trim().split('=');
    return { ...acc, [splitVal[0]]: splitVal[1] ?? '' };
  }, {});
  return headerDictionary;
}


const contentDispositionValues = parseHeaderValue(headers['content-disposition']);

if(headers.hasOwnProperty('content-disposition') &&
            contentDispositionValues['filename']) {
  let fileName = headers['content-disposition'];

  fileName = fileName.trim();

  fileName = contentDispositionValues['filename'];

  fileName = fileName.replace(new RegExp('"', 'g'), '');

  fileName = fileName.replace(/[/\\]/g, '');
  
  console.log(fileName);
}

In our codebase, we solved this problem by always passing in a fileName optional property when constructing the DownloaderHelper, but I'm hoping this will help eliminate a frustrating edge case we experienced. Thank you :)

The best package out there!

Hey! I've spent hours trying got, axios, download and some others. This one is the best package to download a file, quick and simple. Thanks for it!

Add an event during the downloadRequest phase

Hello!
Can you add an event during the downloadRequest phase?
When the downloadRequest phase request is successful, send an event containing the file information (such as file size) that will be downloaded, so that I can save the file information for recovery download when the event is triggered.

UnhandledPromiseRejection-Error when server responds with 403

I have encountered an error where a promise rejection isn't properly caught inside the module.
It happens when the server responds with status 403.

Here's the log:

(node:29744) UnhandledPromiseRejectionWarning: Error: Response status was 403
    at ClientRequest.<anonymous> (D:\Code\NodeJS\njoy\node_modules\node-downloader-helper\dist\index.js:1:5974)
    at Object.onceWrapper (events.js:300:26)    at ClientRequest.emit (events.js:210:5) 
    at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:583:27)       
    at HTTPParser.parserOnHeadersComplete (_http_common.js:115:17)
    at TLSSocket.socketOnData (_http_client.js:456:22)
    at TLSSocket.emit (events.js:210:5)     
    at addChunk (_stream_readable.js:308:12)    at readableAddChunk (_stream_readable.js:289:11)
    at TLSSocket.Readable.push (_stream_readable.js:223:10)
(node:29744) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:29744) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. 
In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

The .on('error') handler is working and catching the error, but it seems like it is thrown again afterwards.
I tried wrapping the download constructor inside a try...catch block but that didn't help.

Any ideas why this is happening?

Skip not working for already downloaded file

    const dl = new DownloaderHelper(fileURL, __dirname, {
      override: {
        skip: true,
        skipSmaller: false,
      },
    })

I'm currently using this configuration to skip files that have already been downloaded. According the readme, this should work. However, it is still re-downloading the file. Am I using it wrong?

Note: It is overwriting the old file.

Response status was 500

There are two types of file download links, starting with http://10.154.3.130/drive/newDownloadFile.do?appFileId=d50996dc730211ea9ac2fa163... and http://10.154.3.158:8090becpan5201805111716578384cc46ccf8ae6f43b6a6b5....

The former link comes from the server, the latter link downloads the file directly from the Object-based Storage.

The former can be successfully downloaded in the browser, but using node-downloader-helper will report an error and return Response status was 500.
The latter can be downloaded successfully using node-downloader-helper.

Can you help me? Thanks! @hgouveia

added a background download feature

Can you please add a download feature in the background, so when we switch our browser, the download process will still run on our vps server, it might sound like a cron job.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.