saltwaterc / http-request Goto Github PK

your dependency on "mmagic" requires Python, so it does not install on Windows. Really shouldn't need Python for node modules anyhow.

gyp ERR! configure error
gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
gyp ERR! stack     at failNoPython (c:\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:103:14)
gyp ERR! stack     at c:\nodejs\node_modules\npm\node_modules\node-gyp\lib\configure.js:64:11
gyp ERR! stack     at FSReqWrap.oncomplete (evalmachine.:99:15)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "c:\\nodejs\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd c:\Projects\workspace\uie\metrics\node_modules\http-request\node_modules\mmmagic
gyp ERR! node -v v0.12.1
gyp ERR! node-gyp -v v1.0.2
gyp ERR! not ok
npm ERR! Windows_NT 6.1.7601
npm ERR! argv "c:\\nodejs\\node.exe" "c:\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "install" "http-request" "--save"
npm ERR! node v0.12.1
npm ERR! npm  v2.5.1
npm ERR! code ELIFECYCLE

npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-gyp rebuild'.

Hi,

sorry, another issue :)

When using the HEAD-function of http-get and requesting an URL that returns a error (404, 403, 502, etc.) the result object is not set. For example try this URL:
http://t.co/w0ErWk1f

This resovles to:
http://www.republicof3.com/5-ux-trends-of-2012-continuing-in-2013/

When making a HEAD-request for the t.co URL the resulting URL returns a 403 error, but it would be nice when the result object has at least the last URL that was called, otherwise you don't have a clue what URL was responsible for the error.

Thank you,
Max

When using http.get(options, callback), I get this warning in my console:

Warning: The options.bufferType = "string" option is deprecated. Use options.bufferType = "buffer"; instead.

Node version: v0.10.15
Module versions:

├─┬ [email protected]
│ ├── [email protected]
│ └── [email protected]

I get into a file and the callback is never triggered.

http.get({url:url}, encryptedURL, function (err, res) {console.log(err,res);});

This never prints in my case.

However the file is downloaded without errors.

You can read bytes expected (from content-length header) and bytes received so far. Would be especially useful for downloading huge files.

http.on('progress', function(bytesExpected, bytesTotal) {
  // => logic
});

I've found that URLs that require authorization(http://user:[email protected]) are not requested properly.
I had to disable

        if (url.auth) {
            console.log('Brokern Auth!');
            opt.headers.authorization = new Buffer(url.auth).toString('base64');
        }

in tools.js, and to add simple

opt.auth = url.auth;

few strings above, to make it work properly

If http-request is the successor to http-get, please clean up npm. It's really confusing having two different npm packages pointing to the same git repo.

If http-get is retired, please deprecate the package on npm.

streams encoded with anything other than utf-8 will fail loading.
I think that can be corrected if we detect the encoding and set the buffer options.encoding to binary if it's other than utf-8, but I can't figure out the right place to do that.
Has anybody tried this?

There is a problem with gzbz2 at the moment.
Any alternatives?

23 Nov 15:03:43 - Downloading file: http://www.canalpatrimonio.com/imagftp/im158430botticelli_birth_venus_2.jpg 

/var/www/illuminatis/application/scripts/node_modules/http-get/lib/http-get.js:259
                            var gunzip = gunzipstream.wrap(res, {encoding: null});
                                 ^
TypeError: Cannot call method 'wrap' of undefined
    at ClientRequest.<anonymous> (/var/www/illuminatis/application/scripts/node_modules/http-get/lib/http-get.js:259:34)
    at ClientRequest.emit (events.js:67:17)
    at HTTPParser.onIncoming (http.js:1226:11)
    at HTTPParser.onHeadersComplete (http.js:102:31)
    at Socket.ondata (http.js:1125:24)
    at TCP.onread (net.js:334:27)

gzbz2 module is installed, but I see you do not require the module, but instead a file relative to http-get.js ¿?

EDIT: ok I don't understand where to put the gzbz2 module or how to disable gzip. I tried "nogzip" option but still breaks...

Hi,

first of all: thank you for http-get, by far the best working lib for making http-requests in nodejs!

I just found one issue: try to get this url using http-get:
http://t.co/zf2bOPDi
t.co redirects you to this url:
http://www.spex.de/2013/01/06/spex-mixtape-342-bytefm-januar-2013/#bytefmplayer

and with http-get i get an 404 error because the stuff after the # is send to the server too, which should not be the issue, so http-get should report under result.url this url and should remove the part after the # when it tries to get the data.

Thank you,
Max

Hi,

don't know if this is http-get only related or an issue with node 0.10.10:
i get from time to time when i fetch a lot of URLs this error:

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: stream.push() after EOF
    at readableAddChunk (_stream_readable.js:146:15)
    at IncomingMessage.Readable.push (_stream_readable.js:127:10)
    at HTTPParser.parserOnBody [as onBody] (http.js:143:22)
    at Socket.socketOnData [as ondata] (http.js:1556:20)
    at TCP.onread (net.js:510:27)

any ideas how to fix this?
Thank you!

Hi,

sorry, today is my big testing day :)
When i try to get this URL:
http://www.amazon.co.uk/Toasters-Kitchen-Appliances-Home-Garden/b?ie=UTF8&node=11716951

i get this error-object:

{ [Error: Parse Error]
bytesParsed: 20,
code: 'HPE_INVALID_CONSTANT',
headers:
{ date: 'Wed, 03 Jul 2013 14:52:20 GMT',
server: 'Server',
'x-amz-id-1': 'A42B6F61ACC04C2EB8DE',
'x-amz-id-2': 'w/dGiH/4tTFFRtLq9QdWOm4H/6mMIGkocOA7Jwoket0cOIGKtudh1A==',
'content-type': 'text/html;charset=UTF-8',
'content-encoding': 'gzip',
vary: 'Accept-Encoding,User-Agent',
'set-cookie':
[ 'session-id=279-3003121-1937300; Domain=.amazon.co.uk; Expires=Tue, 28-Jun-2033 14:52:20 GMT; Path=/',
'session-id-time=2003583140l; Domain=.amazon.co.uk; Expires=Tue, 28-Jun-2033 14:52:20 GMT; Path=/' ],
'transfer-encoding': 'chunked' },
url: 'http://www.amazon.co.uk/Toasters-Kitchen-Appliances-Home-Garden/b?ie=UTF8&node=11716951',
method: 'GET' }

Any idea why?

Thank you,
Max

Shows this message, doesn't work.
Node v0.8.16

I noticed that http-get only returns results if the HTTP status code is 200. For some status codes (like 400 & 403), per the HTTP/1.1 spec, more information can be returned in the body. I'm seeing this with doing calls to last.fm's API - if you pass some incorrect data (wrong username or password), the HTTP code is 403 but it also includes valid XML describing the error. As is, the http-get library ends up not getting any of this content. Could some of these error codes be included along with 200?

I was able to get it to work by using a local copy and adding case statements for the two codes I gave above, but I'm developing a small API I'd like to share and would love it if the latest version supported these.

I'm getting DEPTH_ZERO_SELF_SIGNED_CERT errors despite using "noSslValidation" option:

      http.get({                                                      
                    url : tiles_url,                                            
                    // We don't need ssl validation during tests                
                    noSslValidation : true                                      
                }, ...

The stack trace:

     Error: DEPTH_ZERO_SELF_SIGNED_CERT                                         
      at SecurePair.<anonymous> (tls.js:1370:32)                                
      at SecurePair.maybeInitFinished (tls.js:982:10)                           
      at CleartextStream.read [as _read] (tls.js:469:13)                        
      at CleartextStream.Readable.read (_stream_readable.js:320:10)             
      at EncryptedStream.write [as _write] (tls.js:366:25)                      
      at doWrite (_stream_writable.js:226:10)                                   
      at writeOrBuffer (_stream_writable.js:216:5)                              
      at EncryptedStream.Writable.write (_stream_writable.js:183:11)            
      at write (_stream_readable.js:583:24)                                     
      at flow (_stream_readable.js:592:7)                                       
      at Socket.pipeOnReadable (_stream_readable.js:624:5)                      
      at emitReadable_ (_stream_readable.js:408:10)                             
      at emitReadable (_stream_readable.js:404:5)                               
      at readableAddChunk (_stream_readable.js:165:9)                           
      at Socket.Readable.push (_stream_readable.js:127:10)                      
      at TCP.onread (net.js:528:21)                          

I've also exported NODE_TLS_REJECT_UNAUTHORIZED=0 but have the same problem.

Node version is v0.10.26

Hi guys,

Any particular reason to block node requirement to v0.6.11? Any used API not in v0.6.10?

Using async.auto I am running an http.get() with timeout. After the timeout error, http.get is ran again causing a Callback is already called error. I am getting this error because http.get is ran twice, one time after the timeout.

Does this module work in Windows? I get the following error when running 'npm install http-request'

gyp http GET http://nodejs.org/dist/v0.10.22/node-v0.10.22.tar.gz
gyp http 200 http://nodejs.org/dist/v0.10.22/node-v0.10.22.tar.gz
gyp WARN install got an error, rolling back install
gyp ERR! configure error
gyp ERR! stack Error: incorrect header check
gyp ERR! stack at Zlib._binding.onerror (zlib.js:295:17)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "node" "C:\Program Files\nodejs\node_modules\npm\node_modu
les\node-gyp\bin\node-gyp.js" "rebuild"
gyp ERR! cwd C:\Testing\v1\node_modules\http-request\node_modules\mmmagic
gyp ERR! node -v v0.10.22
gyp ERR! node-gyp -v v0.11.0
gyp ERR! not ok
npm ERR! [email protected] install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script.
npm ERR! This is most likely a problem with the mmmagic package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get their info via:
npm ERR! npm owner ls mmmagic
npm ERR! There is likely additional logging output above.

npm ERR! System Windows_NT 6.1.7601
npm ERR! command "C:\Program Files\nodejs\node.exe" "C:\Program Files\nod
ejs\node_modules\npm\bin\npm-cli.js" "install" "http-request"
npm ERR! cwd C:\Testing\v1
npm ERR! node -v v0.10.22
npm ERR! npm -v 1.3.14
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR! C:\Testing\v1\npm-debug.log
npm ERR! not ok code 0

Hi,

the documentation says that result (if successful, of course) contains the final url that was used in result.url, but it looks like this is not set, or am i wrong?

Thank you,
Max

Hello

I'm trying to scrap a site with iso-8859-1

    <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

I'm getting � for the accents. How can I configure the encoding for iso-8859-1?

Cheers,
Pablo Cantero

Hi. It would be useful to have an option to preserve the timestamps when downloaded files are saved to this. This happens to be default behaviour of wget and other download software http://www.gnu.org/software/wget/manual/wget.html

By default, when a file is downloaded, it's timestamps are set to match those from the remote file. This allows the use of ‘--timestamping’ on subsequent invocations of wget. However, it is sometimes useful to base the local file's timestamp on when it was actually downloaded; for that purpose, the ‘--no-use-server-timestamps’ option has been provided.

Modern web browsers don't preserve timestamps:

• Firefox https://bugzilla.mozilla.org/show_bug.cgi?id=178506
• Chrome http://code.google.com/p/chromium/issues/detail?id=4574

The redirects with relative paths use the host from the URL instead of using it from the request host header.

In the documentation, the signature for each is something like get(options, file, callback). In the examples, however, it doesn't always follow this pattern.

http.get('http://localhost/get', function (err, res) {
    if (err) {
        console.error(err);
        return;
    }   
    console.log(res.code, res.headers, res.buffer.toString());
});

In this case, it follows the pattern get(file, callback). Is this intentional? Is the options parameter optional and your code intuits when there is not options passed in?

Hi,

i am using node-request and when i try to get this URL:
http://biocache.ala.org.au/occurrence/bc5de9c6-41fe-49df-9e7c-2217399112cf

I get this errormessage:
Error: 140063509739328:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:../deps/openssl/openssl/ssl/s23_clnt.c:766:

with wget, curl and in the browser the URL is working, any idea why? Or is this a problem with the Openssl- version (the standard one) have on my ubuntu 12.04 LTS?

Thank you,
Max