node-modules / urllib Goto Github PK

Request HTTP(s) URLs in a complex world.

License: MIT License

JavaScript 1.46% TypeScript 98.54%

request curl curl-library httpclient urllib2 urllib http-client undici

urllib's Issues

Parse Error cause callback twice!!

'use strict';

var co = require('co');
var urllib = require('../');

co(function* () {
  var result = yield urllib.requestThunk('http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php');
  console.log('status: %s, body size: %d, headers: %j',
    result.res.statusCode, result.data.length, result.res.headers);
})();

$ DEBUG=urllib node --harmony examples/co_urllib.js 
  urllib Request#1 GET http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php with headers {"User-Agent":"node-urllib/2.0.1 node/v0.11.14 (darwin 14.0.0; x64)"}, options.path: /go/market/promotion-act/shengxian1111_mobile.php +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php `req response` event emit: status 200, headers: {"server":"Tengine","date":"Fri, 31 Oct 2014 15:06:04 GMT","content-type":"text/html; charset=GB2312","transfer-encoding":"chunked","connection":"keep-alive","vary":"Accept-Encoding","expires":"Fri, 31 Oct 2014 15:06:04 GMT","cache-control":"max-age=0"} +67ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1179 +3ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 11520 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +6ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +2ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +2ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +2ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 790 +2ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 642 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 2880 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +4ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 2880 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 2880 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +2ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 325 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1107 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 2880 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +2ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +2ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +2ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 2880 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +2ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1210 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 222 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +2ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +3ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 2880 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1440 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 202 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 1231 +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res data` event emit, size 117 +1ms
  urllib Request#1 /go/market/promotion-act/shengxian1111_mobile.php `req error` event emit, RequestError: Parse Error (req "error") +0ms
  urllib [143ms] 0 bytes HTTP -1 GET /go/market/promotion-act/shengxian1111_mobile.php +1ms
  urllib Request#1 /go/market/promotion-act/shengxian1111_mobile.php: `req close` event emit +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res aborted` event emit, total size 115025 +1ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res end` event emit, total size 115025, _dumped: false +0ms
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: Remote socket was terminated before `response.end()` was called +1ms
[urllib:warn] [Fri Oct 31 2014 23:06:04 GMT+0800 (CST)] [worker:3654] GET http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php callback twice!!!
  urllib Request#1 http://www.tmall.com/go/market/promotion-act/shengxian1111_mobile.php: `res close` event emit, total size 115025 +0ms
/Users/mk2/git/urllib/node_modules/co/index.js:292
    throw err;
          ^
Error: Parse Error
    at Error (native)
    at Socket.socketOnData (_http_client.js:309:20)
    at Socket.emit (events.js:107:17)
    at readableAddChunk (_stream_readable.js:162:16)
    at Socket.Readable.push (_stream_readable.js:125:10)
    at TCP.onread (net.js:514:20)

request from urllib, couldn't be captured by local http proxy? (charles)

so it's hard to debug, because i can't see what's actually posted to server

去除错误信息里面的 Request#%d

导致同一个错误的 message 都不一致，�做数据统计的时候比较麻烦。可以放到一个扩展字段里面？err.requestId ?

Error message triggered if callback not provided

If I call urllib.request() without specifying the optional callback parameter, I see this perplexing error in the console:

[urllib:warn] [Thu May 01 2014 10:03:27 GMT-0700 (PDT)] [worker:4225] POST http://localhost:4000/webhook_cb callback twice!!!

I certainly wasn't calling the callback twice, in fact I wasn't using the callback at all. From looking at the code, it looks like this console warning is completely superfluous.

POST空data会导致nginx报411 Length Required

lib/urllib.js 105-111行：

  if (body) {
    var length = body.length;
    if (!Buffer.isBuffer(body)) {
      length = Buffer.byteLength(body);
    }
    options.headers['Content-Length'] = length;
  }

后面应该加上

 else {
    options.headers['Content-Length'] = 0
  }

否则当没有data或content时，nginx会报错，因为没有Content-Length。

request timeout should req.destroy() instead of req.abort()

Make sure req socket destroy and emit close event. Otherise, keepalive agent socket will leak...

http default protocol for URL argument

Suggestion: Could we have "http" being default protocol if none is specified in the URL argument. Several libraries (curl, postman) follow this convention.

urllib.request('www.server.com');

Example: URL "www.server.com" will be rewritten as "http://www.server.com" since no protocol is specified.

how to send https request with a .p12 certificate?

hello, my code is:

var url = "https://api.mch.weixin.qq.com/mmpaymkttransfers/sendredpack";

var options = {
    method: "POST",
    data: content,
    dataType: "text",
    pfx: p12_path,
    passphrase: params.mch_id,
    ca: ca_path,
    rejectUnauthorized: true
};

urllib.request(url, options, callback);

p12_path is .p12 file path, and params.mch_id is the password of the .p12 file

it seems not work

thanks very much

native-or-bluebird is deprecated. please switch to any-promise

https://github.com/kevinbeaty/any-promise

this implementation supports more promise implementations and allows the end-user to configure the promise implementation.

Auto convert data to json string when content-type is 'json'

var params = {
  headers: {'Content-Type': 'application/json'},
  data: {foo: 'bar', now: new Date()}
};
urllib.request(url, params, callback);

Error: Hostname/IP doesn't match certificate's altnames while using https

I am getting below error if I try to use https urls having custom headers:

Error: Hostname/IP doesn't match certificate's altnames
at SecurePair. (tls.js:1379:23)
at SecurePair.EventEmitter.emit (events.js:92:17)
at SecurePair.maybeInitFinished (tls.js:982:10)
at CleartextStream.read as _read

This is my code:

urllib.request(urlToCall, { rejectUnauthorized:false, headers:requestHeaders }, function (err, data, response) {
if (err) {
throw err; // you need to handle error
}
});

As per documentation, I added rejectUnauthorized to false, but still I am getting same error. Can you please help me to know why this error occurs?Since I am comfortable with this library I don't want to switch to another library just because of this issue.

detect agent socket event

Make ConnectionTimeout meaning timeout on connection...

Promise unit tests failed

AssertionError: expected {
  'accept-ranges': 'bytes',
  connection: 'Keep-Alive',
  'content-length': '6021',
  'content-type': 'text/html',
  date: 'Thu, 20 Nov 2014 04:32:02 GMT',
  'last-modified': 'Fri, 14 Nov 2014 22:18:43 GMT',
  'proxy-connection': 'Keep-Alive',
  server: 'nginx',
  via: '1.1 NET-CENTER'
} to have property connection of 'keep-alive' (got 'Keep-Alive')

RequestError: Parse Error

我在Express里面有一个简单的返回，但是urllib似乎认为这个不是JSON。得到的error为

{ [RequestError: Parse Error, POST some_url]
  bytesParsed: 0,
  code: 'HPE_INVALID_CONSTANT',
  name: 'RequestError' }

回答方

app.all('/test', function(req, res) {
  var result = ["hello", "world"];
  res.end( JSON.stringify(result) ); //这个的确是JSON啊
});

请求方

urllib.request(url, {
  type: 'POST',
  data: {
    id: 1,
    type: 'text',
    text: 'hello'
  },
  timeout: 4500,
  dataType: 'json'
}, function(err, data) {
  console.log(err);
  console.log(data);
});

deprecate requestThunk?

error may trigger multiple times due to a bug, causing program crashing

See nodejs/node-v0.x-archive#5851

  req.once('error', function (err) {
    if (!__err && err.name === 'Error') {
      err.name = 'RequestError';
    }
    err = __err || err;
    debug('Request#%d %s `req error` event emit, %s: %s', reqId, options.path, err.name, err.message);
    done(err);
  });

no support for proxy?

I ctrl+f in urllib's readme and source code, seems there is no any code for proxy?

Auto set `Transfer-Encoding: chunked ` on streaming request

返回stream无法直接解压

var co = require('co');
var path = require('path');
var urllib = require('urllib');
var zlib = require('zlib');
var tar = require('tar');
co(function*() {
  var url = 'http://registry.npm.taobao.org/co/download/co-4.5.4.tgz';
  var stream = yield urllib.request(url, {
    streaming: true
  });
  yield unpack(path.join(__dirname, '/co'), stream.res);
})
.then(function() {
  console.log('end');
})
.catch(function(e) {
  console.log(e);
});

function unpack(dest, stream) {
  return function(callback) {
    stream
    .pipe(zlib.createGunzip())
    .pipe(tar.Extract({ path: dest }))
    .on('finish', callback)
    .once('end', callback);
  };
}

结果报错了：

events.js:141
      throw er; // Unhandled 'error' event
            ^
Error: incorrect header check
    at Zlib._handle.onerror (zlib.js:353:17)

上面的代码改成var stream = fs.createReadStream('./co-4.5.4.tgz');是没有问题的。

自动添加 'Content-Type': 'application/x-www-form-urlencoded'

当传递了 POST url , args.data = {foo: 'bar'} 并且未设置 Content-Type ，需要自动添加 'Content-Type': 'application/x-www-form-urlencoded' header

try to reappear fin packet loss

nodejs/node-v0.x-archive#8135

provide remoteAddress info in global response event

var httpclient = require('urllib').create();

httpclient.on('response', function (info) {
  error: err,
  ctx: args.ctx,
  req: {
    url: url,
    remoteAddress: remoteAddress,
    options: options,
    size: requestSize,
  },
  res: res
});

httpclient.request('http://nodejs.org', function (err, body) {
  console.log('body size: %d', body.length);
});

README on Promise

The readme on Promise part is not proper. Actually, the project uses any-promise while not register bluebird. So people should be told to register their promise, otherwise the native promise would be used.

req abort event

nodejs/node#945

Api not friendly

Only on method

http.request(url[, options][, callback])

But the options object is so big, That is very hard to config.

I think superagent provide api will more firendly.

global `response` event

urllib.on('response', function (info) {
  info.error
  info.res
  info.data
});

http_proxy support

cnpm/cnpmjs.org#607

timing

res.data is null when request an http blank file

As the code below:

var res = yield urllib.request(map.urlpath, {
    headers: {
      host: context.host
    }
});

console.log(res.data); //=>null

The res.data should not be null but should be String ''

self signed certificate in certificate chain

self signed certificate in certificate chain
这个问题要如何处理

上传文件时需要设置 agent 的 keepAliveTimeout 为 0

var fs = require('fs')
var Agent = require('agentkeepalive')
var HttpsAgent = Agent.HttpsAgent
var urllib = require('urllib')
var formstream = require('formstream')

var form = formstream();
form.stream('file', fs.createReadStream('./ddd.txt'), 'ddd.txt')

urllib.request('REMOTE', {
  method: 'POST',
  agent: new Agent(),
  httpsAgent: new HttpsAgent({
    //keepAliveTimeout: 0,  这行不设置的话，传一会儿会报错
  }),
  headers: form.headers(),
  stream: form,
  timeout: 1000 * 60 * 60,  // 60min
}).then(res => console.log(res), err => console.log(err))

{ [ResponseError: socket hang up (req "error"), POST

...

  status: -1,
  headers: {},
  res:
   { status: -1,
     statusCode: -1,
     headers: {},
     size: 0,
     aborted: false,
     rt: 35883,
     keepAliveSocket: false,
     data: undefined,

default user-agent

Should it be if (!options.headers['User-Agent'] && !options.headers['user-agent']) { rather than if (!options.headers['User-Agent'] || !options.headers['user-agent']) { in lib/urllib.js on line 280 ?

when will data in callback function be undefined?

urllib.request(url, options, function(err, body, resp){

        if(err){
            callback(err);
            return;
        }

        if(body.errcode){
            callback(body);
            return;
        }

        // logic
    });

we found body is undefined while err is null (means no error happend), then program crash because of "Cannot read property 'errcode' of null"

http(s).request(url) is not save, should try {} catch() {}

https://github.com/nodejs/node/blob/master/lib/_http_outgoing.js#L308

the outgoing implemention will throw error when header value contains invaild ascii character

code like this:

urllib.request('test', {headers: {'x-test': '中文'}}, function(){})

query params will be ecnode twice on node 0.11+

/get?sql=SELECT%2520*%2520from%2520table&data=%25E5%2591%25B5%25E5%2591%25B5

support promise

var urllib = require('urllib');

urllib.request('http://nodejs.org').then(function (result) {
  console.log('status: %s, body size: %d, headers: %j', result.res.statusCode, result.data.length, result.res.headers);
}).error(function (err) {
  console.error(err);
});

how to post xml data

`req.end` should be called in any http request

server code:

require('http').createServer(function (req, res) {
  req.on('end', function () {
    res.end('hello world');
  });
}).listen(8124);

client code:

var urllib = require('urllib');

urllib.request('http://localhost:8124/hello', function (err, data, res) {
  console.log(err);
});

Differentce between node v0.8.14 and v0.10.21:

v0.8.14, http request ok
v0.10.21, RequestTimeoutError

var stream = yield urllib.request(url, {
  streaming: true
});

this.status = stream.status;
this.headers = stream.headers;
this.body = stream;

node-modules / urllib Goto Github PK

urllib's Issues

Recommend Projects

Recommend Topics

Recommend Org