node-js-libs / curlrequest Goto Github PK

By default the location option is set to true. It is not possible to disable this without also specifying max-redirs due to the following bit:

//Follow location by default
if (!options['max-redirs']) {
    options.location = true;
    options['max-redirs'] = 3;
}

Solution:

    //Follow location by default
    if (!options.location) {
        options.location = true;
    }

    if (!options['max-redirs']) {
        options['max-redirs'] = 3;
    }

URL to curl request:
https://bibles.org/v2/passages.js?q[]=john+3:1-5&version=eng-ESV

Returns error:
Error: URL malformed. The syntax was not correct.

I think I've narrowed the issue to the URL having brackets, since removing the brackets allows the request to run (though brackets are necessary for a proper response).

API: http://bibles.org/pages/api/documentation/passages
Note: API Key required to run query.

sample url
https://api.getbase.com/v2/deals?custom_fields[my_custom_field]=0

The script works but the zip file is corrupted. What am I doing wrong here?

// curl -LOk  -X 'GET' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:51.0) Gecko/20100101 Firefox/51.0' -H 'Referer: https://nseindia.com/products/content/equities/equities/archieve_eq.htm' -H 'Upgrade-Insecure-Requests: 1' 'https://nseindia.com/content/historical/EQUITIES/2000/MAR/cm7MAR2000bhav.csv.zip'


var curl = require('curlrequest');
var fs   = require( 'fs' );

var options = {
    url: 'https://nseindia.com/content/historical/EQUITIES/2000/MAR/cm7MAR2000bhav.csv.zip',
    verbose: true,
  	headers: { 
  		'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:51.0) Gecko/20100101 Firefox/51.0',
  		'Referer': 'https://nseindia.com/products/content/equities/equities/archieve_eq.htm',
  		'Upgrade-Insecure-Requests': '1'	
  	}
};

var file = fs.createWriteStream("bhav.zip");

curl.request(options, function (err, data) {
    file.write(data);
});

How to tell it not to verify ssl certificate?

Hi,

Just encountered a strange thing. When using your library to retrieve weather messages using the following code, I ended up with some missing HTML. Here's the snippet:

var curl = require('curlrequest');
var code = icao_code.toUpperCase();
var NOAA_scraper_curlopts;
var url = "http://aviationweather.gov/adds/metars/index.php?station_ids="+code+"&chk_metars=on&chk_tafs=on&std_trans=standard&submit=1";
NOAA_scraper_curlopts = {
url : url,
method:'GET'
};
curl.request(NOAA_scraper_curlopts,function(err,data){NOAA_Scraper_Callback(err,data);});

'data' contains the retrieved HTML but it never goes to the end of the online content.

Any idea why this is occuring ? I ended up choosing your lib cause it connected flawlessly to a SSL site I wanted to scrape. Hope you'll find out what's going on as I have no clue so far.

Thanks for having a look.

Uncaught Error: spawn curl ENOENT at exports._errnoException this error only happening on windows 10. need to use spawn tried cross-spawn no luck

Thanks in advance.

Why I'm fine with a create crud op when done with a get request from browser or curl desktop client while when I do the same with curlrequest in nodejs app I recieve following error

The server didn't reply anything, which here is considered an error.

I am trying to perform a curl command to a RabbitMQ broker- this requires passing the credentials, but there is no client support for 'auth'

Hi there, i'm trying to send some parameter via POST in the query string and i have something like that:

var _url = 'htttp://url_to_post_data'
var options = {
        url: _url,
        include: true,
        method: "post",
        headers: {
          queryString: {
             argument1: "test",
             argument2: "test2" 
          } 
       }
    };

curl.request(options, function (err, data) {
     console.log(data)
})

But there is no way the POST request submits the query string form-data.

This code works 5-6 months ago, but when I got back and update may node to 0.10.31, I got the error below:

options.curl_path appears in index.js, but isn't used anywhere.

How would someone get the last redirect URL?

I am using include: true to include headers, but I don't need body in the response. How would someone exclude it?

Using gleak, I found some globals in one of my projects.

I traced it down to the first line of proxy.js, which starts off with "e=".

Hello, I haven't find out the proxy parameter, does the curlrequest supports -x or --proxy command?

Ok, i kind of figured it out :-)
So the include flag is important here and the sent request header is not visible, but the options that are passed to curl - otherwise i can intercept the request and check the header with http://requestb.in. That worked for me.

I'm trying to connect to a webdav server that requires user credentials. When I pass an empty string as the user and the connection times out, the values passed to the callback are undefined and null. Also, when I don't pass the user credentials at all (on the same server that requires login) the second argument (return value) passed to the callback is empty.

I'm not sure if there is a part of the webdav protocol that handles user auth, but it would be nice to have errors for both of these situations.

Here are the 'arguments' to the callback function when the user is passed as an empty string:

{ '0': undefined,
  '1': null,
  '2': 
   { cmd: 'curl',
     args: 
      [ '--silent',
        '--show-error',
        '--no-buffer',
        '--url',
        '***REMOVED***',
        '--user',
        '',
        '--verbose',
        '--max-time',
        10,
        '--request',
        'GET',
        '--location',
        '--max-redirs',
        3,
        '--header',
        'Accept: */*',
        '--header',
        'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3',
        '--header',
        'Accept-Language: en-US,en;q=0.8',
        '--user-agent',
        'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)' ],
     time: 10006 } }

const options = {
curl_path: '/path/to/curl'
}

it seems , curl_path doesn't work, it still using default cmd

How would i make this curl command work with curlrequest?

curl -X POST -i -F [email protected] -u user:password https://url.com/upload

I read #20 but i can't see how to add user and password.

Hello,

Could you please help me to generate a curl request with your library such as:

curl --digest -u username:password "http://httpbin.org" --form imageURL=@"image.jpg" -X PUT

My tentative fails:

err: Failed to initialize.
meta.args["--silent","--show-error","--no-buffer","--url","http://httpbin.org/put","--form",{"imageURL":"https://../test.jpg"},"--digest","--username","xxXXxxxXXX","--password","xxXXXxxxXXX","--verbose","--request","put","--location","--max-redirs",3,"--header","Accept: /","--header","Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3","--header","Accept-Language: en-US,en;q=0.8","--user-agent","Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13"]

To make this work on windows, I added a configuration for the path to curl after line 231.

BTW. This is great! Keep it up!

I'm on OSX 10.9.3 and whenever I run a script that hits localhost:1337, I receive this error in my callback:

Couldn't resolve host. The given remote host was not resolved.

If I pass in a function as the options.process parameter, it is called with the correct response from the server.

Regardless of whether the server returns a status of 400 or 200, I receive the same error message.

I have tried using http://127.0.0.1:1337 and http://localhost:1337 and they both trigger that error message.

Hi chriso,
I like your lib, how can I use to for file upload ?

Like: curl -i -F name=test -F [email protected] http://example.org/upload
Thank you

Hi, chriso.
I'm gavin of korea. (So, you could feel odd in my English.)

I found some issue on rapid request situation.
When curlrequest.request called rapidly, callback's data argument get "" string often.
With no error object and no stdrerr stream.
Also i checked packets, and it's good. there's no problem.

And I solved this issue, I think.

I changed bind of "exit" event to "close". And "" string output disappeared.

//Handle curl exit
curl.on('exit', function (code) {
  try {
    err = code;

to

curl.on('close', function (code) {
  try {
    err = code;

According to this API doc,
"exit" event can be emitted before stdout stream flushed.

How you think about this modification?

I got caught in this, but debugging I see from line 210 in index.js:

key = key.replace(/[_-]/g, ' ').split(' ').map(function (str) {
                if (str.length) {
                    str = str[0].toUpperCase() + str.substr(1);
                }
                return str;
            }).join('-');
            headers[key] = options.headers[key];

So if the key in your passed in options.headers object doesn't match the one that's output after this transformation, then it will be undefined when running curl.

as in request parameters .

When I make a post request which contains a large body I get this error:

internal/child_process.js:319
    throw errnoException(err, 'spawn');
    ^

Error: spawn E2BIG
    at exports._errnoException (util.js:1050:11)
    at ChildProcess.spawn (internal/child_process.js:319:11)
    at Object.exports.spawn (child_process.js:378:9)
    at module.exports (/Users/Leon/Desktop/demo-repository/node_modules/curlrequest/spawn.js:19:30)
    at Object.exports.request (/Users/Leon/Desktop/demo-repository/node_modules/curlrequest/index.js:248:17)
    at Object.<anonymous> (/Users/Leon/Desktop/demo-repository/test.js:40:10)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)

When I try to send a smaller file (smaller body) it works. When I send the request using Postman I works as well even with the large file.

I have request that work in the curl in the command line but not work with your library on node js.

The request in curl command line(WORK):

curl -X POST "http://run.selfiecash.me:8080/predict" -d '{
"service":"getAge",
"parameters":{
"output":{
"best":5
}
},
"data":["https://scontent-frx5-1.cdninstagram.com/t51.2885-19/11856752_515409661959330_1395087449_a.jpg"]
}'

The request through your library(Not Work):

var options = {
url: 'http://run.selfiecash.me:8080/predict',
method: "post",
include:true,

            headers:{
                'Accept' : 'application/json',
                'Content-Length' : post_data.length,
                "Content-Type" : "application/json"
            },
            data:data,

        };

        curl.request(options, function (err2, parts) {
            parts = parts.split('\r\n');
            var data = parts.pop();

            console.log(err2,parts);
        });

Why i get error 400 ?

I was looking at #43 but I'm not sure if this is related or not. I was wondering if its possible to specify an SSL certificate and it's passphrase when making a curl connection to a WebDAV server.

If you look at this article:

http://www.qed42.com/blog/using-curl-webdav-two-factor-authentication

It shows exactly how to do this on the command line. I just tested this on OSX and it worked perfectly. You specify an SSL certificate and it's keyphrase/passphrase along with the typical username/password when making the connection.

Is the same thing is possible with this library? Can you pass the cert and passphrase as curl options or something?

Was having issues with centos 5.7 and curl when using this package. Was getting failed to intialize. Tracked it down the installed version of curl not supporting the --no-keepalive method. It works fine in debian 2.6.32-5

say i am using curlrequest to make a post call

the parts i get from the call back like this

[ 'HTTP/1.1 404 Not Found', 'Server: Apache-Coyote/1.1', 'Content-Type: text/html;charset=utf-8', 'Content-Length: 1005', 'Date: Fri, 20 Apr 2016 04:16:11 GMT', '', 'some content' ]

is there any way to get the status code 404 more easily?

I have experienced problems with POST request to websites with encoding set to ISO-8859-1.
All data is encoded with encodeURIComponent which, as I understand it, is encoding for UTF-8:

    //Parse POST data
    if (options.data && typeof options.data === 'object') {
        var data = [];
        for (var key in options.data) {
            data.push(encodeURIComponent(key) + '=' + encodeURIComponent(options.data[key]));
        }
        options.data = data.join('&');
    }

To encode for ISO-8859-1 I use escape() instead of encodeURIComponent()

I'm using request for a site without a valid SSL certificate, but I don't found options to curl accept this certificate.
In PHP I'm using this options and don't found in curlrequest:

curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($curl, CURLOPT_SSLVERSION, 3);

Hi,

as I see the http method is specified with the "--request" curl option - which, however, doesn't work with e.g. "head" requests. As the curl(1) manpage says about --request/-X

"This option only changes the actual word used in the HTTP request, it does not alter the way curl behaves. So for example if you want to make a proper HEAD request, using -X HEAD will not suffice. You need to use the -I, --head option."

Also:
"The method string you set with -X will be used for all requests, which if you for example use -L, --location may cause unintended side-effects when curl doesn't change request method according to the HTTP 30x response codes - and similar."

Hi,

Perhaps this is already possible, but then i apparently didn't find it..
Would it be possible to extend the API to provide the response code?
I'm thinking of a way like:

curl.request(options, function (err, data, code) { //.. });

Meta (the current third argument) would move to the fourth position.

There are some web services out there that use response codes to tell the user something. there seems to be some deviation in how services handle that.

For example, take the TVMaze API. You can request information about series on that API, but if you request something of a series it doesn't know it returns a 404. A very valid not found response. Then there are other services out there that to output a 200 OK code, but with a (json) output indicating that whatever was queried for was not found.

Cheers,
Mark

When you use curl on the command line, you can do the following to download a file and save it locally:

curl 'https://example/file.txt' > file.txt

or

curl 'https://example/image.jpg' > image.jpg

You don't have to worry about whether the file is ASCII or Binary. It just works.

So with curlrequest, the data you get in your callback in a file buffer. You don't know if it's binary or ascii and have to figure that out in order to write it locally. Is it possible to specify an outfile so that curl will just write the file without having to worry about it like on the command line?

I have very simple app in sailsjs which uses curl request for simple get request but fails with following error on heroku, is this heroku or some issue with this

�[36m2015-09-16T21:07:02.031430+00:00 app[web.1]:�[0m Check for new additions
�[36m2015-09-16T21:07:21.671439+00:00 app[web.1]:�[0m Total committees found are 30
�[36m2015-09-16T21:07:21.671445+00:00 app[web.1]:�[0m Processing Them
�[36m2015-09-16T21:07:21.671446+00:00 app[web.1]:�[0m
�[36m2015-09-16T21:07:21.872619+00:00 app[web.1]:�[0m /app/node_modules/curlrequest/index.js:260
�[36m2015-09-16T21:07:21.872624+00:00 app[web.1]:�[0m curl.stdout.on('data', function (data) {
�[36m2015-09-16T21:07:21.872626+00:00 app[web.1]:�[0m ^
�[36m2015-09-16T21:07:21.872627+00:00 app[web.1]:�[0m TypeError: Cannot read property 'on' of undefined
�[36m2015-09-16T21:07:21.872628+00:00 app[web.1]:�[0m at /app/node_modules/curlrequest/index.js:260:20
�[36m2015-09-16T21:07:21.872630+00:00 app[web.1]:�[0m at module.exports (/app/node_modules/curlrequest/spawn.js:19:9)
�[36m2015-09-16T21:07:21.872631+00:00 app[web.1]:�[0m at Object.exports.request (/app/node_modules/curlrequest/index.js:257:17)
�[36m2015-09-16T21:07:21.872632+00:00 app[web.1]:�[0m at /app/crontab/committees.js:27:22
�[36m2015-09-16T21:07:21.872634+00:00 app[web.1]:�[0m at wrapper (/app/node_modules/sails/node_modules/waterline/node_modules/lodash/index.js:3602:19)
�[36m2015-09-16T21:07:21.872635+00:00 app[web.1]:�[0m at applyInOriginalCtx (/app/node_modules/sails/node_modules/waterline/lib/waterline/utils/normalize.js:421:80)
�[36m2015-09-16T21:07:21.872637+00:00 app[web.1]:�[0m at wrappedCallback (/app/node_modules/sails/node_modules/waterline/lib/waterline/utils/normalize.js:320:18)
�[36m2015-09-16T21:07:21.872638+00:00 app[web.1]:�[0m at callback.success (/app/node_modules/sails/node_modules/waterline/node_modules/switchback/lib/normalize.js:33:31)
�[36m2015-09-16T21:07:21.872639+00:00 app[web.1]:�[0m at _switch (/app/node_modules/sails/node_modules/waterline/node_modules/switchback/lib/factory.js:48:28)
�[36m2015-09-16T21:07:21.872640+00:00 app[web.1]:�[0m at returnResults (/app/node_modules/sails/node_modules/waterline/lib/waterline/query/finders/basic.js:392:9)
�[36m2015-09-16T21:07:21.872641+00:00 app[web.1]:�[0m at /app/node_modules/sails/node_modules/waterline/lib/waterline/query/finders/basic.js:272:16
�[36m2015-09-16T21:07:21.872642+00:00 app[web.1]:�[0m at /app/node_modules/sails/node_modules/waterline/lib/waterline/query/finders/operations.js:82:7
�[36m2015-09-16T21:07:21.872644+00:00 app[web.1]:�[0m at /app/node_modules/sails/node_modules/waterline/node_modules/async/lib/async.js:49:16
�[36m2015-09-16T21:07:21.872646+00:00 app[web.1]:�[0m at Object.async.forEachOf.async.eachOf (/app/node_modules/sails/node_modules/waterline/node_modules/async/lib/async.js:227:20)
�[36m2015-09-16T21:07:21.872647+00:00 app[web.1]:�[0m at Object.async.forEach.async.each (/app/node_modules/sails/node_modules/waterline/node_modules/async/lib/async.js:206:22)
�[36m2015-09-16T21:07:21.872648+00:00 app[web.1]:�[0m at /app/node_modules/sails/node_modules/waterline/lib/waterline/query/finders/operations.js:433:11
�[36m2015-09-16T21:07:21.894708+00:00 app[web.1]:�[0m Error waiting for process to terminate: No child processes

How do I make a request like this
curl -u 'USER' https://api.github.com/user/repos -d "{"name":"REPO"}"

Hi when making multiple requests the headers get appended so the headers being sent resemble:

--header Accept: / --header Accept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.3 --header Accept-Language: en-US,en;q=0.8 --header Accept: */ --header Accept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.3 --header Accept-Language: en-US,en;q=0.8 --header Accept: */ --header Accept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.3 --header Accept-Language: en-US,en;q=0.8 --header Accept: */ --header Accept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.3 --header Accept-Language: en-US,en;q=0.8 --header Accept: */ --header Accept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.3 --header Accept-Language: en-US,en;q=0.8 --header Accept: */ --header Accept-Charset: ISO-8859-1,utf-8;q=0.7,;q=0.3 --header Accept-Language: en-US,en;q=0.8 --header Accept: */ --header Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 --header Accept-Language: en-US,en;q=0.8 --user-agent Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.809.0 Safari/535.1

Should these not be replaced?

Cheers

Richard

I am attempting to connect to an external server over HTTPS with basic authentication to download an image in binary format. When setting the enconding to null, I get the error "Failed to initialize". Full code below:

exports.slideImageData = function (req, res){
var id = req.params.id;
var prefix = "data:image/png;base64, ";
var options = {
url: config.jive.domain + 'api/core/v3/people/' + id + '/images/1/data'
, user: config.jive.user + ':' + config.jive.pass
, encoding: null
};
curl.request(options, function (err, result){
if (err){
console.log('error: ' + err)
} else {
var base64Image = new Buffer(result, 'binary').toString('base64');
base64Image = prefix + base64Image;
res.send('');
}
});
}

Also, this is intermittent, sometimes it works, usually does not:

`
'use strict';

const curl = require('curlrequest')
;

const feed = 'http://theartofcharmpodcast.theartofcharm.libsynpro.com/rss'; # Around 2.5MB

let options = {
url: feed,
verbose: true,
stderr: true,
retries: 3,
timeout: true,
headers: {
'Accept-Encoding': 'gzip'
}
};
curl.request(options, (err, data) => {
if (err) {
console.error (: C ${feed} [ERROR] ${err});
} else {
if ( ! data) {
console.log (: B ${feed} [NO DATA]);
} else {
console.log (: B ${feed} ${data.length});
}
}
});

`

Currently, it only works for me if I capitalize the C in curl. It's a bit confusing on my end, because the examples all use lowercase, and no one else has reported this from what I can see. Might I have something on my end that's causing this?

Works: Curl.request(options, function(error, result){});
Error: curl.request(options, function(error, result){});

The exact error I'm getting is as follows:
Exception while invoking method 'helloMethod' ReferenceError: curl is not defined

I couldn't find anything in the documentation.

Do you have a way of handling this?

Many Thanks!

Josh

Hi, Is there any format/syntax to upload a file to a location?

Reason:

I would be able to download a file from a location, is there anyway to upload a file to specific location, then what would be the options or any other params that I would need to pass/set?

Thanks in advance.

no issue, pls close

How would someone set http version to 1.0?

https://curl.haxx.se/docs/manpage.html

-0, --http1.0

(HTTP) Tells curl to use HTTP version 1.0 instead of using its internally preferred: HTTP 1.1.

--http1.1

(HTTP) Tells curl to use HTTP version 1.1. This is the internal default version. (Added in 7.33.0)

--http2

(HTTP) Tells curl to issue its requests using HTTP 2. This requires that the underlying libcurl was built to support it. (Added in 7.33.0)

--http2-prior-knowledge

(HTTP) Tells curl to issue its non-TLS HTTP requests using HTTP/2 without HTTP/1.1 Upgrade. It requires prior knowledge that the server supports HTTP/2 straight away. HTTPS requests will still do HTTP/2 the standard way with negotiated protocol version in the TLS handshake.

HTTP/2 support in general also requires that the underlying libcurl was built to support it. (Added in 7.49.0)

I'm using curlrequest to fetch some remote files from a web server and save them locally (using curl's "output" option). If the response is a 404, it saves the 404 HTML page as the local file.

How can I do things like check the response before taking action? I'm seeing the process option but not sure how to use it. I tried this:

let options = {
  ...
  process: res => {
    console.log(res)
  }
}

but that didn't seem to do anything. It says you pass a function. Not sure.

Is it possible to do what I want?