Giter VIP home page Giter VIP logo

node-elasticsearch-client's Introduction

No longer maintained

build status Elastic Search Client

A node.js client for elasticsearch (http://www.elasticsearch.com).

Installation

npm install elasticsearchclient

Executing commands on elasticsearch

ElasticSearchClient = require('elasticsearchclient');

var serverOptions = {
    host: 'localhost',
    port: 9200,
    pathPrefix:'optional pathPrefix',
    secure: true||false,
    //Optional basic HTTP Auth
    auth: {
   	username: process.env.ES_USERNAME,
    	password: process.env.ES_PASSWORD
  }
};

OR specify multiple hosts

var serverOptions = {
    hosts:[
        {
            host: 'localhost',
            port: 9200
        }]
};
var elasticSearchClient = new ElasticSearchClient(serverOptions);

var qryObj = {
    field : term
}

elasticSearchClient.search('my_index_name', 'my_type_name', qryObj)
    .on('data', function(data) {
        console.log(JSON.parse(data))
    })
    .on('done', function(){
        //always returns 0 right now
    })
    .on('error', function(error){
        console.log(error)
    })
    .exec()

//Canonical search
 elasticSearchClient.search('my_index_name', 'my_type_name', qryObj, function(err, data){
    console.log(JSON.parse(data))
 })

//Search call as a reusable object with a canonical callback
mySearchCall = elasticSearchClient.search('my_index_name', 'my_type_name', qryObj);
//Do it once
mySearchCall.exec(function(err, data){
    console.log(JSON.parse(data))
})
//Do it twice
mySearchCall.exec(function(err, data){
    console.log(JSON.parse(data))
})



elasticSearchClient.index(indexName, typeName, document, id, options)
    .on('data', function(data) {
        console.log(data)
    })
    .exec()

//Bulk index example
var commands = []
commands.push({ "index" : { "_index" :'my_index_name', "_type" : "my_type_name"} })
commands.push({field1:'value',field2:'value'})


    elasticSearchClient.bulk(commands, {})
            .on('data', function(data) {})
            .on('done', function(done){})
            .on('error', function(error){})
            .exec();

Twitter stream: Seems to be a problem in the es with deleting streams. May require a node restart for modifications/deletes to take effect

elasticSearchClient.createOrModifyTwitterRiver(riverName, riverData)
    .on('data', function (data) {
        console.log(data)
        assert.ok(JSON.parse(data).ok)
    })
    .on('error',
    function (error) {
        console.log(error)
        assert.ok(false)
    }).exec()

Code coverage

Run code coverage by executing:

$npm run-script coverage

*Requires visionmedia/jscoverage

What can i do?

Most of the API (http://www.elasticsearch.org/guide/reference/api/) is implemented.

License

Copyright (c) 2011 Phillip Rosen

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

node-elasticsearch-client's People

Contributors

bratchenko avatar brianstarke avatar da-mkay avatar draco2003 avatar elliotf avatar ferhatsb avatar hmalphettes avatar jandre avatar jhpinson avatar juzerali avatar kersten avatar khushil avatar kyleamathews avatar mattallty avatar nanek avatar phillro avatar revington avatar sixcircuit avatar skyfetch avatar snmaynard avatar unikoid avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-elasticsearch-client's Issues

Bulk Error

I am getting:

{"error":"Failed to derive xcontent from org.elasticsearch.common.bytes.ChannelBufferBytesReference@5299e322"}

when attempting to submit a bulk index command array. I am using bons.ai ES. Any thoughts? I am not finding an specific references to the 'ChannelBufferBytesReference'

Thanks

Bulk API upload deletes previous index

I setup an empty index with 5 shards and 2 replicas and then ran the bulk upload method using node-elasticsearch-client, this resulted in the index being deleted and recreated with a 1shard/1replica configuration!

Is this a default behavior in node-elasticsearch-client code?
If so, then how can I get around it?

Please note that the timestamps in this snapshot are in reverse order:
screen shot 2013-08-08 at 4 49 38 pm

could do with more docs?

Have to dig into your code to find the calls eg for 'deleteDocument'..

Also - does this client lib support ES providing version number on indexing for optimistic locking?

Manipulation of arguments array fails

I pulled this out of core.js (and added console.log)

//Pull the callback and set it false to not clobber id.
if(typeof arguments[arguments.length-1]=='function'){
    console.log('yes');
    cb=arguments[arguments.length-1];
    console.log(cb);
    arguments[arguments.length-1]=false;
    console.log(cb);
}

When running via coffee-script, the callback stopped working. And I found this:

yes
[Function]
false

Notice how the cb variable changes value when the arguments array is changed. Why I have no idea – but I would personally avoid manipulating arguments.

For kicks I tested this:

var args = Array.prototype.slice.call(arguments, 0);

//Pull the callback and set it false to not clobber id.
if(typeof args[args.length-1]=='function'){
    console.log('yes');
    cb=args[args.length-1];
    console.log(cb);
    args[args.length-1]=false;
    console.log(cb);
}

And it produced the expected

yes
[Function]
[Function]

index & type support for bulk

"The endpoints are /_bulk, /{index}/_bulk, and {index}/type/_bulk. When the index or the index/type are provided, they will be used by default on bulk items that don’t provide them explicitly."

Looks like the driver only supports the /_bulk endpoint.

JSON query builder - any interest?

Hey there. First, thanks for this module, it works pretty well.

I'm looking to gauge the interest and get some feedback on a JSON query builder I've been working on. I've been using it to compensate for my apparent inability to write ES queries manually.

Basically, you can do things like this:

var qb = require('query-builder');
var fields = ['field1', 'field2', 'field3'];
var query = qb.or(qb.termArray(fields, 'my-value')).done(0, 10);

which will produce the following JS object

{
  "query": {
    "constant_score": {
      "filter": {
        "or": [
          {
            "term": {
              "field1": "my-value"
            }
          },
          {
            "term": {
              "field2": "my-value"
            }
          },
          {
            "term": {
              "field3": "my-value"
            }
          }
        ]
      }
    }
  },
  "from": 0,
  "size": 10
}

In general, it handles wrapping filters/queries as necessary, and will warn if trying to throw a query in a place that only takes a filter.

I was also thinking of making things more chainy, so it would look like

var query = qb.orStart().terms(fields, 'my-value').orEnd().done(0, 10);

but it seemed more complicated at the time.

Would other people use this? The chaining version or the way it is now? New module or part of this module?

Thanks!

NPM out of date

Last version in NPM is 0.4.0; would you mind updating when you get a chance?

.index shouldn't modify document

When calling .index, it removes the id property from the passed in document. That caught me off guard and created a subtle bug to track down.

It looks like you can save the id into a variable and add it back into the document before returning.

Chris

Add a utility method for reindexing data from one index to another

A utility method for reindexing data from one index to another would be very helpful if the mapping for an index has changed.

I know of some php and perl alternatives that do this:
http://euphonious-intuition.com/2012/08/reindexing-your-elasticsearch-data-with-scanscroll/
https://github.com/clintongormley/ElasticSearch.pm#reindex
But I find myself wondering if this administrative task can be added to node clients such as this one, so folks don't have to jump languages?

Why is hashlib present?

Hashlib compilation is failing on my system. I see that it is requireed in the tests but not used anywhere. I wonder why is it there for?

npm version

When you released 0.2 on npm, it looks like 0.1.8 disappeared. It would be great if you could keep older versions up to not force users to upgrade immediately.

Timeout

Is there a recommended timeout value. I believe I am getting a block on my event loop because cause my ES query is failing. I have solved the ES cluster issue, and found this option in the new version.

Are there best practices for this value?

Thanks

Using a cluster config with multiple hosts

Morning. I am trying to setup and test the multiple hosts options for an es client. I have found that the following does not handle a failed host. If the first host fails to respond the client does not fail over to the secondary. Is this expected?

var esClient = new esLib( {
  hosts:[
    { host: 'es1', port: 9300 },
    { host: 'es1', port: 9400 }
  ]
});

Is the discoverNodes() function implemented? Any help would be great. We are using this package in production.

count with object as query

Line 111 in core.js
return this.createCall({path : path, method : 'GET', data : JSON.stringify(query)}, this.clientOptions);

You must do a POST insteadof GET

thanks

Update npm

It looks like the version on npm hasn't been updated in a while. In particular, I'm interested in getting the changes to bulk from e969bca.

Thanks,

Chris

Example is wrong!

Hey,

I looked at your initial example, and I am not sure if you meant to do this, but your "id optional" is in the wrong paramater

elasticSearchClient.index('my_index_name', 'my_type_name', /* _id (optional) */'1111', {'name':'name'})
.on('data', function(data) {
console.log(data)
})
.exec()

it should be in the fourth paramater of the "index" function call

Please enhance readme w/ authN example

Although it only took 5 mins to find:

if (this.auth) {
    request.setHeader("Authorization", "Basic " + new Buffer(this.auth.username + ":" + this.auth.password).toString('base64'))
}

It would still be nice to have a sample of how to specify username / password in this client on the README for quicker reference :)

var serverOptions = {
  host: process.env.ES_HOST,
  port: process.env.ES_PORT,
  secure: process.env.ES_SECURE,
  auth: {
    username: process.env.ES_USERNAME,
    password: process.env.ES_PASSWORD
  }
};

Travis build failing

Travis build is failing because travis doesn't have ElasticSearch server in its infrastructure and tests are looking for one at localhost:9200. So the build is actually not broken, travis is just unable to run them. I am looking for a way to make it work.

Meanwhile no need to worry, the development code is untouched.

Content-Length header is absent

Module sends invalid request wich make it unusable if we place elastic behind ngnx.

As we run:

elasticSearchClient.createIndex("test_index")
    .on('data', function(data) {
        console.log("results -->");
        console.log(data);
    }).exec();

We get:

<html>
<head><title>411 Length Required</title></head>
<body bgcolor="white">
<center><h1>411 Length Required</h1></center>
<hr><center>nginx</center>
</body>
</html>

deleteByQuery function does not work

The variable 'path' inside the function is:
var path = '/' + indexName + '/' + typeName + '/_query';

I found it not working with the elastic search version 0.19.8

the function works fine if I modify the variable as
var path = '/' + indexName + '/' + typeName

Please correct the source code.

Multiget request is malformed

I ran a multiget request as shown in https://github.com/phillro/node-elasticsearch-client/blob/master/test/core.test.js#L108: elasticSearchClient.multiget(indexName, objName, [ "sushi" ], {})
against ES v0.20.4 instance and it failed with:

{
    "ok": false,
    "error": "docs array element should include an object"
}

This error message originates from https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/get/MultiGetRequest.java#L-243 so I think there is perhaps a fix required to the way a request is formed by the client.

In comparison, running a multiget request as shown in https://github.com/phillro/node-elasticsearch-client/blob/master/test/core.test.js#L123:
elasticSearchClient.multiget(indexName, objName, [{_id: "sushi"}], {})
works just fine.

bug in closeIndex

This is obviously wrong:

ElasticSearchClient.prototype.closeIndex = function(indexName, options, cb) {
   /* ... */
   var path = '/' + indexName + '/_open';
   /* ... */
}

ECONNRESET when trying to create an index

Perhaps this is the wrong place but I was looking for some help solving this problem while using this node library.

At this point I'm initializing my client with the default values elasticsearch normally uses so

options = {
  port: 9200,
  host: "localhost",
  pathPrefix: "test",
  secure: true
};

Now switching the secure flag to being a false value doesn't throw an error but does leave me with No handler found for uri [test/Test_Makes] and method [PUT] data. Frankly I have no idea what this means.

Basically my question(s) is/are, what does the secure flag here do when initializing my ES client? What does not having a handler mean there?

Again, sorry for polluting the issues with this question set. I tried scouring through the ElasticSearch API to make connections between this wrapper library and their API but alas came up empty handed.

Many Thanks.

MEMORY LEAK

I found a serious issue with the error handling in ElasticSearchClient.
I have narrowed it down on my end to the ERROR event

When elasticSearch returns an error, or is unsuccessful in the case of
a shard failure an EventEmitter memory leak occurs.

The error callBack variable is always returning as undefined for me, even in my working code now. But when i do nothing with it no memory leak occurs.

Here is our package.json depends

{
  "dependencies": {
    "express": "3.0.0",
    "jade": ">= 0.28.1",
    "mongoose": "3.5.7",
    "elasticsearchclient": "0.4.0",
    "3scale":"0.4.0",
    "newrelic":"0.9.20"
  },
  "engines": {
    "node": "0.8.21",
    "npm": "1.1.65"
  }
}

ES CLIENT FUNCTION CAUSING MEM LEAK

eSClient.search(index, type, qryObj)
.on('data', function(data) {
    var esData = JSON.parse(data);
    // check for missed errors
    if (esData.hits === undefined){
        console.warn(esData);
        Item.find({"item_name":new RegExp(phrase)},function(err, items){
            res.json(items);
        });
    }else{
        var results_set = new Results_set(esData,fields);
        res.json(results_set);
    }
})
.on('done', function(){
    //always returns 0 right now
})
.on('error', function(error){
//even when there is an error, the error event never gets called.
    console.warn(error);
})
.exec();

ES CLIENT FUNCTION WORKING WITHOUT MEM LEAK

eSClient.search(index, type, qryObj,function(error,data){
//error === undefined
  var esData = JSON.parse(data);
  // check for missed errors
  if (esData.hits === undefined){
    console.warn(esData);
  }else{
    var results_set = new Results_set(esData,fields);
    res.json(results_set);
  }
});

Canonical callbacks clobbered when passing options parameter

Do to the way parameters are normalized, passing an options parameter as well as a cb parameter results in the callback being set to false.

For example:

var doc = { data : 'helo' };

// This works
elastic.index('index', 'type', doc, 'id1', function() {...});

// This does not
elastic.index('index', 'type', doc, 'id1', { parent: 'blah' }, function() {...});

I might have time to do a PR in a couple days.

Document Question

I have run into confusion with the bulk es documentation on your readme.

var commands = []
commands.push({ "index" : { "_index" :'my_index_name', "_type" : "my_type_name"} })
commands.push({field1:'value',field2:'value'})

elasticSearchClient.bulk(commands, {})
        .on('data', function(data) {})
        .on('done', function(done){})
        .on('error', function(error){})
        .exec();

I assume, and I am sure other have as well that sending a large batch would use the following:
commands.push({ "index" : { "_index" :'my_index_name', "_type" : "my_type_name"} })
commands.push({field1:'value1',field2:'value'})
commands.push({field1:'value2',field2:'value'})
commands.push({field1:'value3',field2:'value'})
commands.push({field1:'value4',field2:'value'})

Through trial an error we found that we needed to include a 'index' statement before each document to bulk index.

commands.push({ "index" : { "_index" :'my_index_name', "_type" : "my_type_name"} })
commands.push({field1:'value1',field2:'value'})
commands.push({ "index" : { "_index" :'my_index_name', "_type" : "my_type_name"} })
commands.push({field1:'value2',field2:'value'})
commands.push({ "index" : { "_index" :'my_index_name', "_type" : "my_type_name"} })
commands.push({field1:'value3',field2:'value'})
commands.push({ "index" : { "_index" :'my_index_name', "_type" : "my_type_name"} })
commands.push({field1:'value4',field2:'value'})

Are we using the bulk correctly? Is this expected? Your repo, just need a few examples of implementations!

Thanks

Console.log left in

var request = client.request(reqOptions);
    console.log(request)
    request.on('error',function(error){
        self.emit("error", error)
    })

Line 38 of elasticSearchCall.js is doing some zealous logging. Can you update and push out a new version to npm?

Thanks!

Chris

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.