Elasticsearch - pushResults and pushErrors doesn't push the data to downstream about node-etl HOT 4 CLOSED

Thanks @girishgoudapatil. What elasticsearch client version / server version are you using? Are you using etl.collect above to assemble bulks above the etl.elastic.index? Can you verify that the data is actually uploaded to ElasticSearch?

Here is the corresponding test:
https://github.com/ZJONSSON/node-etl/blob/master/test/elastic-test.js#L31-L43

 it('pipes data into elasticsearch',function() {
    var i = 0;
    var upsert = etl.elastic.index(client,'test','test',{pushResult:true});

    return data.stream()
      .pipe(etl.map(function(d) {
        d._id = i++;
        return d;
      }))
      .pipe(etl.collect(100))
      .pipe(upsert)
      .promise()
      .then(function(d) {
        assert.equal(d[0].items.length,3);
        assert.deepEqual(d[0].items[0].body,data.data[0]);
      });
  });

https://circleci.com/gh/ZJONSSON/node-etl/138

  elastic bulk insert
    ✓ pipes data into elasticsearch (746ms)
    ✓ retrieves data back from elasticsearch (2036ms)
    ✓ streams results with elastic.find
    ✓ streams results with elastic.scroll (608ms)

from node-etl.

@ZJONSSON thanks for the quick response.

I am using [email protected] module , below code returns empty dataset for me in the resolve.

  .pipe(etl.collect(500))
  .pipe(etl.elastic.index(esClient,index,'test', {
    concurrency : 5,
    pushResult : true,
    pushErrors : true
  }))
  .promise()
  .then(function (data) {
    console.log(data);
    return upload(tmpFilename, s3key);
  })

actually pushing everything in one go under resolve would be memory intensive if you are processing millions of records

from node-etl.

Ah I spotted the culprit. pushErrors currently supersedes pushResult (see here) - meaning that if you elect pushErrors then pushResults is ignored (you receive nothing downstream unless records errored). I think this should probably be the other way around to eliminate confusion - will post a patch.

Your code should work if you simply remove pushErrors from the options in your original example. (pushResults pushes both successes and errors). Let me know if that works

from node-etl.

See #46 - please reopen if the issue persists
(published on npm as [email protected])

from node-etl.