Light

geronime / es-reindex Goto Github PK

View Code? Open in Web Editor NEW

63.0 4.0 43.0 23 KB

Ruby script to copy ElasticSearch index (reindex).

License: MIT License

Ruby 100.00%

es-reindex's Introduction

es-reindex - simple ruby script for copying ElasticSearch index

Simple ruby script to copy and reindex ElasticSearch index, copying index settings and mapping(s).

Progress and time estimation is displayed during the scrolling process.

Requirements

Ruby 1.8.6 or newer is required, use rvm for convenience.

Following gems are required via Gemfile:

[rest-client] (https://github.com/archiloque/rest-client)
[oj] (https://github.com/ohler55/oj)

You can install the requirements locally via bundler:

$ bundle install --path=.bundle

Usage

Refer to script's help:

$ ./es-reindex.rb -h

Script to copy particular ES index including its (re)creation w/options set
and mapping copied.

Usage:

  ./es-reindex.rb [-r] [-f <frame>] [source_url/]<index> [destination_url/]<index>

    - -r - remove the index in the new location first
    - -f - specify frame size to be obtained with one fetch during scrolling
    - -u - update existing documents (default: only create non-existing)
    - optional source/destination urls default to http://127.0.0.1:9200

Changelog

0.0.8: Optimization in string concat (@nara)
0.0.7: Document header arguments _timestamp and _ttl are copied as well
0.0.6: Document headers in bulks are now assembled and properly JSON dumped
0.0.5: Merge fix for trailing slash in urls (@ichinco), formatting cleanup
0.0.4: Force create only, update is optional (@pgaertig)
0.0.3: Yajl -> Oj
0.0.2: repated document count comparison
0.0.1: first revision

License

es-reindex is copyright (c)2012 Jiri Nemecek, and released under the terms of the MIT license. See the LICENSE file for the gory details.

es-reindex's People

Contributors

Stargazers

Watchers

Forkers

pgaertig ichinco daifu ebichuman nara mkrogemann mojolingo otiai10 scottgeary francescopace adler vend summera magec roweryan ihpu alienxp03 alongat nono-gdv octopuce

es-reindex's Issues

Unexpected Character

es-reindex.rb:141:inload': unexpected character at line 1, column 276468 load.c:420 from es-reindex.rb:141:in <main>'

Getting the above error when trying to reindex.

Retrying POST ERROR: RestClient::BadRequest - 400 Bad Request

Hi,

I'm trying the script for copying an index but it always breaks down after reaching 1.000 documents.

% ruby es-reindex.rb -f 50 foo foo-backup
Copying 'http://127.0.0.1:9200/foo' to 'http://127.0.0.1:9200/foo-backup'...
  Confirm or hit Ctrl-c to abort...

Copying 'http://127.0.0.1:9200/foo' to 'http://127.0.0.1:9200/foo-backup'... 
    1000/63836 (1.6%) done in 0:00:00, E.T.A.: 2014-01-29 17:33:02 +0100.
Retrying POST ERROR: RestClient::BadRequest - 400 Bad Request

Retrying POST ERROR: RestClient::BadRequest - 400 Bad Request

Retrying POST ERROR: RestClient::BadRequest - 400 Bad Request

I have tried different frame sizes and defining no frame size at all but it always breaks down when it reaches 1.000 docs.

System:
ruby 2.0.0-p353
elasticsearch 0.90.10

Regards,

No handler found for uri [/NEWINDEXNAME] and method [POST] with elasticsearch 5.2

I keep getting the same error when using es-reindex:

No handler found for uri [/NEWINDEXNAME] and method [POST]

Retrying POST ERROR: RestClient::BadRequest - 400 Bad Request

The problem is caused by:

POST should now be PUT
Target index must be lowercase
index.uuid should be removed from the new index
index.version should be removed from the new index
index.creation_date should be removed from the new index

missing _timestamp

_timestamp field, when present, is not copied over

Gemified

If anyone is interested, we've taken this excellent script, and expanded it into a full gem with a few more capabilities :

Project: https://github.com/mojolingo/es-reindex /
Gem: https://rubygems.org/gems/es-reindex

Thanks for the script geronime!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.