mojolingo / es-reindex Goto Github PK

This project forked from geronime/es-reindex

Ruby script to copy ElasticSearch index (reindex).

License: MIT License

Ruby 100.00%

es-reindex's Issues

Option to skip document count

I use this tool to merge several indices into a new index, which will ofc mean that the document count will always be NOT EQUAL, which is intended.

This process is a bit slow for me, so I would like the option to skip it.

Problem when I try to copy an index with -nm

I have this problem (ES 2.2):

/var/lib/gems/2.0.0/gems/es-reindex-0.3.3/bin$ ./es-reindex -nm http://127.0.0.1/Old_index http://127.0.0.1/New_Index
I, [2016-03-04T12:29:12.094427 #27207]  INFO -- : Copying 'http://127.0.0.1/Old_index' to 'http://127.0.0.1/New_Index'.
Confirm or hit Ctrl-c to abort...

/var/lib/gems/2.0.0/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:146:in `__raise_transport_error': [400]  (Elasticsearch::Transport::Transport::Errors::BadRequest)
        from /var/lib/gems/2.0.0/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:256:in `perform_request'
        from /var/lib/gems/2.0.0/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
        from /var/lib/gems/2.0.0/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/client.rb:125:in `perform_request'
        from /var/lib/gems/2.0.0/gems/elasticsearch-api-1.0.15/lib/elasticsearch/api/namespace/common.rb:21:in `perform_request'
        from /var/lib/gems/2.0.0/gems/elasticsearch-api-1.0.15/lib/elasticsearch/api/actions/indices/exists.rb:46:in `block in exists'
        from /var/lib/gems/2.0.0/gems/elasticsearch-api-1.0.15/lib/elasticsearch/api/utils.rb:191:in `__rescue_from_not_found'
        from /var/lib/gems/2.0.0/gems/elasticsearch-api-1.0.15/lib/elasticsearch/api/actions/indices/exists.rb:45:in `exists'
        from /var/lib/gems/2.0.0/gems/es-reindex-0.3.3/lib/es-reindex.rb:105:in `create_destination'
        from /var/lib/gems/2.0.0/gems/es-reindex-0.3.3/lib/es-reindex.rb:80:in `copy!'
        from /var/lib/gems/2.0.0/gems/es-reindex-0.3.3/lib/es-reindex.rb:15:in `block in copy!'
        from /var/lib/gems/2.0.0/gems/es-reindex-0.3.3/lib/es-reindex.rb:13:in `tap'
        from /var/lib/gems/2.0.0/gems/es-reindex-0.3.3/lib/es-reindex.rb:13:in `copy!'
        from ./es-reindex:30:in `<main>'

and this is that appears in /var/log/elasticsearch/node.log


[2016-03-03 19:03:02,399][INFO ][rest.suppressed          ] /New_Index/mappings/_mapping Params: {index=New_Index, type=mappings}
RemoteTransportException[[elastic-1][172.16.87.59:9300][indices:admin/mapping/put]]; nested: MapperParsingException[Root mapping definition has unsupported parameters:  [checkpoint : {properties={@timestamp={type=date, format=strict_date_optional_time||epoch_millis}, @version={type=string}, CookieI={type=string}, CookieR={type=string}, DCE-RPC Interface UUID={type=string}, DCE-RPC Interface UUID-1={type=string}, DCE-RPC Interface UUID-2={type=string}, DCE-RPC Interface UUID-3={type=string}, ICMP={type=string}, ICMP Code={type=string}, ICMP Type={type=string}, IKE IDs:={type=string}, IKE:={type=string}, Log ID={type=string}, NAT_addtnl_rulenum={type=string}, NAT_rulenum={type=string}, Severity={type=string}, User={type=string}, __nsons={type=string}, __p_dport={type=string}, __policy_id_tag={type=string}, __pos={type=string}, action={type=string}, auth_encryption_methods={type=string}, auth_method={type=string}, auth_method3={type=string}, bytes={type=string}, client={type=string}, client_inbound_bytes={type=string}, client_inbound_interface={type=string}, client_inbound_packets={type=string}, client_name={type=string}, client_outbound_bytes={type=string}, client_outbound_interface={type=string}, client_outbound_packets={type=string}, comment={type=string}, community={type=string}, connection_uid={type=string}, cvpn_category={type=string}, description={type=string}, device_identification={type=string}, dst={type=string}, dstkeyid={type=string}, duration={type=string}, elapsed={type=string}, encryption fail reason:={type=string}, encryption failure:={type=string}, event_type={type=string}, failure_impact={type=string}, first_hit_time={type=string}, fw_subproduct={type=string}, has_accounting={type=string}, hit={type=string}, host={type=string}, i_f_dir={type=string}, i_f_name={type=string}, inzone={type=string}, last_hit_time={type=string}, log_id={type=string}, log_sys_message={type=string}, login_timestamp={type=string}, message={type=string}, message_info={type=string}, methods:={type=string}, msgid={type=string}, office_mode_ip={type=string}, orig={type=string}, origin_sic_name={type=string}, outzone={type=string}, packets={type=string}, path={type=string}, peer gateway={type=string}, policy={type=string}, product={type=string}, proto={type=string}, reason={type=string}, reject_category={type=string}, rpc_prog={type=string}, rule={type=string}, rule_name={type=string}, rule_uid={type=string}, s_port={type=string}, scheme:={type=string}, segment_time={type=string}, server_inbound_bytes={type=string}, server_inbound_interface={type=string}, server_inbound_packets={type=string}, server_outbound_bytes={type=string}, server_outbound_interface={type=string}, server_outbound_packets={type=string}, service={type=string}, service_id={type=string}, session_timeout={type=string}, special_properties={type=string}, src={type=string}, srckeyid={type=string}, start_time={type=string}, status={type=string}, svc={type=string}, tags={type=string}, time={type=string}, tunnel_protocol={type=string}, update_service={type=string}, user={type=string}, user_dn={type=string}, user_group={type=string}, uuid={type=string}, version={type=string}, vpn_feature_name={type=string}, vpn_user={type=string}, xlatedport={type=string}, xlatedst={type=string}, xlatesport={type=string}, xlatesrc={type=string}}}]];
Caused by: MapperParsingException[Root mapping definition has unsupported parameters:  [checkpoint : {properties={@timestamp={type=date, format=strict_date_optional_time||epoch_millis}, @version={type=string}, CookieI={type=string}, CookieR={type=string}, DCE-RPC Interface UUID={type=string}, DCE-RPC Interface UUID-1={type=string}, DCE-RPC Interface UUID-2={type=string}, DCE-RPC Interface UUID-3={type=string}, ICMP={type=string}, ICMP Code={type=string}, ICMP Type={type=string}, IKE IDs:={type=string}, IKE:={type=string}, Log ID={type=string}, NAT_addtnl_rulenum={type=string}, NAT_rulenum={type=string}, Severity={type=string}, User={type=string}, __nsons={type=string}, __p_dport={type=string}, __policy_id_tag={type=string}, __pos={type=string}, action={type=string}, auth_encryption_methods={type=string}, auth_method={type=string}, auth_method3={type=string}, bytes={type=string}, client={type=string}, client_inbound_bytes={type=string}, client_inbound_interface={type=string}, client_inbound_packets={type=string}, client_name={type=string}, client_outbound_bytes={type=string}, client_outbound_interface={type=string}, client_outbound_packets={type=string}, comment={type=string}, community={type=string}, connection_uid={type=string}, cvpn_category={type=string}, description={type=string}, device_identification={type=string}, dst={type=string}, dstkeyid={type=string}, duration={type=string}, elapsed={type=string}, encryption fail reason:={type=string}, encryption failure:={type=string}, event_type={type=string}, failure_impact={type=string}, first_hit_time={type=string}, fw_subproduct={type=string}, has_accounting={type=string}, hit={type=string}, host={type=string}, i_f_dir={type=string}, i_f_name={type=string}, inzone={type=string}, last_hit_time={type=string}, log_id={type=string}, log_sys_message={type=string}, login_timestamp={type=string}, message={type=string}, message_info={type=string}, methods:={type=string}, msgid={type=string}, office_mode_ip={type=string}, orig={type=string}, origin_sic_name={type=string}, outzone={type=string}, packets={type=string}, path={type=string}, peer gateway={type=string}, policy={type=string}, product={type=string}, proto={type=string}, reason={type=string}, reject_category={type=string}, rpc_prog={type=string}, rule={type=string}, rule_name={type=string}, rule_uid={type=string}, s_port={type=string}, scheme:={type=string}, segment_time={type=string}, server_inbound_bytes={type=string}, server_inbound_interface={type=string}, server_inbound_packets={type=string}, server_outbound_bytes={type=string}, server_outbound_interface={type=string}, server_outbound_packets={type=string}, service={type=string}, service_id={type=string}, session_timeout={type=string}, special_properties={type=string}, src={type=string}, srckeyid={type=string}, start_time={type=string}, status={type=string}, svc={type=string}, tags={type=string}, time={type=string}, tunnel_protocol={type=string}, update_service={type=string}, user={type=string}, user_dn={type=string}, user_group={type=string}, uuid={type=string}, version={type=string}, vpn_feature_name={type=string}, vpn_user={type=string}, xlatedport={type=string}, xlatedst={type=string}, xlatesport={type=string}, xlatesrc={type=string}}}]]
        at org.elasticsearch.index.mapper.DocumentMapperParser.checkNoRemainingFields(DocumentMapperParser.java:171)
        at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:159)
        at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:100)
        at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:435)
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:257)
        at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:458)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:762)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

check_docs has no guarantee of providing useful information if either src or dest are added to during reindex.

Can't make it run

Probably is a ultra simple problem, but I'm not a ruby expert, sorry.
I have ruby 2.0, with gem elasticsearch installed. When I go to es-reindex/bin, I executed the binary and this is what happen:

$ ./es-reindex
/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:53:in `require': cannot load such file -- es-reindex (LoadError)
        from /usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:53:in `require'
        from ./es-reindex:3:in `<main>'

What is missing? Should I move the folder somewhere else?

Changing the mapping should trigger a reindex

Created Snapshot of connect_messages-development_20150201214010 in 10.203.175.32:9200:/tmp/es_snapshots
ConnectMessage existing mappings = [[:body, {:type=>"string"}], [:created_at, {:type=>"date"}], [:from, {:type=>"string"}], [:stanza, {:type=>"string"}], [:to, {:type=>"string"}], [:type, {:type=>"string"}], [:updated_at, {:type=>"date"}], [:visibility, {:type=>"string"}]]
ConnectMessage new mappings      = [[:body, {:type=>"string"}], [:created_at, {:type=>"date"}], [:from, {:type=>"string"}], [:stanza, {:type=>"string"}], [:to, {:type=>"string"}], [:type, {:type=>"string"}], [:updated_at, {:type=>"date"}], [:visibility, {:type=>"string"}]]
Skipping action due to guard callbacks

Model definition changed from this:

attribute :type,   String, mapping: { type: 'string', index: 'not_analyzed' }

to this:

attribute :type,   String, mapping: { type: 'string', index: 'simple' }

Need documentation on Rails integration

The README needs a section on how to use this with Rails, especially if we can find a convenient way to trigger re-index on model changes.

Before and After each callback isn't passing in doc

When you try to use the callbacks for before_each and after_each it will throw a ArgumentError: wrong number of arguments (0 for 1) because the lambda function is expecting at least 1 argument but isn't receiving it.

Unable to use this gem

It depends on activesupport (because it tries to use #present? on its options, which is not present in normal ruby, at least not in my setup).

Defining this method on Object makes it work until it fails in next place:

NoMethodError: undefined method `[]' for nil:NilClass
    from <HOME>/.gems-2.1.5/gems/es-reindex-0.3.0/lib/es-reindex.rb:134:in `get_settings'
    from <HOME>/.gems-2.1.5/gems/es-reindex-0.3.0/lib/es-reindex.rb:107:in `create_destination'
    from <HOME>/.gems-2.1.5/gems/es-reindex-0.3.0/lib/es-reindex.rb:80:in `copy!'
    from <HOME>/.gems-2.1.5/gems/es-reindex-0.3.0/lib/es-reindex.rb:15:in `block in copy!'
    from <HOME>/.gems-2.1.5/gems/es-reindex-0.3.0/lib/es-reindex.rb:13:in `tap'
    from <HOME>/.gems-2.1.5/gems/es-reindex-0.3.0/lib/es-reindex.rb:13:in `copy!'
    from (irb):19
    from /opt/rubies/ruby-2.1.5/bin/irb:11:in `<main>'

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.