Comments (15)
Hey Luca, 9 hours is a really long time. What does your search_data
method look like? Are you doing any N + 1 queries?
from searchkick.
I index only 3 string fields, nothing heavy. Also I noticed that indexing is slow in general, even with another model, it can just do about 1000 records every 5-7 seconds.
Using find_in_batches in the console is much much faster than that.
from searchkick.
Is Elasticsearch installed locally? (want to rule of the possibility to network latency)
from searchkick.
Yeah, with the default configuration
from searchkick.
How long are the string fields? Really long fields would increase indexing time.
from searchkick.
Also, does indexing start out slow, or does it become slow after a while? (possible memory issue)
from searchkick.
The fields are 80 chars max and the indexing starts on about 1000 every 5 secs, slowing down to 1000 every 7-8 secs.
The bigger problem is the used memory, grows constantly to over 6gb. By using find_in_batches this shouldn't happen.
I will try to replicate this behavior with a small app and put it on github.
from searchkick.
I made a new app with the bare minimum to test searchkick, with the same model and fields of my main app...and creating and indexing 100.000 entries it's a lot faster.
Maybe there's some other gem in my app that mess things up. I will do some tests on them and report backβ¦argh π
from searchkick.
One thing I've noticed is tire calls to_hash
when indexing documents - regardless of search_data
. Might help with the investigation...
from searchkick.
Yeah I noticed it too. I profiled the importing, take a look at the graph and the stack, seems like Carrierwave it's the culprit too (i have two uploaders in the model)
from searchkick.
I overloaded to_hash to return only the data I needed and the importing flies! That was the problem, the default one puts every field in the hash π
from searchkick.
I tried to reindex with that change and it takes only 1 hour now, but the RAM usage is still high: 2,4GB.
from searchkick.
Ok, I can close this. It's a gem (i need to find which) that i have in the development group in Gemfile that was causing the RAM issue. Tried to comment all the group and now it's going smooth, 250.000 entries and only 148mb of RAM π
For others having the same problem, watch out! :)
Thanks for your help @ankane!
from searchkick.
Let me know what you find. I'll see what I can do about to_hash
.
from searchkick.
The problem is the Bullet gem, I think it keeps some kind of cache of the objects, but I haven't checked the sources.
Without it and by redefining to_hash, the indexing is fast. Done in 45 minutes using only 150MB of memory :)
from searchkick.
Related Issues (20)
- Add documentation for client_options HOT 1
- Lazy load not working HOT 2
- named query search example (_name) HOT 1
- Feature Request: Implement none Method HOT 3
- Scoped include in search_import not being loaded HOT 1
- Unable to index model with composite pk HOT 1
- `scroll_each` API to work with batches like `find_each`
- While using opensearch-ruby got error: end of file reached (Faraday::ConnectionFailed) HOT 1
- Default index name doesn't use UTC time HOT 1
- minor issue when when doing a full reindex in queue mode HOT 1
- Make "Searchkick.search" for multi-model searches work with Rails low-level caching HOT 4
- Why is `_script` filters commented out in the source code? Possible to add support for scripts? HOT 2
- Index different objects to different indices HOT 2
- Integrate partial reindexing with mode_async bulk reindexing
- `scope_results` outputs incorrect warnings HOT 1
- rails production env indexmissing issue HOT 1
- Ability to set body_options globally in the same way you can model_options
- Clarify use of ActiveRecord `where` in `search_import` HOT 1
- Unable to load dynamic synonyms with Elastic HOT 1
- Support for Dynamoid (DynamoDB) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from searchkick.