Slow importing about searchkick HOT 15 CLOSED

Hey Luca, 9 hours is a really long time. What does your search_data method look like? Are you doing any N + 1 queries?

from searchkick.

I index only 3 string fields, nothing heavy. Also I noticed that indexing is slow in general, even with another model, it can just do about 1000 records every 5-7 seconds.

Using find_in_batches in the console is much much faster than that.

from searchkick.

Is Elasticsearch installed locally? (want to rule of the possibility to network latency)

from searchkick.

Yeah, with the default configuration

from searchkick.

How long are the string fields? Really long fields would increase indexing time.

from searchkick.

Also, does indexing start out slow, or does it become slow after a while? (possible memory issue)

from searchkick.

The fields are 80 chars max and the indexing starts on about 1000 every 5 secs, slowing down to 1000 every 7-8 secs.

The bigger problem is the used memory, grows constantly to over 6gb. By using find_in_batches this shouldn't happen.

I will try to replicate this behavior with a small app and put it on github.

from searchkick.

I made a new app with the bare minimum to test searchkick, with the same model and fields of my main app...and creating and indexing 100.000 entries it's a lot faster.

Maybe there's some other gem in my app that mess things up. I will do some tests on them and report back…argh 😔

from searchkick.

One thing I've noticed is tire calls to_hash when indexing documents - regardless of search_data. Might help with the investigation...

from searchkick.

Yeah I noticed it too. I profiled the importing, take a look at the graph and the stack, seems like Carrierwave it's the culprit too (i have two uploaders in the model)

from searchkick.

I overloaded to_hash to return only the data I needed and the importing flies! That was the problem, the default one puts every field in the hash 😓

from searchkick.

I tried to reindex with that change and it takes only 1 hour now, but the RAM usage is still high: 2,4GB.

from searchkick.

Ok, I can close this. It's a gem (i need to find which) that i have in the development group in Gemfile that was causing the RAM issue. Tried to comment all the group and now it's going smooth, 250.000 entries and only 148mb of RAM 😄

For others having the same problem, watch out! :)

Thanks for your help @ankane!

from searchkick.

Let me know what you find. I'll see what I can do about to_hash.

from searchkick.

The problem is the Bullet gem, I think it keeps some kind of cache of the objects, but I haven't checked the sources.

Without it and by redefining to_hash, the indexing is fast. Done in 45 minutes using only 150MB of memory :)

from searchkick.