🚀 Intelligent search made easy
Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.
Searchkick handles:
- stemming -
tomatoesmatchestomato - special characters -
jalapenomatchesjalapeño - extra whitespace -
dishwashermatchesdish washer - misspellings -
zuchinimatcheszucchini - custom synonyms -
popmatchessoda
Plus:
- query like SQL - no need to learn a new query language
- reindex without downtime
- easily personalize results for each user
- autocomplete
- “Did you mean” suggestions
- supports many languages
- works with Active Record and Mongoid
Check out Searchjoy for analytics and Autosuggest for query suggestions
🍊 Battle-tested at Instacart
- Getting Started
- Querying
- Indexing
- Intelligent Search
- Instant Search / Autocomplete
- Aggregations
- Testing
- Deployment
- Performance
- Advanced Search
- Reference
- Contributing
Install Elasticsearch or OpenSearch. For Homebrew, use:
brew install elastic/tap/elasticsearch-full
brew services start elasticsearch-full
# or
brew install opensearch
brew services start opensearchAdd these lines to your application’s Gemfile:
gem "searchkick"
gem "elasticsearch" # select one
gem "opensearch-ruby" # select oneThe latest version works with Elasticsearch 7 and 8 and OpenSearch 1 and 2. For Elasticsearch 6, use version 4.6.3 and this readme.
Add searchkick to models you want to search.
class Product < ApplicationRecord
searchkick
endAdd data to the search index.
Product.reindexAnd to query, use:
products = Product.search("apples")
products.each do |product|
puts product.name
endSearchkick supports the complete Elasticsearch Search API and OpenSearch Search API. As your search becomes more advanced, we recommend you use the search server DSL for maximum flexibility.
Query like SQL
Product.search("apples", where: {in_stock: true}, limit: 10, offset: 50)Search specific fields
fields: [:name, :brand]Where
where: {
expires_at: {gt: Time.now}, # lt, gte, lte also available
orders_count: 1..10, # equivalent to {gte: 1, lte: 10}
aisle_id: [25, 30], # in
store_id: {not: 2}, # not
aisle_id: {not: [25, 30]}, # not in
user_ids: {all: [1, 3]}, # all elements in array
category: {like: "%frozen%"}, # like
category: {ilike: "%frozen%"}, # ilike
category: /frozen .+/, # regexp
category: {prefix: "frozen"}, # prefix
store_id: {exists: true}, # exists
_not: {store_id: 1}, # negate a condition
_or: [{in_stock: true}, {backordered: true}],
_and: [{in_stock: true}, {backordered: true}]
}Order
order: {_score: :desc} # most relevant first - defaultAll of these sort options are supported
Limit / offset
limit: 20, offset: 40Select
select: [:name]These source filtering options are supported
Searches return a Searchkick::Relation object. This responds like an array to most methods.
results = Product.search("milk")
results.size
results.any?
results.each { |result| ... }By default, ids are fetched from the search server and records are fetched from your database. To fetch everything from the search server, use:
Product.search("apples", load: false)Get total results
results.total_countGet the time the search took (in milliseconds)
results.tookGet the full response from the search server
results.responseNote: By default, Elasticsearch and OpenSearch limit paging to the first 10,000 results for performance. This applies to the total count as well.
Boost important fields
fields: ["title^10", "description"]Boost by the value of a field (field must be numeric)
boost_by: [:orders_count] # give popular documents a little boost
boost_by: {orders_count: {factor: 10}} # default factor is 1Boost matching documents
boost_where: {user_id: 1}
boost_where: {user_id: {value: 1, factor: 100}} # default factor is 1000
boost_where: {user_id: [{value: 1, factor: 100}, {value: 2, factor: 200}]}Boost by recency
boost_by_recency: {created_at: {scale: "7d", decay: 0.5}}You can also boost by:
Use a * for the query.
Product.search("*")Plays nicely with kaminari and will_paginate.
# controller
@products = Product.search("milk", page: params[:page], per_page: 20)View with kaminari
<%= paginate @products %>View with will_paginate
<%= will_paginate @products %>By default, results must match all words in the query.
Product.search("fresh honey") # fresh AND honeyTo change this, use:
Product.search("fresh honey", operator: "or") # fresh OR honeyBy default, results must match the entire word - back will not match backpack. You can change this behavior with:
class Product < ApplicationRecord
searchkick word_start: [:name]
endAnd to search (after you reindex):
Product.search("back", fields: [:name], match: :word_start)Available options are:
| Option | Matches | Example |
|---|---|---|
:word |
entire word | apple matches apple |
:word_start |
start of word | app matches apple |
:word_middle |
any part of word | ppl matches apple |
:word_end |
end of word | ple matches apple |
:text_start |
start of text | gre matches green apple, app does not match |
:text_middle |
any part of text | een app matches green apple |
:text_end |
end of text | ple matches green apple, een does not match |
The default is :word. The most matches will happen with :word_middle.
To specify different matching for different fields, use:
Product.search(query, fields: [{name: :word_start}, {brand: :word_middle}])To match a field exactly (case-sensitive), use:
Product.search(query, fields: [{name: :exact}])To only match the exact order, use:
Product.search("fresh honey", match: :phrase)Searchkick stems words by default for better matching. apple and apples both stem to appl, so searches for either term will have the same matches.
Searchkick defaults to English for stemming. To change this, use:
class Product < ApplicationRecord
searchkick language: "german"
endSee the list of languages. A few languages require plugins:
chinese- analysis-ik pluginchinese2- analysis-smartcn pluginjapanese- analysis-kuromoji pluginkorean- analysis-openkoreantext pluginkorean2- analysis-nori pluginpolish- analysis-stempel pluginukrainian- analysis-ukrainian pluginvietnamese- analysis-vietnamese plugin
You can also use a Hunspell dictionary for stemming.
class Product < ApplicationRecord
searchkick stemmer: {type: "hunspell", locale: "en_US"}
endDisable stemming with:
class Image < ApplicationRecord
searchkick stem: false
endExclude certain words from stemming with:
class Image < ApplicationRecord
searchkick stem_exclusion: ["apples"]
endOr change how words are stemmed:
class Image < ApplicationRecord
searchkick stemmer_override: ["apples => other"]
endclass Product < ApplicationRecord
searchkick search_synonyms: [["pop", "soda"], ["burger", "hamburger"]]
endCall Product.reindex after changing synonyms. Synonyms are applied at search time before stemming, and can be a single word or multiple words.
For directional synonyms, use:
search_synonyms: ["lightbulb => halogenlamp"]The above approach works well when your synonym list is static, but in practice, this is often not the case. When you analyze search conversions, you often want to add new synonyms without a full reindex.
For Elasticsearch 7.3+ and OpenSearch, we recommend placing synonyms in a file on the search server (in the config directory). This allows you to reload synonyms without reindexing.
pop, soda
burger, hamburgerThen use:
class Product < ApplicationRecord
searchkick search_synonyms: "synonyms.txt"
endAnd reload with:
Product.search_index.reload_synonymsYou can use a library like ActsAsTaggableOn and do:
class Product < ApplicationRecord
acts_as_taggable
scope :search_import, -> { includes(:tags) }
def search_data
{
name_tagged: "#{name} #{tags.map(&:name).join(" ")}"
}
end
endSearch with:
Product.search(query, fields: [:name_tagged])By default, Searchkick handles misspelled queries by returning results with an edit distance of one.
You can change this with:
Product.search("zucini", misspellings: {edit_distance: 2}) # zucchiniTo prevent poor precision and improve performance for correctly spelled queries (which should be a majority for most applications), Searchkick can first perform a search without misspellings, and if there are too few results, perform another with them.
Product.search("zuchini", misspellings: {below: 5})If there are fewer than 5 results, a 2nd search is performed with misspellings enabled. The result of this query is returned.
Turn off misspellings with:
Product.search("zuchini", misspellings: false) # no zucchiniSpecify which fields can include misspellings with:
Product.search("zucini", fields: [:name, :color], misspellings: {fields: [:name]})When doing this, you must also specify fields to search
If a user searches butter, they may also get results for peanut butter. To prevent this, use:
Product.search("butter", exclude: ["peanut butter"])You can map queries and terms to exclude with:
exclude_queries = {
"butter" => ["peanut butter"],
"cream" => ["ice cream", "whipped cream"]
}
Product.search(query, exclude: exclude_queries[query])You can demote results by boosting by a factor less than one:
Product.search("butter", boost_where: {category: {value: "pantry", factor: 0.5}})Search 🍨🍰 and get ice cream cake!
Add this line to your application’s Gemfile:
gem "gemoji-parser"And use:
Product.search("🍨🍰", emoji: true)Control what data is indexed with the search_data method. Call Product.reindex after changing this method.
class Product < ApplicationRecord
belongs_to :department
def search_data
{
name: name,
department_name: department.name,
on_sale: sale_price.present?
}
end
endSearchkick uses find_in_batches to import documents. To eager load associations, use the search_import scope.
class Product < ApplicationRecord
scope :search_import, -> { includes(:department) }
endBy default, all records are indexed. To control which records are indexed, use the should_index? method.
class Product < ApplicationRecord
def should_index?
active # only index active records
end
endIf a reindex is interrupted, you can resume it with:
Product.reindex(resume: true)For large data sets, try parallel reindexing.
- when you install or upgrade searchkick
- change the
search_datamethod - change the
searchkickmethod
- app starts
There are four strategies for keeping the index synced with your database.
- Inline (default)
Anytime a record is inserted, updated, or deleted
- Asynchronous
Use background jobs for better performance
class Product < ApplicationRecord
searchkick callbacks: :async
endJobs are added to a queue named searchkick.
- Queuing
Push ids of records that need updated to a queue and reindex in the background in batches. This is more performant than the asynchronous method, which updates records individually. See how to set up.
- Manual
Turn off automatic syncing
class Product < ApplicationRecord
searchkick callbacks: false
endAnd reindex a record or relation manually.
product.reindex
# or
store.products.reindex(mode: :async)You can also do bulk updates.
Searchkick.callbacks(:bulk) do
Product.find_each(&:update_fields)
endOr temporarily skip updates.
Searchkick.callbacks(false) do
Product.find_each(&:update_fields)
endOr override the model’s strategy.
product.reindex(mode: :async) # :inline or :queueData is not automatically synced when an association is updated. If this is desired, add a callback to reindex:
class Image < ApplicationRecord
belongs_to :product
after_commit :reindex_product
def reindex_product
product.reindex
end
endIf you have a default scope that filters records, use the should_index? method to exclude them from indexing:
class Product < ApplicationRecord
default_scope { where(deleted_at: nil) }
def should_index?
deleted_at.nil?
end
endIf you want to index and search filtered records, set:
class Product < ApplicationRecord
searchkick unscope: true
endThe best starting point to improve your search by far is to track searches and conversions. Searchjoy makes it easy.
Product.search("apple", track: {user_id: current_user.id})See the docs for how to install and use. Focus on top searches with a low conversion rate.
Searchkick can then use the conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches. This can make a huge difference on the quality of your search.
Add conversion data with:
class Product < ApplicationRecord
has_many :conversions, class_name: "Searchjoy::Conversion", as: :convertable
has_many :searches, class_name: "Searchjoy::Search", through: :conversions
searchkick conversions: [:conversions] # name of field
def search_data
{
name: name,
conversions: searches.group(:query).distinct.count(:user_id)
# {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
}
end
endReindex and set up a cron job to add new conversions daily. For zero downtime deployment, temporarily set conversions: false in your search calls until the data is reindexed.
A performant way to do conversions is to cache them to prevent N+1 queries. For Postgres, create a migration with:
add_column :products, :search_conversions, :jsonbFor MySQL, use :json, and for others, use :text with a JSON serializer.
Next, update your model. Create a separate method for conversion data so you can use partial reindexing.
class Product < ApplicationRecord
searchkick conversions: [:conversions]
def search_data
{
name: name,
category: category
}.merge(conversions_data)
end
def conversions_data
{
conversions: search_conversions || {}
}
end
endDeploy and reindex your data. For zero downtime deployment, temporarily set conversions: false in your search calls until the data is reindexed.
Product.reindexThen, create a job to update the conversions column and reindex records with new conversions. Here’s one you can use for Searchjoy:
class UpdateConversionsJob < ApplicationJob
def perform(class_name, since: nil, update: true, reindex: true)
model = Searchkick.load_model(class_name)
# get records that have a recent conversion
recently_converted_ids =
Searchjoy::Conversion.where(convertable_type: class_name).where(created_at: since..)
.order(:convertable_id).distinct.pluck(:convertable_id)
# split into batches
recently_converted_ids.in_groups_of(1000, false) do |ids|
if update
# fetch conversions
conversions =
Searchjoy::Conversion.where(convertable_id: ids, convertable_type: class_name)
.joins(:search).where.not(searchjoy_searches: {user_id: nil})
.group(:convertable_id, :query).distinct.count(:user_id)
# group by record
conversions_by_record = {}
conversions.each do |(id, query), count|
(conversions_by_record[id] ||= {})[query] = count
end
# update conversions column
model.transaction do
conversions_by_record.each do |id, conversions|
model.where(id: id).update_all(search_conversions: conversions)
end
end
end
if reindex
# reindex conversions data
model.where(id: ids).reindex(:conversions_data)
end
end
end
endRun the job:
UpdateConversionsJob.perform_now("Product")And set it up to run daily.
UpdateConversionsJob.perform_later("Product", since: 1.day.ago)Order results differently for each user. For example, show a user’s previously purchased products before other results.
class Product < ApplicationRecord
def search_data
{
name: name,
orderer_ids: orders.pluck(:user_id) # boost this product for these users
}
end
endReindex and search with:
Product.search("milk", boost_where: {orderer_ids: current_user.id})Autocomplete predicts what a user will type, making the search experience faster and easier.
Note: To autocomplete on search terms rather than results, check out Autosuggest.
Note 2: If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s much faster to load all records into JavaScript and autocomplete there (eliminates network requests).
First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.
class Movie < ApplicationRecord
searchkick word_start: [:title, :director]
endReindex and search with:
Movie.search("jurassic pa", fields: [:title], match: :word_start)Typically, you want to use a JavaScript library like typeahead.js or jQuery UI.
First, add a route and controller action.
class MoviesController < ApplicationController
def autocomplete
render json: Movie.search(params[:query], {
fields: ["title^5", "director"],
match: :word_start,
limit: 10,
load: false,
misspellings: {below: 5}
}).map(&:title)
end
endNote: Use load: false and misspellings: {below: n} (or misspellings: false) for best performance.
Then add the search box and JavaScript code to a view.
<input type="text" id="query" name="query" />
<script src="jquery.js"></script>
<script src="typeahead.bundle.js"></script>
<script>
var movies = new Bloodhound({
datumTokenizer: Bloodhound.tokenizers.whitespace,
queryTokenizer: Bloodhound.tokenizers.whitespace,
remote: {
url: '/movies/autocomplete?query=%QUERY',
wildcard: '%QUERY'
}
});
$('#query').typeahead(null, {
source: movies
});
</script>
class Product < ApplicationRecord
searchkick suggest: [:name] # fields to generate suggestions
endReindex and search with:
products = Product.search("peantu butta", suggest: true)
products.suggestions # ["peanut butter"]Aggregations provide aggregated search data.
products = Product.search("chuck taylor", aggs: [:product_type, :gender, :brand])
products.aggsBy default, where conditions apply to aggregations.
Product.search("wingtips", where: {color: "brandy"}, aggs: [:size])
# aggregations for brandy wingtips are returnedChange this with:
Product.search("wingtips", where: {color: "brandy"}, aggs: [:size], smart_aggs: false)
# aggregations for all wingtips are returnedSet where conditions for each aggregation separately with:
Product.search("wingtips", aggs: {size: {where: {color: "brandy"}}})Limit
Product.search("apples", aggs: {store_id: {limit: 10}})Order
Product.search("wingtips", aggs: {color: {order: {"_key" => "asc"}}}) # alphabeticallyAll of these options are supported
Ranges
price_ranges = [{to: 20}, {from: 20, to: 50}, {from: 50}]
Product.search("*", aggs: {price: {ranges: price_ranges}})Minimum document count
Product.search("apples", aggs: {store_id: {min_doc_count: 2}})Script support
Product.search("*", aggs: {color: {script: {source: "'Color: ' + _value"}}})Date histogram
Product.search("pear", aggs: {products_per_year: {date_histogram: {field: :created_at, interval: :year}}})For other aggregation types, including sub-aggregations, use body_options:
Product.search("orange", body_options: {aggs: {price: {histogram: {field: :price, interval: 10}}}})Specify which fields to index with highlighting.
class Band < ApplicationRecord
searchkick highlight: [:name]
endHighlight the search query in the results.
bands = Band.search("cinema", highlight: true)View the highlighted fields with:
bands.with_highlights.each do |band, highlights|
highlights[:name] # "Two Door <em>Cinema</em> Club"
endTo change the tag, use:
Band.search("cinema", highlight: {tag: "<strong>"})To highlight and search different fields, use:
Band.search("cinema", fields: [:name], highlight: {fields: [:description]})By default, the entire field is highlighted. To get small snippets instead, use:
bands = Band.search("cinema", highlight: {fragment_size: 20})
bands.with_highlights(multiple: true).each do |band, highlights|
highlights[:name].join(" and ")
endAdditional options can be specified for each field:
Band.search("cinema", fields: [:name], highlight: {fields: {name: {fragment_size: 200}}})You can find available highlight options in the Elasticsearch reference.
Find similar items.
product = Product.first
product.similar(fields: [:name], where: {size: "12 oz"})class Restaurant < ApplicationRecord
searchkick locations: [:location]
def search_data
attributes.merge(location: {lat: latitude, lon: longitude})
end
endReindex and search with:
Restaurant.search("pizza", where: {location: {near: {lat: 37, lon: -114}, within: "100mi"}}) # or 160kmBounded by a box
Restaurant.search("sushi", where: {location: {top_left: {lat: 38, lon: -123}, bottom_right: {lat: 37, lon: -122}}})Note: top_right and bottom_left also work
Bounded by a polygon
Restaurant.search("dessert", where: {location: {geo_polygon: {points: [{lat: 38, lon: -123}, {lat: 39, lon: -123}, {lat: 37, lon: 122}]}}})Boost results by distance - closer results are boosted more
Restaurant.search("noodles", boost_by_distance: {location: {origin: {lat: 37, lon: -122}}})Also supports additional options
Restaurant.search("wings", boost_by_distance: {location: {origin: {lat: 37, lon: -122}, function: "linear", scale: "30mi", decay: 0.5}})You can also index and search geo shapes.
class Restaurant < ApplicationRecord
searchkick geo_shape: [:bounds]
def search_data
attributes.merge(
bounds: {
type: "envelope",
coordinates: [{lat: 4, lon: 1}, {lat: 2, lon: 3}]
}
)
end
endSee the Elasticsearch documentation for details.
Find shapes intersecting with the query shape
Restaurant.search("soup", where: {bounds: {geo_shape: {type: "polygon", coordinates: [[{lat: 38, lon: -123}, ...]]}}})Falling entirely within the query shape
Restaurant.search("salad", where: {bounds: {geo_shape: {type: "circle", relation: "within", coordinates: {lat: 38, lon: -123}, radius: "1km"}}})Not touching the query shape
Restaurant.search("burger", where: {bounds: {geo_shape: {type: "envelope", relation: "disjoint", coordinates: [{lat: 38, lon: -123}, {lat: 37, lon: -122}]}}})Searchkick supports single table inheritance.
class Dog < Animal
endIn your parent model, set:
class Animal < ApplicationRecord
searchkick inheritance: true
endThe parent and child model can both reindex.
Animal.reindex
Dog.reindex # equivalent, all animals reindexedAnd to search, use:
Animal.search("*") # all animals
Dog.search("*") # just dogs
Animal.search("*", type: [Dog, Cat]) # just cats and dogsNotes:
-
The
suggestoption retrieves suggestions from the parent at the moment.Dog.search("airbudd", suggest: true) # suggestions for all animals
-
This relies on a
typefield that is automatically added to the indexed document. Be wary of defining your owntypefield insearch_data, as it will take precedence.
To help with debugging queries, you can use:
Product.search("soap", debug: true)This prints useful info to stdout.
See how the search server scores your queries with:
Product.search("soap", explain: true).responseSee how the search server tokenizes your queries with:
Product.search_index.tokens("Dish Washer Soap", analyzer: "searchkick_index")
# ["dish", "dishwash", "washer", "washersoap", "soap"]
Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search")
# ["dishwashersoap"] - no match
Product.search_index.tokens("dishwasher soap", analyzer: "searchkick_search2")
# ["dishwash", "soap"] - match!!Partial matches
Product.search_index.tokens("San Diego", analyzer: "searchkick_word_start_index")
# ["s", "sa", "san", "d", "di", "die", "dieg", "diego"]
Product.search_index.tokens("dieg", analyzer: "searchkick_word_search")
# ["dieg"] - match!!See the complete list of analyzers.
As you iterate on your search, it’s a good idea to add tests.
For performance, only enable Searchkick callbacks for the tests that need it.
Rails 6 enables parallel tests by default. Add to your test/test_helper.rb:
class ActiveSupport::TestCase
parallelize_setup do |worker|
Searchkick.index_suffix = worker
# reindex models
Product.reindex
# and disable callbacks
Searchkick.disable_callbacks
end
endAnd use:
class ProductTest < ActiveSupport::TestCase
def setup
Searchkick.enable_callbacks
end
def teardown
Searchkick.disable_callbacks
end
def test_search
Product.create!(name: "Apple")
Product.search_index.refresh
assert_equal ["Apple"], Product.search("apple").map(&:name)
end
endAdd to your test/test_helper.rb:
# reindex models
Product.reindex
# and disable callbacks
Searchkick.disable_callbacksAnd use:
class ProductTest < Minitest::Test
def setup
Searchkick.enable_callbacks
end
def teardown
Searchkick.disable_callbacks
end
def test_search
Product.create!(name: "Apple")
Product.search_index.refresh
assert_equal ["Apple"], Product.search("apple").map(&:name)
end
endAdd to your spec/spec_helper.rb:
RSpec.configure do |config|
config.before(:suite) do
# reindex models
Product.reindex
# and disable callbacks
Searchkick.disable_callbacks
end
config.around(:each, search: true) do |example|
Searchkick.callbacks(nil) do
example.run
end
end
endAnd use:
describe Product, search: true do
it "searches" do
Product.create!(name: "Apple")
Product.search_index.refresh
assert_equal ["Apple"], Product.search("apple").map(&:name)
end
endUse a trait and an after create hook for each indexed model:
FactoryBot.define do
factory :product do
# ...
# Note: This should be the last trait in the list so `reindex` is called
# after all the other callbacks complete.
trait :reindex do
after(:create) do |product, _evaluator|
product.reindex(refresh: true)
end
end
end
end
# use it
FactoryBot.create(:product, :some_trait, :reindex, some_attribute: "foo")Check out setup-elasticsearch for an easy way to install Elasticsearch:
- uses: ankane/setup-elasticsearch@v1And setup-opensearch for an easy way to install OpenSearch:
- uses: ankane/setup-opensearch@v1For the search server, Searchkick uses ENV["ELASTICSEARCH_URL"] for Elasticsearch and ENV["OPENSEARCH_URL"] for OpenSearch. This defaults to http://localhost:9200.
Create an initializer config/initializers/elasticsearch.rb with:
ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"Then deploy and reindex:
rake searchkick:reindex:allChoose an add-on: Bonsai, SearchBox, or Elastic Cloud.
For Elasticsearch on Bonsai:
heroku addons:create bonsai
heroku config:set ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`For OpenSearch on Bonsai:
heroku addons:create bonsai --engine=opensearch
heroku config:set OPENSEARCH_URL=`heroku config:get BONSAI_URL`For SearchBox:
heroku addons:create searchbox:starter
heroku config:set ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`For Elastic Cloud (previously Found):
heroku addons:create foundelasticsearch
heroku addons:open foundelasticsearchVisit the Shield page and reset your password. You’ll need to add the username and password to your url. Get the existing url with:
heroku config:get FOUNDELASTICSEARCH_URLAnd add elastic:password@ right after https:// and add port 9243 at the end:
heroku config:set ELASTICSEARCH_URL=https://elastic:[email protected]:9243Then deploy and reindex:
heroku run rake searchkick:reindex:allCreate an initializer config/initializers/opensearch.rb with:
ENV["OPENSEARCH_URL"] = "https://es-domain-1234.us-east-1.es.amazonaws.com:443"To use signed requests, include in your Gemfile:
gem "faraday_middleware-aws-sigv4"and add to your initializer:
Searchkick.aws_credentials = {
access_key_id: ENV["AWS_ACCESS_KEY_ID"],
secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"],
region: "us-east-1"
}Then deploy and reindex:
rake searchkick:reindex:allCreate an initializer with:
ENV["ELASTICSEARCH_URL"] = "https://user:password@host:port"
# or
ENV["OPENSEARCH_URL"] = "https://user:password@host:port"Then deploy and reindex:
rake searchkick:reindex:allWe recommend encrypting data at rest and in transit (even inside your own network). This is especially important if you send personal data of your users to the search server.
Bonsai, Elastic Cloud, and Amazon OpenSearch Service all support encryption at rest and HTTPS.
Create an initializer with multiple hosts:
ENV["ELASTICSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"
# or
ENV["OPENSEARCH_URL"] = "https://user:password@host1,https://user:password@host2"Create an initializer with:
Searchkick.client_options[:reload_connections] = trueSee the docs for Elasticsearch or Opensearch for a complete list of options.
Add the following to config/environments/production.rb:
config.lograge.custom_options = lambda do |event|
options = {}
options[:search] = event.payload[:searchkick_runtime] if event.payload[:searchkick_runtime].to_f > 0
options
endSee Production Rails for other good practices.
Significantly increase performance with faster JSON generation. Add Oj to your Gemfile.
gem "oj"This speeds up all JSON generation and parsing in your application (automatically!)
Significantly increase performance with persistent HTTP connections. Add Typhoeus to your Gemfile and it’ll automatically be used.
gem "typhoeus"To reduce log noise, create an initializer with:
Ethon.logger = Logger.new(nil)If you run into issues on Windows, check out this post.
By default, all string fields are searchable (can be used in fields option). Speed up indexing and reduce index size by only making some fields searchable.
class Product < ApplicationRecord
searchkick searchable: [:name]
endBy default, all string fields are filterable (can be used in where option). Speed up indexing and reduce index size by only making some fields filterable.
class Product < ApplicationRecord
searchkick filterable: [:brand]
endNote: Non-string fields are always filterable and should not be passed to this option.
For large data sets, you can use background jobs to parallelize reindexing.
Product.reindex(mode: :async)
# {index_name: "products_production_20170111210018065"}Once the jobs complete, promote the new index with:
Product.search_index.promote(index_name)You can optionally track the status with Redis:
Searchkick.redis = Redis.newAnd use:
Searchkick.reindex_status(index_name)You can also have Searchkick wait for reindexing to complete
Product.reindex(mode: :async, wait: true)You can use ActiveJob::TrafficControl to control concurrency. Install the gem:
gem "activejob-traffic_control", ">= 0.1.3"And create an initializer with:
ActiveJob::TrafficControl.client = Searchkick.redis
class Searchkick::BulkReindexJob
concurrency 3
endThis will allow only 3 jobs to run at once.
You can specify a longer refresh interval while reindexing to increase performance.
Product.reindex(mode: :async, refresh_interval: "30s")Note: This only makes a noticable difference with parallel reindexing.
When promoting, have it restored to the value in your mapping (defaults to 1s).
Product.search_index.promote(index_name, update_refresh_interval: true)Push ids of records needing reindexing to a queue and reindex in bulk for better performance. First, set up Redis in an initializer. We recommend using connection_pool.
Searchkick.redis = ConnectionPool.new { Redis.new }And ask your models to queue updates.
class Product < ApplicationRecord
searchkick callbacks: :queue
endThen, set up a background job to run.
Searchkick::ProcessQueueJob.perform_later(class_name: "Product")You can check the queue length with:
Product.search_index.reindex_queue.lengthFor more tips, check out Keeping Elasticsearch in Sync.
Searchkick supports routing, which can significantly speed up searches.
class Business < ApplicationRecord
searchkick routing: true
def search_routing
city_id
end
endReindex and search with:
Business.search("ice cream", routing: params[:city_id])Reindex a subset of attributes to reduce time spent generating search data and cut down on network traffic.
class Product < ApplicationRecord
def search_data
{
name: name,
category: category
}.merge(prices_data)
end
def prices_data
{
price: price,
sale_price: sale_price
}
end
endAnd use:
Product.reindex(:prices_data)Searchkick makes it easy to use the Elasticsearch or OpenSearch DSL on its own.
Create a custom mapping:
class Product < ApplicationRecord
searchkick mappings: {
properties: {
name: {type: "keyword"}
}
}
endNote: If you use a custom mapping, you'll need to use custom searching as well.
To keep the mappings and settings generated by Searchkick, use:
class Product < ApplicationRecord
searchkick merge_mappings: true, mappings: {...}
endAnd use the body option to search:
products = Product.search(body: {query: {match: {name: "milk"}}})View the response with:
products.responseTo modify the query generated by Searchkick, use:
products = Product.search("milk", body_options: {min_score: 1})or
products =
Product.search("apples") do |body|
body[:min_score] = 1
endTo access the Elasticsearch::Client or OpenSearch::Client directly, use:
Searchkick.clientTo batch search requests for performance, use:
products = Product.search("snacks")
coupons = Coupon.search("snacks")
Searchkick.multi_search([products, coupons])Then use products and coupons as typical results.
Note: Errors are not raised as with single requests. Use the error method on each query to check for errors.
Search across multiple models with:
Searchkick.search("milk", models: [Product, Category])Boost specific models with:
indices_boost: {Category => 2, Product => 1}Check out this great post on the Apartment gem. Follow a similar pattern if you use another gem.
Searchkick also supports the scroll API. Scrolling is not intended for real time user requests, but rather for processing large amounts of data.
Product.search("*", scroll: "1m").scroll do |batch|
# process batch ...
endYou can also scroll batches manually.
products = Product.search("*", scroll: "1m")
while products.any?
# process batch ...
products = products.scroll
end
products.clear_scrollBy default, Elasticsearch and OpenSearch limit paging to the first 10,000 results. Here’s why. We don’t recommend changing this, but if you really need all results, you can use:
class Product < ApplicationRecord
searchkick deep_paging: true
endIf you just need an accurate total count, you can instead use:
Product.search("pears", body_options: {track_total_hits: true})To query nested data, use dot notation.
Product.search("san", fields: ["store.city"], where: {"store.zip_code" => 12345})You can use custom mapping and searching to index vectors and perform k-nearest neighbor search. See the examples for Elasticsearch and OpenSearch.
Reindex one record
product = Product.find(1)
product.reindexReindex multiple records
Product.where(store_id: 1).reindexReindex associations
store.products.reindexRemove old indices
Product.search_index.clean_indicesUse custom settings
class Product < ApplicationRecord
searchkick settings: {number_of_shards: 3}
endUse a different index name
class Product < ApplicationRecord
searchkick index_name: "products_v2"
endUse a dynamic index name
class Product < ApplicationRecord
searchkick index_name: -> { "#{name.tableize}-#{I18n.locale}" }
endPrefix the index name
class Product < ApplicationRecord
searchkick index_prefix: "datakick"
endFor all models
Searchkick.index_prefix = "datakick"Use a different term for boosting by conversions
Product.search("banana", conversions_term: "organic banana")Multiple conversion fields
class Product < ApplicationRecord
has_many :searches, class_name: "Searchjoy::Search"
# searchkick also supports multiple "conversions" fields
searchkick conversions: ["unique_user_conversions", "total_conversions"]
def search_data
{
name: name,
unique_user_conversions: searches.group(:query).distinct.count(:user_id),
# {"ice cream" => 234, "chocolate" => 67, "cream" => 2}
total_conversions: searches.group(:query).count
# {"ice cream" => 412, "chocolate" => 117, "cream" => 6}
}
end
endand during query time:
Product.search("banana") # boost by both fields (default)
Product.search("banana", conversions: "total_conversions") # only boost by total_conversions
Product.search("banana", conversions: false) # no conversion boostingChange timeout
Searchkick.timeout = 15 # defaults to 10Set a lower timeout for searches
Searchkick.search_timeout = 3Change the search method name
Searchkick.search_method_name = :lookupChange search queue name
Searchkick.queue_name = :search_reindexEager load associations
Product.search("milk", includes: [:brand, :stores])Eager load different associations by model
Searchkick.search("*", models: [Product, Store], model_includes: {Product => [:store], Store => [:product]})Run additional scopes on results
Product.search("milk", scope_results: ->(r) { r.with_attached_images })Specify default fields to search
class Product < ApplicationRecord
searchkick default_fields: [:name]
endTurn off special characters
class Product < ApplicationRecord
# A will not match Ä
searchkick special_characters: false
endTurn on stemming for conversions
class Product < ApplicationRecord
searchkick stem_conversions: true
endMake search case-sensitive
class Product < ApplicationRecord
searchkick case_sensitive: true
endNote: If misspellings are enabled (default), results with a single character case difference will match. Turn off misspellings if this is not desired.
Change import batch size
class Product < ApplicationRecord
searchkick batch_size: 200 # defaults to 1000
endCreate index without importing
Product.reindex(import: false)Use a different id
class Product < ApplicationRecord
def search_document_id
custom_id
end
endAdd request parameters like search_type
Product.search("carrots", request_params: {search_type: "dfs_query_then_fetch"})Set options across all models
Searchkick.model_options = {
batch_size: 200
}Reindex conditionally
class Product < ApplicationRecord
searchkick callbacks: false
# add the callbacks manually
after_commit :reindex, if: -> (model) { model.previous_changes.key?("name") } # use your own condition
endReindex all models - Rails only
rake searchkick:reindex:allTurn on misspellings after a certain number of characters
Product.search("api", misspellings: {prefix_length: 2}) # api, apt, no ahiNote: With this option, if the query length is the same as prefix_length, misspellings are turned off with Elasticsearch 7 and OpenSearch 1
Product.search("ah", misspellings: {prefix_length: 2}) # ah, no ahaBigDecimal values are indexed as floats by default so they can be used for boosting. Convert them to strings to keep full precision.
class Product < ApplicationRecord
def search_data
{
units: units.to_s("F")
}
end
endElasticsearch and OpenSearch are eventually consistent, meaning it can take up to a second for a change to reflect in search. You can use the refresh method to have it show up immediately.
product.save!
Product.search_index.refreshDue to the distributed nature of Elasticsearch and OpenSearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:
class Product < ApplicationRecord
searchkick settings: {number_of_shards: 1}
endFor convenience, this is set by default in the test environment.
Searchkick 5 supports both the elasticsearch and opensearch-ruby gems. Add the one you want to use to your Gemfile:
gem "elasticsearch"
# or
gem "opensearch-ruby"If using the deprecated faraday_middleware-aws-signers-v4 gem, switch to faraday_middleware-aws-sigv4.
Also, searches now use lazy loading:
# search not executed
Product.search("milk")
# search executed
Product.search("milk").to_aYou can reindex relations in the background:
store.products.reindex(mode: :async)
# or
store.products.reindex(mode: :queue)And there’s a new option for models with default scopes.
Check out the changelog for the full list of changes.
View the changelog.
Thanks to Karel Minarik for Elasticsearch Ruby and Tire, Jaroslav Kalistsuk for zero downtime reindexing, and Alex Leschenko for Elasticsearch autocomplete.
Everyone is encouraged to help improve this project. Here are a few ways you can help:
- Report bugs
- Fix bugs and submit pull requests
- Write, clarify, or fix documentation
- Suggest or add new features
To get started with development:
git clone https://github.com/ankane/searchkick.git
cd searchkick
bundle install
bundle exec rake testFeel free to open an issue to get feedback on your idea before spending too much time on it.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent