clojurewerkz / elastisch Goto Github PK

There's some APIs in elastisch that support trailing named arguments. For instance native.document.search

(defn search
  "Performs a search query across one or more indexes and one or more mapping types.

   Examples:

   (require '[clojurewerkz.elastisch.native.document :as doc])
   (require '[clojurewerkz.elastisch.query :as q])

   (doc/search \"people\" \"person\" :query (q/prefix :username \"appl\"))"
  [index mapping-type & {:as options}]
  (let [ft                  (es/search (cnv/->search-request index mapping-type options))
        ^SearchResponse res (.actionGet ft)]
    (cnv/search-response->seq res)))

This is handy when I'm directly calling search, ala

(es-doc/seach "foo" "bar" :size 10)

but is a PITA when trying to interact with search from another method that takes options. For instance, if I wanted to have a search method that I want to take a query and filter option, I end up having to write awkward code like:

(ns my.custom.type.that.has.defaults.and.such ...)

(defn search [name & {:as options}]
  (es-doc/search (concat ["myindex" "mytype" :size 15]
                         (flatten (vec (select-keys options [:query :filter]))))))

Rather than start my whole rant against hating the trailing named argument pattern (versus just passing a map), I'm asking here that when elastisch supports trailing named arguments it also supports passing them in as a map.

There may be a clojure pattern that makes this easier that I'm unaware of, but on the receiving side this the following works:

(defn search
  "Performs a search query across one or more indexes and one or more mapping types.

   Examples:

   (require '[clojurewerkz.elastisch.native.document :as doc])
   (require '[clojurewerkz.elastisch.query :as q])

   (doc/search \"people\" \"person\" :query (q/prefix :username \"appl\"))"
  [index mapping-type & args]
  (let [options (if (map? args) args (apply array-map args))
        ft                  (es/search (cnv/->search-request index mapping-type options))
        ^SearchResponse res (.actionGet ft)]
    (cnv/search-response->seq res)))

I'm aware throwing that if in all over the place is annoying, but such is the world of trailing named args.

Document count fails. Here is the code I am using:

search> (esi/create "idx")
{:ok true, :acknowledged true}
search> (esd/create "idx" :person {:name "ponzao"})
{:ok true, :_index "idx", :_type ":person", :_id "IoC_2JvFTRGESc_pECKUpQ", :_version 1}
search> (esd/count "idx" :person)
{:count 0,
 :_shards
 {:failures
  [{:shard 0,
    :reason
    "BroadcastShardOperationFailedException[[idx][0] ]; nested: QueryParsingException[[idx] Failed to parse]; nested: ElasticSearchParseException[Failed to derive xcontent from org.elasticsearch.common.bytes.ChannelBufferBytesReference@2e7227a8]; ",
    :index "idx"}
   {:shard 1,
    :reason
    "BroadcastShardOperationFailedException[[idx][1] ]; nested: QueryParsingException[[idx] Failed to parse]; nested: ElasticSearchParseException[Failed to derive xcontent from org.elasticsearch.common.bytes.ChannelBufferBytesReference@2e7227a8]; ",
    :index "idx"}
   {:shard 2,
    :reason
    "BroadcastShardOperationFailedException[[idx][2] ]; nested: QueryParsingException[[idx] Failed to parse]; nested: ElasticSearchParseException[Failed to derive xcontent from org.elasticsearch.common.bytes.ChannelBufferBytesReference@2e7227a8]; ",
    :index "idx"}
   {:shard 3,
    :reason
    "BroadcastShardOperationFailedException[[idx][3] ]; nested: QueryParsingException[[idx] Failed to parse]; nested: ElasticSearchParseException[Failed to derive xcontent from org.elasticsearch.common.bytes.ChannelBufferBytesReference@2e7227a8]; ",
    :index "idx"}
   {:shard 4,
    :reason
    "BroadcastShardOperationFailedException[[idx][4] ]; nested: QueryParsingException[[idx] Failed to parse]; nested: ElasticSearchParseException[Failed to derive xcontent from org.elasticsearch.common.bytes.ChannelBufferBytesReference@2e7227a8]; ",
    :index "idx"}],
  :total 5,
  :failed 5,
  :successful 0}}

Manual HTTP access works:

curl -XGET 'http://localhost:9200/idx/:person/_count'
{"count":1,"_shards":{"total":5,"successful":5,"failed":0}}

The REST one already supports aggregations.

Operations such as

• Create index
• Delete index
• Get indices stats
• Update index settings

should be supported via native API as well.

Possible issue reported in this mailing list thread

Updated for ES 1.2 API.

elastisch http rest client does not support partial document update.
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/partial-updates.html

POST /website/blog/1/_update
{
   "doc" : {
      "tags" : [ "testing" ],
      "views": 0
   }
}

There is clojurewerkz.elastisch.rest.document/update-with-script, but no document api without script for partial document update in elastisch

The docs for update-mapping on http://clojureelasticsearch.info/articles/indexing.html say to use it as:

(defn -main
  [& args]
  (esr/connect! "http://127.0.0.1:9200")
  (esi/create "myapp_development" :settings {:index {:number_of_replicas 1}})
  ;; update a single mapping type for the index
  (esi/update-mapping "myapp_development" "person" :mapping {:properties {:first-name {:type "string" :store "no"}}}))

However, at least for native.index update-mapping actually requires the format to be:

(esi/update-mapping "myapp_development" "person" :mapping {"person" {:properties {:first-name {:type "string" :store "no"}}}})

I'm not sure if this is a bug in the docs or in native. I personally expected it to be the version in the doc, but the version it actually requires is closer to the format used in create-index (but with the :mappings plural key).

The actual error I'm getting is:

Exception in thread "main" java.util.concurrent.ExecutionException: org.elasticsearch.transport.RemoteTransportException: [Joystick]    [inet[/127.0.0.1:9300]][indices/mapping/put]

I can send a PR against https://github.com/clojurewerkz/elastisch.docs if it's just the docs are wrong.

ElasticSearch client post 1.0 uses *Builder classes for requests. We should use them where possible (we currently instantiate *Requests directly in most places).

Assuming an index named products populated with sample data, the following search query does not return any hits:

(document/search "products" "product" :query { :term { :name "macbook" } })

But manually constructing a post with the same query map returns the hit results as expected.

(rest/post (rest/search-url "products") :body { :query { :term { :name "macbook" } } })

Why would this be happening?

clojurewerkz.elastisch.rest.document/search-all-types and similar are not documented and thus cannot be easily discovered (see #2, for example)

Is there any option, how can i log all requests to elasticseach? I want to use it only in developement to see what my app generates.

When calling the connect! function with an known host name, it throws an exception. I guess I shouldn't have been surprised.

For each function in the API would you mind adding the possible exceptions that may occur, so that clients know what to trap?

Elastisch should support persistent HTTP connections for better performance with REST requests. The recent change in Elastisch to make connection and explicit argument for all actions should make this an easy change to implement.

The clojurewerkz.elastisch.rest/connect function could be updated to take a clj-http connection manager as detailed here: https://github.com/dakrone/clj-http#using-persistent-connections. The connection manager would be added as a new field on the clojurewerkz.elastisch.rest/Connection record. Any REST request to Elastic would have the instance of Connection available and would be able to specify the connection manager as an option to clj-http posts.

It looks like the only changes would be in the clojurewerkz.elastisch.rest namespace and would be fairly minimal.

AFAIK it's not possible to set timeouts for the rest-calls.

This would really be a great addition to the library.

Maybe I'm mistaken but I think the terms query is missing. Would you consider a patch adding it?

I am using native client to search documents with hightlight,but it doesn't return highlight value in hits response.Is it an issue in clojurewerkz, or in elasticsearch java client?

Again, a sizable task but we will use a single issue to track progress on it.

This is a pretty sizable task but I will use a single issue for it.

Running the following code against an Elasticsearch cluster that has multiple indices will create test-alias in every index, not just test-index-0, as I would expect.

(require '[clojurewerkz.elastisch.native :as es-native]
         '[clojurewerkz.elastisch.native.index :as es-native-index]
         '[clojurewerkz.elastisch.rest :as es-rest]
         '[clojurewerkz.elastisch.rest.index :as es-rest-index])

(def es-native-conn (es-native/connect [["localhost" 9300]] {"cluster.name" "elasticsearch"}))

(es-native-index/create es-native-conn "test-index-0")
(es-native-index/create es-native-conn "test-index-1")
(es-native-index/update-aliases es-native-conn [{:add {:index "test-index-0" :alias "test-alias"}}])

(es-rest-index/get-aliases (es-rest/connect "http://localhost:9200") "test-index-1")
; returns {:test-index-1 {:aliases {:test-alias {}}}}

I've cloned elastisch, and am trying to run the tests locally. I ran into a problem when running the tests. "lein all test" using leiningen2, and the below test give me the same result. Since the CI build is passing, it seems like this must be a problem with my local server. This is a clone of the master branch on github.

it smells like a local problem, but I'm not sure the best place to look. any ideas?

$ lein -version
Leiningen 2.0.0-preview10 on Java 1.7.0_04 Java HotSpot(TM) 64-Bit Server VM

Here's the test that fails, and the stacktrace:

$ lein test clojurewerkz.elastisch.rest-api.count-test
Reflection warning, clj_http/core.clj:110 - reference to field close can't be resolved.
Reflection warning, clj_http/core.clj:220 - call to org.apache.http.entity.StringEntity ctor can't be resolved.
Reflection warning, clj_http/core.clj:244 - call to writeTo can't be resolved.
Reflection warning, cheshire/generate.clj:84 - call to writeNumber can't be resolved.
Reflection warning, cheshire/generate.clj:84 - call to writeNumber can't be resolved.
Reflection warning, cheshire/generate.clj:84 - call to writeNumber can't be resolved.
Reflection warning, cheshire/generate.clj:104 - call to writeNumber can't be resolved.
Reflection warning, cheshire/custom.clj:133 - call to writeNumber can't be resolved.
Reflection warning, cheshire/custom.clj:212 - reference to field toBigInteger can't be resolved.

lein test clojurewerkz.elastisch.rest-api.count-test
Exception in thread "main" java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:151)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:125)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at clj_http.core$request.invoke(core.clj:223)
at clojure.lang.Var.invoke(Var.java:415)
at clj_http.client$wrap_request_timing$fn__929.invoke(client.clj:473)
at clj_http.client$wrap_lower_case_headers$fn__926.invoke(client.clj:464)
at clj_http.client$wrap_query_params$fn__871.invoke(client.clj:342)
at clj_http.client$wrap_basic_auth$fn__875.invoke(client.clj:358)
at clj_http.client$wrap_oauth$fn__879.invoke(client.clj:368)
at clj_http.client$wrap_user_info$fn__884.invoke(client.clj:381)
at clj_http.client$wrap_url$fn__914.invoke(client.clj:439)
at clj_http.client$wrap_redirects$fn__763.invoke(client.clj:127)
at clj_http.client$wrap_decompression$fn__788.invoke(client.clj:181)
at clj_http.client$wrap_input_coercion$fn__842.invoke(client.clj:276)
at clj_http.client$wrap_output_coercion$fn__838.invoke(client.clj:246)
at clj_http.client$wrap_exceptions$fn__749.invoke(client.clj:89)
at clj_http.client$wrap_accept$fn__853.invoke(client.clj:303)
at clj_http.client$wrap_accept_encoding$fn__858.invoke(client.clj:318)
at clj_http.client$wrap_content_type$fn__849.invoke(client.clj:294)
at clj_http.client$wrap_form_params$fn__894.invoke(client.clj:405)
at clj_http.client$wrap_nested_params$fn__911.invoke(client.clj:429)
at clj_http.client$wrap_method$fn__889.invoke(client.clj:388)
at clj_http.cookies$wrap_cookies$fn__89.invoke(cookies.clj:116)
at clj_http.cookies$wrap_cookie_store$fn__94.invoke(cookies.clj:141)
at clj_http.links$wrap_links$fn__126.invoke(links.clj:50)
at clj_http.client$wrap_unknown_host$fn__919.invoke(client.clj:448)
at clj_http.client$delete.doInvoke(client.clj:561)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at clojurewerkz.elastisch.rest$delete.invoke(rest.clj:39)
at clojurewerkz.elastisch.rest.index$delete.invoke(index.clj:51)
at clojurewerkz.elastisch.fixtures$reset_indexes_STAR_.invoke(fixtures.clj:11)
at clojurewerkz.elastisch.fixtures$reset_indexes.invoke(fixtures.clj:16)
at clojure.test$compose_fixtures$fn__6920$fn__6921.invoke(test.clj:678)
... snipped for brevity
Tests failed.

In rest.clj, record-url should url-encode (at least) the id parameter.

It might be that I'm still pretty new to Elastic Search, but I spent a decent amount of time debugging today because a query wasn't working and it turned out that I was passing :article (keyword) instead of "article" to (esd/search ...) as the mapping-type parameter.

In the docs for creating the index with a mapping here it shows passing the mapping-type like:

{:article ...}

Maybe just better clarification in the docs or a flushed out example in the document create section here would save someone else from the experience I had.

HTTP client already supports it.

_index and _type fields are defaultly included in index, but bulk index operation still add them into fields.

Not sure if this is by design, but "date" type fields are coerced to a string when getting a resource via the native client i.e (esd/get "my_index" "mapping" "id"). I see that an "integer" type field is coerced to an integer. What's the expected behavior?

(esn/connect! [["127.0.0.1" 9300]] {"cluster.name" "elasticsearch_xx"})

Always returns:

NoClassDefFoundError Could not initialize class org.elasticsearch.Version org.elasticsearch.cluster.node.DiscoveryNode. (DiscoveryNode.java:78)

I am running ES 0.90 from homebrew and have set ES_CLUSTER_NAME

Seems mapping does not filter results when using esd/count:

search> (count (:hits (:hits (esd/search index :places :query (q/match-all) :size 1000000))))
1311
search>  (esrsp/count-from (esd/count index :places (q/match-all)))
3420

curl -XGET http://localhost:9200/index/:places/_count
{"count":1311,"_shards":{"total":5,"successful":5,"failed":0}}

Hi,

It seems the elastic search supports to return part of fields of document for performance. Will it be supported?

Thanks

It looks like elastisch only allows use of a global connection to elastic. This breaks cases where an application would want to use multiple connections to elastic. The API functions should take an optional connection in cases where a global connection is not appropriate.

Hi,

Wondering why there is a difference between the way the client var is handled for rest vs native.

For rest the developer using elastisch has to manage the client object. For native is stored inside the client dynamic var.

The latter has issues, for example if a service/app wants to talk to two different ES instances this is unmanageable right? As a user of elastisch I also want to lifecycle manage the connection myself.

A change would probably be breaking to existing lib users, so I can see a downside in switching. Interested to know why the dynamic var approach was originally taken...

thanks,

Jon.

Hello,

I would like to upsert documents that have a parent. Right now
the upsert fn does not have any options to pass the parent. If
there's a interest I could send a PR for the upsert fn that
adds optional arguments in a similar way as the create fn. I
thought about something like this:

(defn upsert
  "Updates or creates a document using provided data"
  ([^Client conn index mapping-type ^String id ^Map doc & args]
     (let [{:keys [parent]} (ar/->opts args)
           req (cnv/->upsert-request index mapping-type id doc)
           req (if parent (.parent req parent) req)
           res (es/update conn req)]
       (cnv/update-response->map (.actionGet res)))))

Roman

New in ElasticSearch 1.0

Currently the format of hits differs between rest and native in regard to the "fields"-key: native uses ":_fields" and rest uses ":fields".

I propose both to use ":fields".

To reproduce that issue:

# I assume, you have an elasticsearch cluster named "mytestcluster" 
# available at 127.0.0.1 on port 9200 (rest-http) and 9300 (native-transport) 
# respectively.

# Steps to reproduce
git clone http://github.com/clojurewerkz/elastisch
cd elastisch
lein repl

(require '[clojurewerkz.elastisch.rest :as r]
         '[clojurewerkz.elastisch.rest.document :as rd]
         '[clojurewerkz.elastisch.rest.index :as ri]
         '[clojurewerkz.elastisch.native :as n]
         '[clojurewerkz.elastisch.native.document :as nd])

(r/connect! "http://127.0.0.1:9200")

(let [mapping-types {:person :properties {:username {:type "string" :store "yes"}}}]
    (ri/create "mytest" :mappings mapping-types))

(rd/create "mytest" "person" {:username "tester"})

(n/connect! [["127.0.0.1" 9300]]
            {"cluster.name" "mytestcluster"})


(def resttest (keys
                (first
                  (:hits
                    (:hits
                      (rd/search-all-indexes-and-types
                        :fields ["username"]
                        :query {:match_all {}}))))))

(def nativetest (keys
                (first
                  (:hits
                    (:hits
                      (nd/search-all-indexes-and-types
                        :fields ["username"]
                        :query {:match_all {}}))))))

(println resttest nativetest)

;; (:_index :_type :_id :_score :fields) (:_version :_score :_id :_type :_index :_fields)
;; nil
;;
;; Should be:
;; (:_index :_type :_id :_score :fields) (:_version :_score :_id :_type :_index :fields)
;;

Change https://github.com/clojurewerkz/elastisch/blob/master/src/clojurewerkz/elastisch/native/conversion.clj#L535

- {:_fields (convert-fields-result (.getFields sh))})]
+ {:fields (convert-fields-result (.getFields sh))})]

PS: I was about to write a complete test-case and pull-request accordingly, but since I couldn't get the tests to run, I hope this is sufficient :)

Hi,

I got a problem using the native api to put and get document, for example, the document put into index is {:a :a-value(keyword type) :b 1111(long type)}, the document becomes {:a :a-value(string type) :b 1111(integer type)} after retrieving from index.

I am not use if it is correct to change the type between the put and get?

Thanks!

The function clojurewerkz.elastisch.rest.document/present? sometime throws a JsonParseException. I suspect this is a bug in elasticsearch itself, but I thought I'd mention it here, just in case.

Here's a partial stack trace.

com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'No': was expecting 'null', 'true', 'false' or NaN
 at [Source: java.io.StringReader@7f7a6b96; line: 1, column: 3]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1524)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:557)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2042)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2018)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._matchToken(ReaderBasedJsonParser.java:1870)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1389)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:679)
        at cheshire.parse$parse.invoke(parse.clj:87)
        at cheshire.core$parse_string.invoke(core.clj:92)
        at cheshire.core$parse_string.invoke(core.clj:88)
        at clojurewerkz.elastisch.rest$get.doInvoke(rest.clj:49)
        at clojure.lang.RestFn.invoke(RestFn.java:439)
        at clojurewerkz.elastisch.rest.document$get.doInvoke(document.clj:85)
        at clojure.lang.RestFn.invoke(RestFn.java:470)
        at clojurewerkz.elastisch.rest.document$present_QMARK_.invoke(document.clj:103)

I'm trying out elastisch and ran into the following problem

   => (document/search :query (query/term :foo "bar")))
     URISyntaxException Illegal character in path at index 29: http://localhost:9200/:query/{:term {:foo "bar"}}/_search java.net.URI$Parser.fail (URI.java:2809)
     RuntimeException Unmatched delimiter: )  clojure.lang.Util.runtimeException (Util.java:156)
   => 

Looks like elastisch generates and invalid url. Any idea why?

Thanks for you help.

Hi there,
Any plans to support bulk requests http://www.elasticsearch.org/guide/reference/api/bulk.html?
In particular bulk inserts.

Thanks

Issuing a delete using the native API to an index that does not already exist causes the index to be created in ElasticSearch.

Start with an empty ElasticSearch instance

$ curl localhost:9200/_cat/indices?v
health index pri rep docs.count docs.deleted store.size pri.store.size

Then try and delete a document using the native api

$ lein repl
elasticsearch-loader.core=> (require '[clojurewerkz.elastisch.native :as es] '[clojurewerkz.elastisch.native.document :as es-document])
elasticsearch-loader.core=> (def es-conn (es/connect [[(env :elasticsearch-host) (env :elasticsearch-port)]] {"cluster.name" (env :elasticsearch-cluster-name)}))
elasticsearch-loader.core=> (es-document/delete es-conn "123" "foo" "1")
{:found false, :found? false, :_index "123", :_type "foo", :_version 1, :_id "1"}

Then check ElasticSearch one more time and you'll see an index was created.

$ curl localhost:9200/_cat/indices?v
health index pri rep docs.count docs.deleted store.size pri.store.size
green 123 1 0 0 0 123b 123b

Using the ElasticSearch Rest API this same thing throws an error saying the index does not exist and probably should do the same here.

elastisch.native.index/create works fine with string keys:

 (esi/create index-name
              :mappings mapping-types
              :settings {"index" {"refresh_interval"  "30s"
                                  "warmer"            {"enabled" "false"}}}))

But the docs suggest using keywords, like so:

  (esi/create index-name
              :mappings mapping-types
              :settings {:index {:refresh_interval  "30s"
                                 :warmer            {:enabled false}}}))

Which throws ClassCastException from native.conversion/->create-index-request

ClassCastException clojure.lang.Keyword cannot be cast to java.lang.String
    org.elasticsearch.common.xcontent.XContentBuilder.writeMap (XContentBuilder.java:1095)
    org.elasticsearch.common.xcontent.XContentBuilder.map (XContentBuilder.java:1015)
    org.elasticsearch.action.admin.indices.create.CreateIndexRequest.settings (CreateIndexRequest.java:175)
    clojurewerkz.elastisch.native.conversion/->create-index-request (conversion.clj:758)
    clojurewerkz.elastisch.native.index/create (index.clj:53)
    gauntlet.elasticsearch/setup-es! (elasticsearch.clj:82)
    gauntlet.elasticsearch/bench!/fn--4961 (elasticsearch.clj:116)
    gauntlet.elasticsearch/bench! (elasticsearch.clj:116)
    user/eval4978 (form-init109469628037641998.clj:1)
    clojure.lang.Compiler.eval (Compiler.java:6619)
    clojure.lang.Compiler.eval (Compiler.java:6582)
    clojure.core/eval (core.clj:2852)

Incorrect docs? Should we change the code?

Hello,

In our app, we escape special characters from user input which is used in a query_string query. Recently, we've been converting our app to use Elastisch and thought it made sense that the code to do so would live in here.

Here is what our current function looks like (which is based on lucene special characters):

(defn escape-special-chars [s]
  (->
    (clojure.string/replace s #"[*\-+!(){}\[\]^\"~?:\\]" #(str "\\" %1))
    (clojure.string/replace #"(&&|\|\|)" #(str "\\" (%1 1)))))

I'd be happy to open a PR if the maintainers agree that this should live in here.

We need to support bulk operations in the native client with the same Clojure API we
have in the REST one.

When we test using our home grown elasticsearch library, we avoid having to boot a separate elasticsearch instance by booting an in-process, local, node. Especially given the native client tests require the elasticsearch library to be pulled down, it seems like it would simplify life for contributors to do this in elastisch.

It does cause a bit of overhead of course when booting up the test suite, since you have to wait for ES to boot, but for us we've felt it was worth it due to the simplification. For elastisch, I also assume by doing this the full set of native tests could be ran in travis.

Now that we're switching over to elastisch, we still kept our local node booter running, but are just doing it on top of elastisch. Our code is mostly:

; This is like the es-native/build-local-node method, but it doesn't force client true.
; When we're building an in-memory node, we don't want it to be a client, it is the server.
(defn build-local-node
  [settings]
  (let [is (cnv/->settings settings)
        nb (.. NodeBuilder nodeBuilder
               (settings is))]
    (.build ^NodeBuilder nb)))

(defn- build-in-memory-node [cluster-settings]
  (-> (build-local-node (assoc cluster-settings "node.local" true))
      es-native/start-local-node))

(defn configure! [configuration]
  (if (configuration :local)
    (es-native/connect-to-local-node! (build-in-memory-node {"cluster.name" "local"}))
    ; stuff that we do in prod/dev/... here
    ))

(defn close []
  (when-not (nil? es-native/*client*)
    (locking es-native/*client*
      (when-let [client es-native/*client*]
        (.close client)
        (alter-var-root #'es-native/*client* (constantly nil))))))

with a set of fixtures that make sure things are booted when the tests start, and close the node down when the tests are done.

So there's 2 proposals in this issue. Proposal 1 is that elastich's test suite use a variant of this instead of requiring a separately booted node. Proposal 2 is that elastisch expose this publicly, so that people can re-use it in their own tests.

Relatedly, we'd probably also expose a method to unset the es-native/client var, since right now if you want to do that you must do an ugly alter-var-root. Also, we'd probably expose a close method in the es-native namespace, which would close down the client (and probably set client to nil).

Again, we have the code already written, so we can send a PR for this, but didn't want to do so until we got agreement on the idea. In particular, we'll have to play around a bit to see how the REST api would function with a local node (I believe it should work, but haven't tested it).

PS, the "we" here is https://engineering.groupon.com, and that's where the issues/PRs that are coming from me and @benjreinhart are coming from

I have a project that works fine using elastisch 1.0.2 and datomic-free 0.8.3848 against an elasticsearch 0.90.0.RC1 server (since 0.20.x and 0.90.0.Beta).

Datomic currently seems to depend on Lucene 3.x API, and specifically includes lucene-core 3.3.0.

lein classpath only shows lucene-core 3.3.0, no other lucene or elasticsearch jars.

I tried switching to elastisch 1.1.0-beta2 to see if is was compatible. It pulls down ES 0.90.0.Beta1 and Lucene 4.1. Now Datomic is having problems and the project won't start. Also, lein classpath shows lots of lucene jars and the full elasticsearch jar.

Is this just beta related, or will the native client in elastisch depend on all these jars?
Will there be a method to avoid pulling in lucene/elasticsearch jars at all and maintain Datomic compatibility until they upgrade to 4.x?

I tested it against 1.2.0 and 1.1.2, all tests pass.

see http://www.elasticsearch.org/guide/reference/api/multi-search.html

I don't think you support it yet (I didn't see it in the docs).
It uses the same format as bulk, so maybe it's something to think about when implementing #11

In one of our apps, we've built our own elastic search client, that we've recently been attempting to replace with elastisch. Overall this has gone pretty well, but we've had a couple issues.

The most recent one is we are handling some errors elastic search throws in our code, and seeing different behavior now that we have moved to elastisch. After digging through things a bit, it seems to be caused by elastisch doing:

 (.get response)

versus

 (.actionGet response)

While this normally isn't noticeable, the primary reason actionGet exists is to wrap the exceptions that happen, and make sure elasticsearch variants get thrown, so for those of us that care about the exceptions, it'd be preferential to use actionGet.

This is also why most examples on the web are using actionGet.

Unless there was a specific reason elastisch uses get, I'd be happy to send a PR fixing this, but wanted to check first.

For reference, actionGet is implemented in https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/support/AdapterActionFuture.java

@Override
public T actionGet() throws ElasticsearchException {
    try {
        return get();
    } catch (InterruptedException e) {
        throw new ElasticsearchInterruptedException(e.getMessage());
    } catch (ExecutionException e) {
        throw rethrowExecutionException(e);
    }
}

Related to #54 another issue we're running into is elastisch doesn't seem to support using the optimistic locking built into elasticsearch. Specifically, on the index-request, one should be able to trigger a setVersion call.

Docs on setVersion: https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/index/IndexRequestBuilder.java#L281

This should probably just be another when in https://github.com/clojurewerkz/elastisch/blob/master/src/clojurewerkz/elastisch/native/conversion.clj#L148

(when version
  (.version version)    

I'm using elasticsearch 0.90.7, so this may be a temporary problem.

A scroll search with a scan search type does not return results with the initial call. A call to the _search/scroll endpoint needs to be made to retrieve the first set of results. This isn't consistent with the behavior of the other search types.

The otherwise very useful scroll-seq endpoint doesn't know about this behavioral inconsistency. When naively called like the following, it assumes the lack of matches returned in the initial search call means there are no matches. scroll-seq returns an empty list.

(scroll-seq (search "index" "mapping" :query (match-all) :search_type "scan"))

There is a work around, make a single scroll call and pass that result set to scroll-sea, but it would be neat if scroll-seq could hide the need to do that for scan search types.