Elasticsearch(versions 6.x and 7.x) datasource connector for Loopback 3.x.
libdirectory has the entire source code for this connector- this is what gets downloaded to your
node_modulesfolder when you runnpm install loopback-connector-esv6 --save --save-exact
- this is what gets downloaded to your
examplesdirectory has a loopback app which uses this connector- this is not published to NPM, it is only here for demo purposes
- it will not be downloaded to your
node_modulesfolder! - similarly the
examples/server/datasources.jsonfile is there for this demo app to use - you can copy their content over to
<yourApp>/server/datasources.jsonor<yourApp>/server/datasources.<env>.jsif you want and edit it there but don't start editing the files insideexamples/serveritself and expect changes to take place in your app!
- it will not be downloaded to your
- this is not published to NPM, it is only here for demo purposes
testdirectory has unit tests- it does not reuse the loopback app from the
examplesfolder - instead, loopback and ES/datasource are built and injected programatically
- this directory is not published to NPM.
- Refer to
.npmignoreif you're still confused about what's part of the published connector and what's not.
- Refer to
- it does not reuse the loopback app from the
- You will find the
datasources.jsonfiles in this repo mention various configurations:elasticsearch-sslelasticsearch-plaindb- You don't need them all! They are just examples to help you see the various ways in which you can configure a datasource. Delete the ones you don't need and keep the one you want. For example, most people will start off with
elasticsearch-plainand then move on to configuring the additional properties that are exemplified inelasticsearch-ssl. You can mix & match if you'd like to have mongo and es and memory, all three! These are basics of the "connector" framework in loooback and not something we added.
- Don't forget to edit your
model-config.jsonfile and point the models at thedataSourceyou want to use.
cd <yourApp>
npm install loopback-connector-esv6 --save --save-exact- This connector will only connect to one index per datasource.
- This package is created to support ElasticSearch v6.x and 7.x only.
docTypeproperty is automatically added in mapping properties which is required to differentiate documents stored in index with loopback model data. It stores loopback modelName value.docType: { type: "keyword", index: true }
- name: name of the connector.
- connector: Elasticsearch driver 'esv6'.
- configuration: Elasticsearch client configuraiton object which includes nodes, authetication and ssl coonfiguration. Please refer this official link for more information on configuraiton.
- index: Name of the ElasticSearch index
eg: shakespeare. - version: specify the major version of the Elasticsearch nodes you will be connecting to. Supported versions: [6, 7]
eg: version: 7 - mappingType: mapping type for provided index. defaults to
basedata. Required only for version: 6 - mappingProperties: An object with properties for above mentioned
mappingType
- indexSettings: optional settings object for creating index.
- defaultSize: Search size limit. Default is 50.
1.Edit datasources.json and set:
"elastic-search-ssl": {
"name": "elasticsearch-example-index-datasource",
"connector": "esv6",
"version": 7,
"index": "example-index",
"configuration": { // Elastic client configuration
"node": "http://localhost:9200",
"requestTimeout": 30000,
"pingTimeout": 3000,
"auth": {
"username": "test",
"password": "test"
},
"ssl": {
"rejectUnauthorized": true
}
},
"defaultSize": 50,
"indexSettings": { // Elastic index settings
"number_of_shards": 2,
"number_of_replicas": 1
},
"mappingType": "basedata", // not required for verison: 7, will be ignored
"mappingProperties": {
"docType": {
"type": "keyword",
"index": true
},
"id": {
"type": "keyword",
"index": true
},
"seq": {
"type": "integer",
"index": true
},
"name": {
"type": "keyword",
"index": true
},
"email": {
"type": "keyword",
"index": true
},
"birthday": {
"type": "date",
"index": true
},
"role": {
"type": "keyword",
"index": true
},
"order": {
"type": "integer",
"index": true
},
"vip": {
"type": "boolean",
"index": true
},
"objectId": {
"type": "keyword",
"index": true
},
"ttl": {
"type": "integer",
"index": true
},
"created": {
"type": "date",
"index": true
}
}
}2.You can peek at /examples/server/datasources.json for more hints.
search_afterfeature of elasticsearch is supported in loopback filter.- For this, you need to create a property in model called
_search_afterwith loopback type["any"]. This field cannot be updated using in connector. - Elasticsearch
sortvalue will return in this field. - You need pass
_search_aftervalue insearchafterkey of loopback filter. - Example filter query for
find.
{
"where": {
"username": "hello"
},
"order": "created DESC",
"searchafter": [
1580902552905
],
"limit": 4
}- Example result.
[
{
"id": "1bb2dd63-c7b9-588e-a942-15ca4f891a80",
"username": "test",
"_search_after": [
1580902552905
],
"created": "2020-02-05T11:35:52.905Z"
},
{
"id": "fd5ea4df-f159-5816-9104-22147f2a740f",
"username": "test3",
"_search_after": [
1580902552901
],
"created": "2020-02-05T11:35:52.901Z"
},
{
"id": "047c0adb-13ea-5f80-a772-3d2a4691d47a",
"username": "test4",
"_search_after": [
1580902552897
],
"created": "2020-02-05T11:35:52.897Z"
}
]- This is useful for pagination. To go to previous page, change sorting order.
totalvalue from elasticsearch for search queries is now supported in loopback response.- For this, you need to create a property in model called
_total_countwith loopback type"number". This field cannot be updated using in connector. - Example response
find.
[
{
"id": "1bb2dd63-c7b9-588e-a942-15ca4f891a80",
"username": "test",
"_search_after": [
1580902552905
],
"_total_count": 3,
"created": "2020-02-05T11:35:52.905Z"
},
{
"id": "fd5ea4df-f159-5816-9104-22147f2a740f",
"username": "test3",
"_search_after": [
1580902552901
],
"_total_count": 3,
"created": "2020-02-05T11:35:52.901Z"
},
{
"id": "047c0adb-13ea-5f80-a772-3d2a4691d47a",
"username": "test4",
"_search_after": [
1580902552897
],
"_total_count": 3,
"created": "2020-02-05T11:35:52.897Z"
}
]- The
examplesdirectory contains a loopback app which uses this connector. - You can point this example at your own elasticsearch instance or use the quick instances provided via docker.
- Do you have both
elasticsearch-sslandelasticsearch-plainin yourdatasources.jsonfile? You just need one of them (not both), based on how you've setup your ES instance. - Did you forget to set
model-config.jsonto point at the datasource you configured? Maybe you are using a different or misspelled name than what you thought you had! - Make sure to configure major version of Elastic in
version - Maybe the version of ES you are using isn't supported by the client that this project uses. Try removing the
elasticsearchsub-dependency from<yourApp>/node_modules/loopback-connector-esv6/node_modulesfolder and then install the latest client:cd <yourApp>/node_modules/loopback-connector-esv6/node_modules- then remove
es6&&es7folder- unix/mac quickie:
rm -rf es6 es7
- unix/mac quickie:
npm install- go back to yourApp's root directory
- unix/mac quickie:
cd <yourApp>
- unix/mac quickie:
- And test that you can now use the connector without any issues!
- These changes can easily get washed away for several reasons. So for a more permanent fix that adds the version you want to work on into a release of this connector, please look into Contributing.
- Feel free to contribute via PR or open an issue for discussion or jump into the gitter chat room if you have ideas.
- I recommend that project contributors who are part of the team:
- should merge
masterintodevelop... if they are behind, before starting thefeaturebranch - should create
featurebranches from thedevelopbranch - should merge
featureintodevelopthen create areleasebranch to:- update the changelog
- close related issues and mention release version
- update the readme
- fix any bugs from final testing
- commit locally and run
npm-release x.x.x -m "<some comment>" - merge
releaseinto bothmasteranddevelop - push
masteranddevelopto GitHub
- should merge
- For those who use forks:
- please submit your PR against the
developbranch, if possible - if you must submit your PR against the
masterbranch ... I understand and I can't stop you. I only hope that there is a good reason likedevelopnot being up-to-date withmasterfor the work you want to build upon.
- please submit your PR against the
npm-release <versionNumber> -m <commit message>may be used to publish. Pubilshing to NPM should happen from themasterbranch. It should ideally only happen when there is something release worthy. There's no point in publishing just because of changes totestorexamplesfolder or any other such entities that aren't part of the "published module" (refer to.npmignore) to begin with.
- How do we enable or disable the logs coming from the underlying elasticsearch client? There may be a need to debug/troubleshoot at times.
- Use the env variable
DEBUG=elasticsearchfor elastic client logs.
- Use the env variable
- How do we enable or disable the logs coming from this connector?
- By default if you do not set the following env variable, they are disabled:
DEBUG=loopback:connector:elasticsearch
- By default if you do not set the following env variable, they are disabled:
- What are the tests about? Can you provide a brief overview?
- Tests are prefixed with
01or02etc. in order to run them in that order by leveraging default alphabetical sorting. - The
02.basic-querying.test.jsfile uses two models to test various CRUD operations that any connector must provide, likefind(), findById(), findByIds(), updateAttributes()etc.- the two models are
UserandCustomer - their ES mappings are laid out in
test/resource/datasource-test.json - their loopback definitions can be found in the first
beforeblock that performs setup in02.basic-querying.test.jsfile ... these are the equivalent of aMyModel.jsonin your real loopback app.- naturally, this is also where we define which property serves as the
idfor the model and if its generated or not
- naturally, this is also where we define which property serves as the
- the two models are
- Tests are prefixed with
- How do we get elasticserch to take over ID generation?
- An automatically generated id-like field that is maintained by ES is
_id. Without some sort of es-field-level-scripting-on-index (if that is possible at all) ... I am not sure how we could ask elasticsearch to take over auto-generating an id-like value for any arbitrary field! So the connector is setup such that addingid: {type: String, generated: true, id: true}will tell it to use_idas the actual field backing theid... you can keep using the doingmodel.idabstraction and in the background_idvalues are mapped to it. - Will this work for any field marked as with
generated: trueandid: true?- No! The connector isn't coded that way right now ... while it is an interesting idea to couple any such field with ES's
_idfield inside this connector ... I am not sure if this is the right thing to do. If you hadobjectId: {type: String, generated: true, id: true}then you won't find a realobjectIdfield in your ES documents. Would that be ok? Wouldn't that confuse developers who want to write custom queries and run 3rd party app against their ES instance? Don't useobjectId, use_idwould have to be common knowledge. Is that ok?
- No! The connector isn't coded that way right now ... while it is an interesting idea to couple any such field with ES's
- An automatically generated id-like field that is maintained by ES is
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent