cakephp / elastic-search Goto Github PK

Hi guys im using the plugin "cakephp/elastic-search": "dev-master" with version 5.5.1 of elastic search, Im install elastic search with home brew on mac and I made all configurations and installations according to the tutorial of cookbook, However, when i try mapping my entity "Categorias" i receive one wrong message with "Wrong parameters for Elastica\Exception\ResponseException". Then I went to check if the elastic was creating the "my_apps_index" in http://localhost:9200/my_apps_index and i receive the message " index_not_found_exception status 404", i want know how i can solve this problem. Can someone help me?

Best regards,
Marcos dornellas

Below are the photos with the problems.

Plugin::load ( 'ElasticSearch', [ 'bootstrap' => true, 'autoload' => true ] );

gives

Error: The application is trying to load a file from the ElasticSearch plugin. 

Make sure your plugin ElasticSearch is in the /app-path/plugins/ directory and was loaded.

Hi, I have this code in my controller:

public function formStupido()
{
        $Companies = TypeRegistry::get('Companies.Companies');
        $companies = $Companies->newEntity(['name' => 'ciao']);
        $savedCompany = $Companies->save($companies);
}

This is my CompaniesType in plugin Companies:

<?php
namespace Companies\Model\Type;

use Cake\ElasticSearch\Type;
use Cake\ElasticSearch\Query;
use Cake\ElasticSearch\FilterBuilder;
use App\Lib\FileUpload;

class CompaniesType extends Type
{
    /**
     * initialize method
     * 
     * @param array $config
     */
    public function initialize(array $config)
    {

    }

    public function beforeMarshal(Event $event, ArrayObject $data, ArrayObject $options)
    {
        debug('marshaller');
        exit();
    }

    public function beforeSave(Event $event, ArrayObject $data, ArrayObject $options)
    {
        debug('before');
        exit();
    }
}
?>

Document is created and no callbacks are called

Hello,
I should use match_phrase_prefix (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html)

it seems not implemented here, at the moment i am testing the query with this json:

{  
   "fields":[  
      "name"
   ],
   "query":{  
      "match_phrase_prefix":{  
         "name":{  
            "query":"test",
            "slop":3
         }
      }
   }
}

how should i build this query? Thanks!

It seems the controller does not exist yet when 'Dispatcher.beforeDispatch' is dispatched, resulting in the Elastic Type Registry not being registered.

See also https://github.com/cakephp/cakephp/blob/3.3.6/src/Http/ActionDispatcher.php#L81: here you can see the controller is only created later on.

For now I have fixed this in my own bootstrap.php by binding on Dispatcher.invokeController and doing the exact same code, but this doesn't seem the way to go for me. I think something should be fixed, either by doing this binding on Dispatcher.invokeController in this plugin or by doing something different, like creating the controller in the Dispatcher.beforeDispatch call. Both options sound ugly to me, but maybe someone can think of a better way.

I have two questions:

1. How can i get the max-score?
2. how can i configure a language analyzer like this: https://www.elastic.co/guide/en/elasticsearch/reference/1.4/analysis-lang-analyzer.html

Shouldn't the project be in packagist?

If not, maybe composer.json should be linking to github.

Please see this issue ruflin/Elastica#1016 and this PR ruflin/Elastica#1016

We're currently running into it and need to upgrade to 3.0.1 or convince them to do a 2.3.x release of Elastica. The plugin is nailing the versions down to ~2.2 right now.

The latest version of ES doesn't require a delete_by_query plugin the functionality is now in core and the query has changed from DELETE /type/query to POST /type/_delete_by_query

ruflin/elastica 5.0.0 has this change

From @ankr on January 15, 2016 11:43

This is throws a fatal error, "undefined method table", when using the ElasticSearch plugin.

https://github.com/cakephp/cakephp/blob/master/src/Datasource/QueryTrait.php#L412

Copied from original issue: cakephp/cakephp#8040

Hello,
i need to use a custom filter inside the sort, is this supported?
This is a piece of my query:

   "sort" : [
       {
          "applications.companies.date" : {
             "mode" :  "max",
             "order" : "desc",
             "nested_path" : "applications.companies",
             "nested_filter" : {
                "term" : { "applications.companies.id" : 41 }
             }
          }
       }
   ]

ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html

If it is not supported, any workaround?
Thanks!

Add the ability to apply validators to nested properties. This may require changes in Cake\Validation to allow this to work.

Hello,
I am tring to get this result:

{
    "query": {
        "filtered": {            
            "filter": {
                "bool": {
                    "must": [
                        {
                            "exists": {
                                "field": "telephone"
                            }
                        },
                        {
                            "exists": {
                                "field": "cv.path"
                            }
                        }
                    ]
                }
            }
        }
    }
}

using this code

         $Type->find()
                ->where(function (FilterBuilder $builder) {
                    return $builder->and(
                        $builder->exists('telephone'),
                        $builder->exists('cv.path')
                    );
                });

the result is:

{  
   "query":{  
      "filtered":{  
         "filter":{  
            "bool":{  
               "must":[  
                  {  
                     "bool":{  
                        "must":[  
                           {  
                              "exists":{  
                                 "field":"telephone"
                              }
                           },
                           {  
                              "exists":{  
                                 "field":"cv.path"
                              }
                           }
                        ]
                     }
                  }
               ]
            }
         }
      }
   }
}

is my code wrong?

Hello,
I have a function where I need to return all the items, using a where clause ,

$return = $this->Mapas->find()->where([
            'estado_id' => $estadoId,
            'municipio_id' => $municipioId,
        ]);

problem,
The function is returning only 10 items , instead of the 46 I need.
how to proceed?

It will be some sort of problems with the paginate or something?
Note: I do not need paginate.
NOTE: If I add for example the limit , (->limit(1000)) , he is returning the 46 items correctly.
thank you so much

As described here you can search through all types like this:

/_search

This can be done using the plugin by getting any type:

$type = TypeRegistry::get('Profiles');
$type->name('');
$results = $type->find()->all();
debug($resutls);

The problem with that is now that all of the documents are of the type that belongs to the type class instance: Profiles.

But the elastic search result set contains the type information. I think the plugin should map the type based on that information to the correct document class.

How can I implement the suggest feature of ES the best using the plugin? Any guidance on this please?

https://www.elastic.co/guide/en/elasticsearch/reference/1.7/search-suggesters-completion.html

Is it expected to get an exception when i try to fetch a non-existing doc?

public function indexToEs(EntityInterface $entity)
    {
        $entity_clone = clone $entity;
        unset($entity_clone->body);

        $items_table = TypeRegistry::get('Items');
        $found = $items_table->get($entity->id); //exception thrown

        if ($found) {
           $indexed = $items_table->patchEntity($found, $entity_clone->toArray());
        } else {
            $indexed = $items_table->newEntity($entity_clone);
        }

        if (!$items_table->save($indexed)) {
            var_dump($entity_clone->getErrors());
        }
    }

I have particular usecase that I have to fetch all the IDs from the elasticsearch and store it in file. I cannot do that (over 200k results) using normal result window aka "result paging" and I need scroll API for this. I have found out a workaround to that, but it is bypassing ORMs hydration and stuff and I cannot for example use map reduction in "cake fashion" . Here is what I have did:

       $filtered = $this->createFilteredQuery($data);
        $search = $myidxType->connection()->getIndex("myidx")->createSearch($filtered);
        $search->setOption("size","10000");
        $search->getQuery()->setFields(["_id"]);
        $scroll = $search->addType("myidx")->scroll('10s');
        $ids = [];
        do {
            $scroll->next();
            $results = $scroll->current();
//            $part = $results->reduce(function ($ids, $element) {
//                $ids[] = $element->id;
//                return $ids;
//            }, []);
//            $ids = array_merge($ids, $part);
        } while ($results->count() > 0);

It would be nice to expose some sort of iterator that would use scrollAPI under the hood.

How can I search documents without specifying a column i.e.

 _search?q=Foo

I tried

$this->Businesses->find('all')->where(['*'=>'Foo']);

Thanks

Hi guys i'm have a problem with amount of record that is appearing in my list, by default 10 records are appearing, I want know how i can increase this setting, someone can help me? Thanks.

There is not place where Cake\ElasticSearch\Datasource\Connection#logQueries is set to the value of $config['log'] causing no way to enable request logging.One way would be to add proper conditional in the contructor as it is done for 'name'

    public function __construct(array $config = [], $callback = null)
    {
        $config += ['index' => '_all'];
        if (isset($config['name'])) {
            $this->configName = $config['name'];
        }
        parent::__construct($config, $callback);
    }

Hi guys, how can i truncate all documents from type? deleteAll() is requiring me for a condition, I wanted the condition be like match_all: {}

Thanks

When i do ->all() on a query object, it returns only first 10 record found.

There is a way to avoid this without forcing with ->limit() ?

With this command
bin/cake bake fixture Agents --table agents -c elastic -p Agents --schema -v

I get this error:

Exception: Call to undefined method Cake\ElasticSearch\Datasource\SchemaCollection::describe() in [/home/dario/www/dario.local/vendor/cakephp/bake/src/Shell/Task/FixtureTask.php, line 178]
2017-01-25 14:39:30 Error: [Error] Call to undefined method Cake\ElasticSearch\Datasource\SchemaCollection::describe()
Stack Trace:
#0 /home/dario/www/dario.local/vendor/cakephp/bake/src/Shell/Task/FixtureTask.php(123): Bake\Shell\Task\FixtureTask->bake('Agents', 'agents')
#1 [internal function]: Bake\Shell\Task\FixtureTask->main('Agents')
#2 /home/dario/www/dario.local/vendor/cakephp/cakephp/src/Console/Shell.php(472): call_user_func_array(Array, Array)
#3 /home/dario/www/dario.local/vendor/cakephp/cakephp/src/Console/Shell.php(465): Cake\Console\Shell->runCommand(Array, false, Array)
#4 /home/dario/www/dario.local/vendor/cakephp/cakephp/src/Console/ShellDispatcher.php(227): Cake\Console\Shell->runCommand(Array, true, Array)
#5 /home/dario/www/dario.local/vendor/cakephp/cakephp/src/Console/ShellDispatcher.php(182): Cake\Console\ShellDispatcher->_dispatch(Array)
#6 /home/dario/www/dario.local/vendor/cakephp/cakephp/src/Console/ShellDispatcher.php(128): Cake\Console\ShellDispatcher->dispatch(Array)
#7 /home/dario/www/dario.local/bin/cake.php(20): Cake\Console\ShellDispatcher::run(Array)
#8 {main}

I'm using Elasticsearch 2.3.1, CakePhp 3.3.12 and cakephp/elastic-search 1.0.x-dev

• Is there any way to get the underlying Elastica instance/query somehow?
• How can I influence / build my own mapping? Looks like MappingSchema is doing it automatically, but how can I override this, for example when I want to use longitude latitude? AFAIR this can be it's own type
• A few examples for the query builder would be awesome. The best something that reassembles the examples from the elastic documentation.

How can I read highlight key in the document when I iterate resultSet ? It is outside the _source at the moment

I want to filter result by score.

Add domain rules similar to how the relational ORM implements them.

Ideally we'll share as much code as possible with the relational ORM. This may require some refactoring in the relational ORM as well.

Hello,
it seems that we cannot attach behavior to a type, is this an issue or does it has not been implemented yet?

I had a problem about pagination in elastic . I try to send a parameter "?page = 2" when using mysql running normally . But when using elastic it does not work .

{
    "message": "Not Found",
    "url": "\/myapp\/api\/categories.json?limit=3\u0026amp;page=2",
    "code": 404,
    "trace": [
        {
            "file": "\/var\/www\/html\/myapp\/vendor\/cakephp\/cakephp\/src\/Controller\/Controller.php",
            "line": 688,
            "function": "paginate",
            "class": "Cake\\Controller\\Component\\PaginatorComponent",
            "type": "-\u003E",
            "args": [
                {

                },
                {
                    "limit": 3
                }
            ]
        },
        {`

I'm using ES2.3.4 and branch 1.0 and during the marshalling process the values passed to the __construct seem incorrect.

_GET /library/apps/search

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "library",
        "_type": "apps",
        "_id": "AVYJCVkoBgZSBgn8x1jS",
        "_score": 1,
        "_source": {
          "title": "this title",
          "questions": [
            {
              "title": "question1"
            },
            {
              "title": "question2"
            }
          ]
        }
      }
    ]
  }
}

Now when I do the following in my controller

        $apps = TypeRegistry::get('Apps')
          ->get('AVYJCVkoBgZSBgn8x1jS');
        debug($apps);

I get the following

/src/Controller/Admin/InstanceController.php (line 64)
object(App\Model\Document\App) {
    [protected] _data => [
        'markNew' => false,
        'markClean' => true,
        'useSetters' => false,
        'source' => 'apps'
    ]
    [protected] _docAsUpsert => false
    [protected] _autoPopulate => false
    [protected] _upsert => null
    [protected] _params => [
        '_id' => [
            'title' => 'this title',
            'questions' => [
                (int) 0 => object(Cake\ElasticSearch\Document) {

                    'title' => 'question1',
                    '[new]' => false,
                    '[accessible]' => [
                        '*' => true
                    ],
                    '[dirty]' => [],
                    '[original]' => [],
                    '[virtual]' => [],
                    '[errors]' => [],
                    '[invalid]' => [],
                    '[repository]' => 'apps'

                },
                (int) 1 => object(Cake\ElasticSearch\Document) {

                    'title' => 'question2',
                    '[new]' => false,
                    '[accessible]' => [
                        '*' => true
                    ],
                    '[dirty]' => [],
                    '[original]' => [],
                    '[virtual]' => [],
                    '[errors]' => [],
                    '[invalid]' => [],
                    '[repository]' => 'apps'

                }
            ],
            'id' => 'AVYJCVkoBgZSBgn8x1jS'
        ],
        '_type' => '',
        '_index' => ''
    ]
    [protected] _rawParams => []
}

Doing $apps->title and $apps->questions result in

Field title does not exist
Elastica\Exception\InvalidException

On inspection of the Document __construct $id and $data seems to being passed incorrectly as

class App extends Document {

    public function __construct($id = '', $data = array(), $type = '', $index = '') {
        parent::__construct($id, $data, $type, $index);
        debug($id);
    }
}

outputs the following (what seems like data rather than id)

[
    'title' => 'this title',
    'questions' => [
        (int) 0 => object(Cake\ElasticSearch\Document) {

            'title' => 'question1',
            '[new]' => false,
            '[accessible]' => [
                '*' => true
            ],
            '[dirty]' => [],
            '[original]' => [],
            '[virtual]' => [],
            '[errors]' => [],
            '[invalid]' => [],
            '[repository]' => 'apps'

        },
        (int) 1 => object(Cake\ElasticSearch\Document) {

            'title' => 'question2',
            '[new]' => false,
            '[accessible]' => [
                '*' => true
            ],
            '[dirty]' => [],
            '[original]' => [],
            '[virtual]' => [],
            '[errors]' => [],
            '[invalid]' => [],
            '[repository]' => 'apps'

        }
    ],
    'id' => 'AVYJCVkoBgZSBgn8x1jS'
]

Sorry for the long post ;-)

Datasource\Connection#_log method is overriding \Elastica\Client#_log without calling parent::_log. This is causing logger to be never set thus resulnting on NPE - logger is set in overriden implementation.

adding parent::_log call removes this error. I am not sure, but looks like one of those methods are surplus. Both of them will log the request (one via debug() and one via log()) but only parent implementation cares about checking if logger actually exists and creates it if needed.

On the other hand, overriding implementation is generating following warning on call
$this->_logger->log($loggedQuery);
Missing argument 2 for Elastica\Log::log(), called in Z:\...\vendor\cakephp\elastic-search\src\Datasource\Connection.php on line 201 and defined [ROOT\vendor\ruflin\elastica\lib\Elastica\Log.php, line 46]

but this must bue due to different logger implementations i guess. To stay with the cake nature, overriding implementation should handle logger creation if other parts of the system are not doing that alread (but it should be set it think, as this is a part of ConnectionInterface).
Maybe bug in TypeRegistry then ?

Hello guys,
Someone can help me? i have an mapping call doacoes and i did a query with a basic condition with "where" but i receive a wrong response "Wrong parameters for Elastica\Exception\ResponseException([string $message [, long $code [, Throwable $previous = NULL]]])". I only get wrong message when i use condition "where", however, when i use $doacao = $this->Doacoes->find(), it works perfectly.

With condition "WHERE"!

Without condition "WHERE"!

or at least give an option to not map it

currently im trying to use elastic search to search mysql database records and i have to map some field to mysql id and then use it while it could be done simpler by just keeping the _id for Documents in plugin

Could you write some example to use?
I would like to use mysql and ElasticSearch simultaneously.

When adding a new document, I can see that you can specify the id from the entity.
this then get set to _id in ES
can i also add the value/field id in my _source?

i.e
create a document with this info in ES $entity = ['id' => 2, 'name' => 'Account Name'];
{
"_index": "sample_index",
"_type": "accounts",
"_id": "2",
"_score": 1,
"_source": {
"name": "Account Name"
}
}

is there a way where I can get this kind of document?
{
"_index": "sample_index",
"_type": "accounts",
"_id": "2",
"_score": 1,
"_source": {
"id": 2,
"name": "Account Name"
}
}

Found an error in the reponse exception class. Needed to change to the following. Not really sure how to use github so just posting it here

public function __construct(Request $request, Response $response)
{
    $this->_request = $request;
    $this->_response = $response;
$error = $response->getError();
    parent::__construct($error['reason']);
}

Hi
i really don't understand the documentation of the elasticsearch plugin in the cookbook.
Is it possible to have an example (like a blog) with associated tables (belongsTo and belongtsToMany) and this plugin and maybe with a search in articles

Thanks

Hello,
I was trying to load a behavior inside a type and i noticed that initialize() is not called when i load the type via TypeRegistry::get('TypeName')

        debug($countQuery->all()->getTotalHits());
        $profilesCount = ($countQuery->count());
        debug($profilesCount);

First debug will give you a different number than second. Except you have the same counts ;)

I expected $countQuery->count() to return the same number as getTotalHits().

While I'm OK with using getTotalHits() after I found the method I don't think the behavior of count() is correct.

If I pull a record from ES and edit it all is fine, however if I create a new entity save it, modify it and resave I get "failed to parse". From my debugging I believe https://github.com/cakephp/elastic-search/blob/master/src/Type.php#L473 should read "unset($data['id']);"

working example (fetch and save)

$menuItemType = TypeRegistry::get('MenuItems');

$menuItem = $menuItemType->find()->firstOrFail();
$menuItem->content = "changed content";
$menuItemType->save($menuItem);

Failing example (new, save and resave)

$menuItemType = TypeRegistry::get('MenuItems');
$menuItem = $menuItemType->newEntity();
$menuItem->action = 'Do something';
$menuItemType->save($menuItem);

$menuItem->content = 'Some content';
$menuItemType->save($menuItem);

Should I be able to use the PaginatorComponent with Elastic Search the same as if using Cake\ORM\Query? I have a plugin, YummySearch, that I'd like to work with ES as well but it requires using Cake\ORM\Query.

Error:
Unknown method "alias"

Code

$Errors = TypeRegistry::get('Errors');
$query = $Errors->find();
$errorLog = $this->paginate($query);

Trace

⟩ Cake\ElasticSearch\Query->__call
CORE/src/Controller/Component/PaginatorComponent.php, line 174
⟩ Cake\ElasticSearch\Query->alias
CORE/src/Controller/Component/PaginatorComponent.php, line 174
⟩ Cake\Controller\Component\PaginatorComponent->paginate
CORE/src/Controller/Controller.php, line 717
⟩ Cake\Controller\Controller->paginate
ROOT/plugins/Admin/src/Controller/ErrorLogController.php, line 39
⟩ Admin\Controller\ErrorLogController->index
CORE/src/Controller/Controller.php, line 440
⟩ Cake\Controller\Controller->invokeAction
CORE/src/Http/ActionDispatcher.php, line 119
⟩ Cake\Http\ActionDispatcher->_invoke
CORE/src/Http/ActionDispatcher.php, line 93
⟩ Cake\Http\ActionDispatcher->dispatch
CORE/src/Routing/Dispatcher.php, line 60
⟩ Cake\Routing\Dispatcher->dispatch
ROOT/webroot/index.php, line 37

My System
CakePHP 3.4.13
CakePHP/elastic-search 0.3.5
PHP Version 5.6.30-11+deb.sury.org~xenial+3
Ubuntu 16.04

Edit: Does this answer my question?

This is a pre-alpha version of an alternative ORM for CakePHP 3.0

\vendor\ruflin\elastica\lib\Elastica\Exception\ResponseException.php (line 34)
[
	'root_cause' => [
		(int) 0 => [
			'type' => 'parsing_exception',
			'reason' => 'no [query] registered for [filtered]',
			'line' => (int) 1,
			'col' => (int) 83
		]
	],
	'type' => 'parsing_exception',
	'reason' => 'no [query] registered for [filtered]',
	'line' => (int) 1,
	'col' => (int) 83
]

The filtered query has been deprecated and removed in ES 5.0. what's plan for ElasticSearch 5.0?

Hi guys, I have a problem with insert or update of a document, when i insert a new row in document and redirect to my index the new row don't appear, however when a press ctr+F5 the row appear, i searched in document of elastic search and find an method call "refresh", but i don't know apply this in cakephp with elastic search, someone can help me?

Add a FormHelper context class to allow ElasticSearch Document Introspection. The context class should provide similar functionality to the relation ORM counterpart.

I updated with composer, but I am getting errors in querying. Errors in Elastica. It's downloading 2.2 version but 5.0 is needed for Latest Elastic Search Version.
How should I update it.

Hi @markstory
I just tested the last commit and initialize() is still not called, there is some problem in #80

I have ran into the following error message
Wrong parameters for Elastica\Exception\ResponseException([string $message [, long $code [, Throwable $previous = NULL]]])

Seems like related to underlying dependency bug ruflin/Elastica#970.
Probably switching to 3.1.1 client would fix the issue

I try to get this query:

GET my_index/_search/
{
  "query": {
    "bool": {
        "should": [
           {
               "nested": {
                  "path":"tags",
                  "query": {
                    "bool": {
                      "must": [
                        {"match": {"tags.name": "Abu Dhabi"}}
                      ]
                    }
                  }
                }
           },
           {
               "nested": {
                  "path":"tags",
                  "query": {
                    "bool": {
                      "must": [
                        {"match": {"tags.name": "Zürich"}}
                      ]
                    }
                  }
                }
           }
        ]
    }
  }
}

I did spend now hours, days to get the query but without success.
All my queries are always must queries not should queries:

[
	'query' => [
		'bool' => [
			'filter' => [
				(int) 0 => [
					'bool' => [
						'must' => [
							(int) 0 => [
								'nested' => [
									'path' => 'tags',
									'query' => [
										'term' => [
											'tags.name' => 'New York'
										]
									]
								]
							]
						]
					]
				]
			]
		]
	]
]

How to turn this in a should query?
Or how to build the query to get articles where at least one tag does match?

Thanks for any help or hints.

Here how my document looks like:

{
   "_index":"my_index",
   "_type":"daily_news",
   "_id":"2",
   "_version":1,
   "_score":1,
   "_source":{
      "title":"Second record",
      "publish_date":"2016-01-01 12:00:00",
      "tags":[
         {
            "name":"New York",
            "identifier":null,
            "keyname":"newyork"
         },
         {
            "name":"Abu Dhabi",
            "identifier":null,
            "keyname":"abudhabi"
         }
      ]
   }
}

Mapping:

"mappings":{
   "daily_news":{
      "properties":{
         "title":{
            "type":"string"
         },
         "publish_date":{
            "type":"string"
         },
         "tags":{
            "include_in_root":true,
            "type":"nested",
            "properties":{
               "identifier":{
                  "type":"string"
               },
               "name":{
                  "type":"string"
               },
               "keyname":{
                  "type":"string"
               }
            }
         }
      }
   }
},