Implement NoSQL Storage for ActiveRecord about yii2 HOT 71 CLOSED

Personal think elasticsearch PHP client, Elastica is very good! this is his github repository URL: https://github.com/ruflin/Elastica

from yii2.

Btw the checkboxes in here are made like this: https://github.com/blog/1375-task-lists-in-gfm-issues-pulls-comments

from yii2.

Most noSQL databases have connection via REST. Will be greate have one defined interface for all Rest client. This give us able have one selected (by developer) REST library for all needs. We just say with which library it should use

from yii2.

You mentioned 2 of my yii extensions so I might share my thoughts/experiences on this issue here: The whole noSQL topic is far too complex to make a "one fits all" solution even possible (I've tried and failed :-) ). I wanted to make things easier with my ActiveResource extension but there are a lot of design flaws in there so it is a bad candidate for reviewing. My suggestion for Yii in this respect: Take a look at Guzzle which is a REST client framework. I takes care of the annoying parts (authentication, caching, asynchronous/parallel curl requests), is extensible via plugins and actively maintained. Yii could offer an interface on top of Guzzle so that there is a well defined API to develop against, maybe in combination with the repository pattern

from yii2.

As I see Guzzle being mentioned here: While Guzzle is indeed a formidable solution with facilities that make inclusion into Yii trivial, I'd like to advise caution. As mentioned before, it is not just a library but an entire framework. It would also introduce a hard dependency on libcurl (it is my impression that the original author simply wanted to spare himself from dealing with all the gory details of socket programming). Just saying 😏

from yii2.

This is almost getting a bit OT, but I have to say: +1 for the repository pattern. Been missing it in Yii a lot of times.

from yii2.

I would love to see HyperDex in this list. I know this is a bit of a longshot but it's got good performance, supports ACID transactions and has strong consistency guarantees and fault tolerance to boot.

It's outperformed MongoDB, Cassandra, and I believe Redis in the YCSB (despite better consistency guarantees and fault tolerance).
It's only downside really is that it's relatively young compared to other solutions.

Here are some links:

http://hyperdex.org/doc/
http://hyperdex.org/performance/
http://hyperdex.org/performance/details/ (more details regarding benchmark)
https://github.com/tritondigital/php-hyperdex

Alternativly I could settle with MongoDB even though I'm uncomfortable with some aspects of the data storage (or non-storage in this case)

from yii2.

Implement NoSQL Storage for ActiveRecord

I am not sure how exactly such implementation should look at the end.
I can not see an universal solution for this.
Different NOSQL storages use different approaches for the data storage and data structure.
There are several types:

• key-value oriented (Redis)
• document oriented (MongoDb)
• graph oriented (Neo4j)

Key-value storages could match the original Yii model paradigm, which relies on architecture, when model has only (or at least mostly) scalar type attributes.

However document oriented storage does not fit the Yii model paradigm. In such storages (as MongoDb) the single storage entry is a huge nested JSON document, which can not be described by simple model with scalar attributes.

Graph oriented database (such as Neo4j) actually could fit the Yii model paradigm. However the search inside such storages is rather different from the common database. Here you are starting with some node and then go to the nodes related with it. Sometimes universal search criteria can not be composed for fetching particular node with the single query.
But the main problem with such storages as Neo4j is they do not support PHP directly and there are no PHP drivers for them. Such storages can be accessed only via REST API.

from yii2.

@Haensel commented in #73 (comment):
I don't know if it is still the case but if you are still planing to implement a mapper for graph databases (Neo4j, OrientDB etc.) I would suggest you to take a look at Tinkerpop Blueprints (https://github.com/tinkerpop/blueprints/wiki). From the wiki:

"Blueprints is analogous to the JDBC, but for graph databases. As such, it provides a common set of interfaces to allow developers to plug-and-play their graph database backend. Moreover, software written atop Blueprints works over all Blueprints-enabled graph databases."

Neo4j and OrientDb both support Blueprints and are the most widely used graph dbs, so it might be worth a look

from yii2.

In case you wonder where to find this: I'm on the redis branch: https://github.com/yiisoft/yii2/tree/redis

from yii2.

Is there an interface or required functions for Connection class? I would like to implement support for Rexster (as per the tinkerpop comment earlier). I've already made a php client for RexPro (which part of is very similar to the Yii Connection.php). I'd like to use it as a helper as a first step even if it's redundant (because I might move on to making a php extension to support rexPro.
I'm going to start now just trying to stick as much as possible to the current file/functionality.

from yii2.

Is there an interface or required functions for Connection class?

Will be there but I am not finished with that yet. Will define the interface or base class, when we have more dbms implemented so I'd be very interested in seeing your class.

from yii2.

ok!
I haven't gone very far yet. I've used your Connection class as a basis and have made one for Rexter. I've left out Transactions and the concept of Isolation (or non-isolation in this case which allows for queries to use bindings set by previous queries) because it will probably need a bit of discussion .
I'll get onto looking into activeRecord and ActiveQuery classes next. Once I get ideas as to what to do about representation of Vertices vs Edges and routes and so forth. I'll probably post my suggestions. (I also should have a look at Neo4Yii to see how Hansael tackled it)

from yii2.

ok having a look around. How would you suggest modifying __get() and __set() for shema-less db structures in activeRecord? This is kind of tricky as it'll catch all.

from yii2.

graph DB and gremlin discussion for graph implementation here : http://www.yiiframework.com/forum/index.php/topic/43863-graph-databases-gremlin-qquery-building/
I reckon I should have something to push online by the end of the week depending on how the discussion goes. a preview version if you will

from yii2.

Sadly haven't had the time to go there, but there is some really interesting stuff at berlinbuzzwords conference this year http://berlinbuzzwords.de/ .

http://berlinbuzzwords.de/program/session-schedule/all

• http://berlinbuzzwords.de/sessions/document-relations-elasticsearch
• http://de.slideshare.net/martijnvg/document-relationsbbuz2013
• http://berlinbuzzwords.de/sessions/couchdb-implementation

from yii2.

I don't mean to plug my own content here but: https://github.com/Sammaye/MongoYii can handle MongoDB in a schemaless manner and I have the intention of converting it to Yii2 as soon as a stable interface for its conversion is made.

Maybe a stripped down version of something like that could be used for core?

from yii2.

It's nothing bad about showing your own content when you think it is good. Looks really interesting, will definitifely take a look at it. Thanks for sharing!

from yii2.

I've been very busy recently but finally shifted towards making Yii independent libraries for graph DBs. Mostly speaking a DB client for rexster and an equivalent of a QueryBuilder for gremlin. Now I'm concentrating on integrating these into Yii2. I was wondering where the correct place for external libraries to reside was? I've been sticking them in the helpers dir for now even though that didn't seem correct. Any help would be appreciated.
Thanks

from yii2.

Normally, for better composer integration, many have taken to adding such things to a vendors folder. This pattern is, of course, also replicated in many languages including Java.

from yii2.

Yeah, I couldn't seem to find one though. Doesn't matter, I'll place them there for now and move on. I'll move them around once I get a better idea.

from yii2.

Support for MongoDB +1 (ActiveRecord, Sessions)
Support for Redis +1 (Cache, Sessions)

from yii2.

Support for Redis +1 (Cache, Sessions)

@jabbon this is already implemented and available on master branch.

Support for MongoDB +1 (ActiveRecord, Sessions)

does it really make sense to store Sessions on MongoDB?

from yii2.

Yes, MongoDB, if done right, can store the session completely in working set, i.e. RAM just like redis

from yii2.

Suggest moving /db/redis to /redis because redis is relatively independent of the other classes under db.

from yii2.

The current classes are but the AR does extend from the db\ActiveRecord
I think we should make a decision when redis an elasticsearch are ready.

from yii2.

By having redis directly under the framework folder, it's like making it a self-contained extension with dependency on Yii. Other directories under db mainly contain drivers which are not the same as redis.

from yii2.

Agree. Will make the move soon.

from yii2.

I vote for Apache Cassandra inclusion - it uses SQL-like language ( CQL ), has PDO driver, i made it work for my closed-source job project.

On the other side, my piece for it may be application-biased (or have some parts of application inside it), and ignorant of some framework guidelines. It's also ridden with backward-compatibility kludges, since i converted from Yii1 and from PHPCassa to PDO - two overhauls far before start ;).

So i it may be better if core team write their own, stumble upon all lack of dependency injection, and ask me questions if stuck. Or should i go over all my code and file issues on all stumbles i had to dodge ?

Anyway, i am willing to help with this.

from yii2.

It would be great if you could submit a PR for this, and we can work together to polish it (unless other @yiisoft/core-developers has experience with Cassandra and willing to work on this). Thanks.

from yii2.

I think we can close this issue now. I am done with all the system I know and can support as I am going to work with them on a regular basis. Others should be provided as extensions or by other core developers. If someone wants to start working on an implementation of an AR feel free to contact me, I can help with that.

from yii2.

This is a wonderful piece of work! Thank you, @cebe and @klimov-paul

from yii2.

:-)

from yii2.

great work guys, thanks!

from yii2.

I will probably start work on a Solr extension in a min, have need of one and it appears there isn't one yet when I search Google.

from yii2.

We decided to use elasticsearch in favor of solr, as it is much more powerful. But that shouldn't stop you from creating a solr extension :)

from yii2.

I also use Solr and would help to write/test/review such extension.

from yii2.

I have just started on a Solr application component which extends from the Pear PHP Solr Client I have started here since I am still quite new to PHP coding in Solr (it is normally another guy who does it, I am an ES and Sphinx guy) but I wouldn't mind someone (like you or whatever) with the knowledge taking something like: https://github.com/phpnode/YiiSolr and making something pure awesome from it.

The idea is to eventually get to that but as I said, I am still a newbie to Solr but it is a requirement of my application so I have to port it over.

from yii2.

So what do you need to get started with it?

from yii2.

You may take the elastcisearch implementeation and mark all code that needs to be adjusted with todos. After that starting to implement connection and basic querying.

from yii2.

Yeah I think that's a good idea actually thanks

from yii2.

I will probably strip the ES code and then try and copy the Pear PHP client over, though not too much since that code is old and decreped so...yeah, just take the basic requirements of Solr (escape functions etc) and then build it into the rest that exists in ES code

from yii2.

There's http://www.solarium-project.org/ too. I haven't looked at the code of the PEAR extension you mentioned, but given how completely awful the majority of other PEAR extensions have been through the years I ever looked at them, perhaps this one is a cleaner code base to use for your implementation (assuming the license is compatible).

from yii2.

Ah yeah, that looks much better I'll take a look and see what I can with
it, maybe use that as the base and the Yii2 classes as abstraction

On 25 April 2014 12:11, rawtaz [email protected] wrote:

There's http://www.solarium-project.org/ too. I haven't looked at the
code of the PEAR extension you mentioned, but given how completely awful
the majority of other PEAR extensions have been through the years I ever
looked at them, perhaps this one is a cleaner code base to use for your
implementation (assuming the license is compatible).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-41381636
.

from yii2.

Wow that Solarium is pretty much a framework in itself

from yii2.

I am unsure if solarium project is fully compatible with Yii2.

It uses the symfony event dispatcher ( https://github.com/symfony/EventDispatcher ) which I don't think would be callable with Yii2's own events. Not only that but it isn't the lightest piece of code.

Anything built from it would require a lot of hacking in Yii2's internals to make sure Yii2 does not interferer with it.

Basically it will be more of a jerry rig if I build from that due to how complete it is. I will still try though I think, it has all of the functions etc that are needed, but then I might also be able to do these simpler if I coded from start and used the ES base

from yii2.

They should not interferre but there is the question of the need of a yii implementation when solarium has everything you need already. What is missing from solarium what a yii extension would bring?

from yii2.

Well I guess it is implementation with Yiis active record etc, I mean I don't really need that, only querying but I am sure others would

from yii2.

Is solr really used as a data storage as it is the case with elasticsearch? If it is not a common usage to create update and delete records in solr it may not be worth the effort of creating AR layer for it.

from yii2.

We were recomended by users on the Solr group to actually use Solr as a
full data store when querying it, so our entire "books" table is duplicated
across MySQL and Solr.

The reason we don't need Active record is because we are usng old code that
relies on cronjobs to send xml files to Solr once every so often.

Thugh Solr is more of a search tech than a database...hmmm

On 25 April 2014 13:31, Carsten Brandt [email protected] wrote:

Is solr really used as a data storage as it is the case with
elasticsearch? If it is not a common usage to create update and delete
records in solr it may not be worth the effort of creating AR layer for it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-41387023
.

from yii2.

I mean SOlarium has an active record layer but not Yii's, maybe it might be
ok

On 25 April 2014 13:59, Sam Millman [email protected] wrote:

We were recomended by users on the Solr group to actually use Solr as a
full data store when querying it, so our entire "books" table is duplicated
across MySQL and Solr.

The reason we don't need Active record is because we are usng old code
that relies on cronjobs to send xml files to Solr once every so often.

Thugh Solr is more of a search tech than a database...hmmm

On 25 April 2014 13:31, Carsten Brandt [email protected] wrote:

Is solr really used as a data storage as it is the case with
elasticsearch? If it is not a common usage to create update and delete
records in solr it may not be worth the effort of creating AR layer for it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-41387023
.

from yii2.

I sync Solr with my main Db by using data importer so I don't manipulate documents directly, but if I had an AR interface available I'd probably consider doing together with AR models from the main Db.
I don't think it will be much different than ES implementation.

from yii2.

The way I use Solr is secondary; I have my main database as usual, and the things that need indexing I feed to Solr explicitly. Then when I need to search (using Solr's capabilities) I query it and it returns the list of PKs of the result. I then use a data provider in Yii as usual to get those records. I do it this way separate concerns, keep my main DB as the true source of the data, and so that I have the usual Yii ecosystem to work with instead of some added-on Solr-ish classes. This way I can easily switch out Solr if I wanted to, replacing it with whatever other thing that gets me the PKs.

from yii2.

For that, if you were for example to create the solr document on
afterSave of the mian Yii active record model it would be fine (maybe
better) to use Solarium active record interface then.

The main question right now is: Should we just rig Solarium into Yii2 or
actually take ES implementation and create a whole new one?

The way @janwas uses it could easily be done by Solarium and it would take
the weight of maintanence off of our shoulders.

keep my main DB as the true source of the data, and so that I have the
usual Yii ecosystem to work with instead of some added-on Solr-ish classes

Yeah you see that would be the problem to using Solarium it would mean that
if you want to switch out Solr there will not a standard interface do that
by. That would be the benefit of creating the Solr library natively in Yii2.

I looked into abstracting Solarium into Yii Query object but it...well it
seemed pointless and like I was reinventing the wheel so that could be
destroyed.

Essentially we gain the power of Solarium but you wouldn't be able to trade
it out so well.

On 25 April 2014 14:06, Jan Waś [email protected] wrote:

I sync Solr with my main Db by using data importer so I don't manipulate
documents directly, but if I had an AR interface available I'd probably
consider doing together with AR models from the main Db.
I don't think it will be much different than ES implementation.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-41389986
.

from yii2.

How often would you exchange out solr? I mean is this a common task that
you need a standard interface for? Or is it a rare task that you would do
in the event that you do change search provider?

On 25 April 2014 14:13, Sam Millman [email protected] wrote:

For that, if you were for example to create the solr document on
afterSave of the mian Yii active record model it would be fine (maybe
better) to use Solarium active record interface then.

The main question right now is: Should we just rig Solarium into Yii2 or
actually take ES implementation and create a whole new one?

The way @janwas uses it could easily be done by Solarium and it would take
the weight of maintanence off of our shoulders.

keep my main DB as the true source of the data, and so that I have the
usual Yii ecosystem to work with instead of some added-on Solr-ish classes

Yeah you see that would be the problem to using Solarium it would mean
that if you want to switch out Solr there will not a standard interface do
that by. That would be the benefit of creating the Solr library natively in
Yii2.

I looked into abstracting Solarium into Yii Query object but it...well it
seemed pointless and like I was reinventing the wheel so that could be
destroyed.

Essentially we gain the power of Solarium but you wouldn't be able to
trade it out so well.

On 25 April 2014 14:06, Jan Waś [email protected] wrote:

I sync Solr with my main Db by using data importer so I don't manipulate
documents directly, but if I had an AR interface available I'd probably
consider doing together with AR models from the main Db.
I don't think it will be much different than ES implementation.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-41389986
.

from yii2.

Since setting up my Solr cores in the project I'm thinking about now years ago, I haven't had a need to switch it out, so no, that's not often :)

Honestly, I'm quite fine with the way I use it, so personally I don't see much point in integrating Solr into AR. But it depends on what one want to use it for. Just searching, or searching AND providing data..

from yii2.

It seems as though the majority of those interested would be happy with a search provider rather than a DB provider. Okay, I am going to start hooking Solarium into Yii2 instread of starting from scratch.

from yii2.

I use it the same way as @rawtaz described and I see the benefit of having a common interface for filtering, getting PKs in result and using them to fetch full records from another storage.

Anyway, it is just another database with a well described interface. If the implementation will be simple enough why not doing that. It is hard to select an existing interface and if such would come with the framework itself it would make adopting such solutions easier.

from yii2.

Yeah the Solarium implementation is very simple, it would be as simple as Yii2's easily; so there is a case there that porting it natively to Yii2 would be reinventing the wheel as well.

from yii2.

I agree :) And trying to create an API that can be common across Solr and other similar systems is probably hard, since they have individual features and query syntax.

Unless there's a good reason to try AR-izing Solr I think it's easiest to just use Solarium or similar directly. I imagine that in doing so one would still wrap it up nicely in an application component and what not, to provide a nice but small local/internal API to feed documents to solr and to query it as well.

from yii2.

An important part of the yii integration would be the implementation of the QueryInterface to allow using it with dataproviders and gridview. https://github.com/yiisoft/yii2/blob/master/framework/db/QueryInterface.php This query object may translate then to a search query that is executed with solarium.

See elasticsearch implementation:
https://github.com/yiisoft/yii2/blob/master/extensions/elasticsearch/QueryBuilder.php#L64
where is converted to elasticsearch filter. where only supports array syntax as it is translated to non-sql.

from yii2.

One other alternative I suppose is to create a specific SolrDataProvider,
that was one thought I had, instead of attempting to make QueryInterface
and Solarium work together.

On 25 April 2014 14:51, Carsten Brandt [email protected] wrote:

An important part of the yii integration would be the implementation of
the QueryInterface to allow using it with dataproviders and gridview.
https://github.com/yiisoft/yii2/blob/master/framework/db/QueryInterface.phpThis query object may translate then to a search query that is executed
with solarium.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/11#issuecomment-41394371
.

from yii2.

Ok it seems this might be quite easy, as a sample this is the syntax I have working under Solarium:

    $query = Yii::$app->solr->createSelect();
    $query->setQuery('edismax&qf=title^20.0 OR description^0.3');
    $result = Yii::$app->solr->select($query);
    var_dump($result->getNumFound());

That's how difficult it will be to use Solr with Yii2. As you will notice it literally is direct portal to Solarium.

The next part is probably the hard one which is to integrate it with widgets as @cebe states.

from yii2.

As a small update, I have this extension actually working now.

It is a bit crude, just an app hook and a data provider.

As a side note, since my solr actually covers multiple document types (and I assume others could) the data provider does not work on standard singular AR models which means I have had to implement a SolrDocumentTrait which provides and interface and inheritance for making specific solr models.

It isn't required to use this and you can make a dataprovider link to a single model as well so you can use the AR of Yii2 to produce sorting etc.

I will be cleaning it up and maybe releasing it today if all goes well.

from yii2.

Great. It may become a pretty popular extension.

from yii2.

Done: http://www.yiiframework.com/extension/yii2-solr/

from yii2.

Ok...this is maybe way too late in the game but I just started realizing that Couchbase is a major NoSql player used in a ton of enterprise apps from Ebay to Apple. Now that things are in beta stage I may be way too late to ask about it? I wish I was more of a NoSql-er. Any idea how difficult it would be/different from the MongoDb ActiveRecord support?

from yii2.

I think you might need to make yourself, it should be quite easy and could
be relatively popular.

On 30 May 2014 07:07, joltup [email protected] wrote:

Ok...this is maybe way too late in the game but I just started realizing
that Couchbase is a major NoSql player used in a ton of enterprise apps
from Ebay to Apple. Now that things are in beta stage I may be way too late
to ask about it? I wish I was more of a NoSql-er. Any idea how difficult it
would be/different from the MongoDb ActiveRecord support?

—
Reply to this email directly or view it on GitHub
#11 (comment).

from yii2.

@joltup it is not too late, you can create an extension for it if you have experience with this DB. You can ask here for help if you have any questions regarding design of the classes. Also check out the other implementations to see how it is done.

from yii2.

@joltup any progress with Couchbase?

from yii2.

as far as I know nobody is working on that, if you want it you can start yourself and ask for help if you need.

from yii2.