Giter VIP home page Giter VIP logo

postgres_ext-serializers's Introduction

PostgresExt-Serializers

Build Status Code Climate Gem Version

This Gem patches the ArraySerializer in Active Model Serializers to offload JSON serialization to PostgreSQL, which is much faster and more memory efficient than serialization in Ruby.

Instead of instantiating lots of ActiveRecord models, which are then serialized to a hash using ActiveModel Serializers and finally converted to JSON, this Gem builds the entire JSON in the database and returns it as a single string that can be passed through to the client unmodified.

How does it work?

You can read Avoid Rails When Generating JSON responses with PostgreSQL by Dan McClain on the DockYard Blog or watch a YouTube video of his Postgres Open 2014 Talk Using PostgreSQL, not Rails, to make Rails faster (Slides) to understand how this Gem works under the hood.

Supported Versions

This gem currently only works with the older active_model_serializers 0.8.x and activerecord 4.0/4.1/4.2. For activerecord 4.2 support you need at least postgres_ext version 2.4.0.

You must be using at least Postgres 9.2, but 9.4 or later is recommended.

Support for newer AMS versions might be added in the future.

Looking for help?

If it is a bug please open an issue on Github. If you need help using the gem please ask the question on Stack Overflow. Be sure to tag the question with DockYard so we can find it.

Installation

Add this line to your application's Gemfile:

gem 'postgres_ext-serializers'

And then execute:

$ bundle

Or install it yourself as:

$ gem install postgres_ext-serializers

Usage

Just require 'postgres_ext/serializers' and use ActiveModel::Serializers as you normally would! Postgres_ext-serializers will take over anytime you try to serialize an ActiveRecord::Relation.

Methods in Serializers and Models

If you are using methods to compute properties for your JSON responses in your models or serializers, postgres_ext-serializers will try to discover a SQL version of this call by looking for a class method with the same name and the suffix __sql. Here's an example:

class MyModel < ActiveRecord::Base
  def full_name
    "#{object.first_name} #{object.last_name}"
  end

  def self.full_name__sql
    "first_name || ' ' || last_name"
  end
end

class MySerializer < ActiveModel::Serializer
  def full_name
    "#{object.first_name} #{object.last_name}"
  end

  def full_name__sql
    "first_name || ' ' || last_name"
  end
end

There is no instance of MyModel created so sql computed properties needs to be a class method. Right now, this string is used as a SQL literal, so be sure to not use untrusted values in the return value.

Note: Postgres date, timestamp or timestamptz format

Postgres 9.2 and 9.3 by default renders dates according to the current DateStyle Postgres setting, but many JSON processors require dates to be in ISO 8601 format e.g. Firefox or Internet Explorer will parse string as invalid date with default DateStyle setting. Postgres 9.4 onwards now uses ISO 8601 for JSON serialization instead.

Developing

To work on postgres_ext-serializers locally, follow these steps:

  1. Run bundle install, this will install (almost) all the development dependencies
  2. Run gem install byebug (not a declared dependency to not break CI)
  3. Run bundle exec rake setup, this will set up the .env file necessary to run the tests and set up the database
  4. Run bundle exec rake db:create, this will create the test database
  5. Run bundle exec rake db:migrate, this will set up the database tables required by the test
  6. Run BUNDLE_GEMFILE='gemfiles/Gemfile.activerecord-4.2.x' bundle install --quiet to create the Gemfile.lock.
  7. Run bundle exec rake test:all to run tests against all supported versions of Active Record (currently 4.0.x, 4.1.x, 4.2.x)

Authors

Dan McClain twitter github

We are very thankful for the many contributors

Versioning

This gem follows Semantic Versioning

Want to help?

Please do! We are always looking to improve this gem.

Legal

DockYard, Inc ยฉ 2014

@dockyard

Licensed under the MIT license

postgres_ext-serializers's People

Contributors

bcardarella avatar danmcclain avatar felixbuenemann avatar mariochavez avatar mcm-ham avatar pboling avatar rlivsey avatar swatijadhav avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

postgres_ext-serializers's Issues

relation "" does not exist error

I keep getting this error when serializing child relationships:

ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR:  relation "" does not exist
LINE 5:                WHERE a.attrelid = '""'::regclass
                                          ^
SELECT a.attname, format_type(a.atttypid, a.atttypmod),
     pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
FROM pg_attribute a LEFT JOIN pg_attrdef d
  ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attrelid = '""'::regclass
 AND a.attnum > 0 AND NOT a.attisdropped
ORDER BY a.attnum

It only happens the first time after the app starts, subsequent times it's fine. It seems to be caused by relation_query.dup.arel_table.name being an empty string but relation_query.arel_table.name is fine. Not sure why .dup is used but removing it fixes this issue, or just simply calling relation_query.arel_table before relation_query.dup also fixes it.

:serializer / :each_serializer ?

I'm really impressed by the gains we can get using postgres_ext-serializers ! Are you guys still working on the project ?

I feel like active_model_serializers 0.9 is gonna bring a lot of changes.

Also I am working on a name spaced environment, relying on relation.klass.active_model_serializer isn't really sustainable, plus I'd like to be able to pass some custom serializers / each_serializer just as I do with active_model_serializers, do you guys think it would be possible to maybe share some pieces of code with AMS ?

Rails 3.2 compatibility?

I'm working on a Rails 3.2 app and was wondering what Rails 4 specific features would need to be back-ported to support 3.2, of any?

Thanks!
Josh

I added require 'postgres_ext/serializers' to my active_model_serializers init file, but I don't see it pulling the JSON from postgres

Hi, this seems so simple to setup, but I can't seem to get any basic functionality to work in terms of moving the JSON from rails to postgres. I tried including the following in my ams init, in my specific serializers and in my models, but it never seems to be activating.

I'm on Rails 4.2.3

and this is what I've tried adding to multiple files:

require 'postgres_ext/serializers'

Really appreciate your help, I know I must be missing something obvious. Love the concept of this gem btw... we were going to hand roll all our serializers in SQL postgres, which would be very tedious and this gem will save us a lot of pain if we can get it to work.

Thanks!

Breaks polymorphic relationships in Ember Data

With postgres_ext-serializers installed polymorphic relationships are serialized as:

images: [{
  id: 1,
  imageable_id: 1,
  imageable_type: 'User'
}]

Which ember data does not support: emberjs/data#1574
Without postgres_ext-serializers installed polymorphic relationships are serialized as:

images: [{
  id: 1,
  imageable: {
    id: 1,
    type: 'User'
  }
}]

Which ember data does support.

Error on render call in controller

This project seems pretty awesome. I was looking forward to trying it out but I can't seem to get anywhere before running into "NoMethodError (undefined method `active_model_serializer' )"

This error occurs on render in the controller action. Anyone have an idea of what is causing this?

Gems:
rails (4.2.1)
active_model_serializers (0.9.3)
postgres_ext-serializers (0.0.2)

Code:

class Api::ProspectsController < Api::BaseController

  def index
    prospects = current_access.prospects
      .search(search_params)
      .includes(:contact, :user_prospects)
      .page(params[:page]).per(params[:per_page])

    render(
      json: prospects,
      meta: {
        total_pages: prospects.total_pages,
        total_count: prospects.total_count
      },
      root: "contacts"
    )
  end
  ...
end

Stacktrace:
NoMethodError (undefined method active_model_serializer' for #<Class:0x007fb36a62cb98>): app/controllers/api/prospects_controller.rb:9:inindex'

Rendered /Users/mfpiccolo/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/actionpack-4.2.1/lib/action_dispatch/middleware/templates/rescues/_source.erb (2.3ms)
Rendered /Users/mfpiccolo/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/actionpack-4.2.1/lib/action_dispatch/middleware/templates/rescues/_trace.text.erb (0.4ms)
Rendered /Users/mfpiccolo/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/actionpack-4.2.1/lib/action_dispatch/middleware/templates/rescues/_request_and_response.text.erb (8.1ms)
Rendered /Users/mfpiccolo/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/actionpack-4.2.1/lib/action_dispatch/middleware/templates/rescues/diagnostics.text.erb (43.5ms)

Rails 4.2.0.beta2 compatibility failed

Using the Rails 4.2.0.beta2, and this gem, anything like: Post.where(id: [1,2,3]) will fail.
Everything is fine when using Rails 4.1.6. Maybe this is a rails error.
Here's the error:

Loading development environment (Rails 4.2.0.beta2)
>> Post.where(id: [1,2])
(Object doesn't support #inspect)
=>
>> Post.where(id: [1,2]).take
NoMethodError: undefined method `relation' for #<ActiveRecord::ConnectionAdapters::AbstractAdapter::SQLString:0x007fea84b27bb0>
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/postgres_ext-2.3.0/lib/postgres_ext/arel/visitors/to_sql.rb:7:in `visit_Array'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/reduce.rb:13:in `visit'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:625:in `visit_Arel_Nodes_In'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/reduce.rb:13:in `visit'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:783:in `block in inject_join'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:781:in `each'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:781:in `each_with_index'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:781:in `each'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:781:in `inject'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:781:in `inject_join'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:641:in `visit_Arel_Nodes_And'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/reduce.rb:13:in `visit'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:258:in `block in visit_Arel_Nodes_SelectCore'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:257:in `each'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:257:in `each_with_index'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/arel-6.0.0.beta1/lib/arel/visitors/to_sql.rb:257:in `visit_Arel_Nodes_SelectCore'
... 17 levels...
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/railties-4.2.0.beta2/lib/rails/commands/console.rb:9:in `start'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/railties-4.2.0.beta2/lib/rails/commands/commands_tasks.rb:68:in `console'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/railties-4.2.0.beta2/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/railties-4.2.0.beta2/lib/rails/commands.rb:17:in `<top (required)>'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:248:in `require'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:248:in `block in require'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:233:in `load_dependency'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:248:in `require'
    from /Users/zweng/workspace/testapi/bin/rails:8:in `<top (required)>'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:242:in `load'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:242:in `block in load'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:233:in `load_dependency'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/activesupport-4.2.0.beta2/lib/active_support/dependencies.rb:242:in `load'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from /usr/local/var/rbenv/versions/2.1.3/lib/ruby/2.1.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from -e:1:in `<main>'>>

Cancan nested hash condition produces invalid SQL

If I serialize this:

Tag.where(note: {name: 'title'})

It produces SQL like this:

SELECT "tags"."id", 
       "tags"."name", 
       "tags"."note_id" 
FROM   "tags" 
WHERE  "note"."name" = 'Title'

Instead of doing a table join on offers.

Support for metadata

AMS allow to provide metadata, like number of total pages by passing the :meta key to render:

render json: posts, meta: { page:4, total_pages: 5 }

We could support this in postgres_ext-serializers by converting meta to JSON and selecting it as a ::json column in postgres via an additional CTE that can be joined to the final result.

Another approach would be to provide a way to generate meta data entirely in the database, which would be faster, but it would probably be harder to implement and less flexible.

has_many :through broken

Serializing a has_many :through relation is currently broken and results in invalid SQL due to bad column references.

postgres_ext-serializers handles these like a normal has_many relation, when instead it needs to lookup the through_reflection and join the underlying join table to be able to lookup the proper ids.

Problem with serializing models with default_scope

Hallow,

If i try to serialize a model, which have a default scope i get an error.

default_scope { order('name ASC') }
PG::GroupingError - ERROR:  column "services.name" must appear in the GROUP BY clause or be used in an aggregate function
LINE 1: ...ds_ids"."id" FROM "service_fields_ids")  ORDER BY name ASC),...

Add support for non-embedded associations

Not everyone uses the "embed" functionality of ActiveModel::Serializers, and it would be nice if postgres_ext-serializers could support associations in this case (and in the meantime, print a warning or include information in the readme about this).

In theory, it shouldn't be too difficult, as it just requires a nested row_to_json (for has_one) or json_agg (for has_many) on the sub serializers query and the where clause. Much of the code for this already exists, but it's just focused on putting the embedded object in root rather than nested within other objects.

Would there be support for this extension of postgres_ext-serializers?

PG::DuplicateAlias error when calling to_json multiple times

If the to_json method is called multiple times on an ActiveModel::ArraySerializer, ActiveRecord throws a PG::DuplicateAlias error, because the internal state is not cleaned up.

Proposed Fix: Extract the instance variables set in initialize into a reset method and call it at the beginning of _postgres_serializable_array.

Jbuilder

Can postgres_ext-serializers work with jbuilder?

Allow selectively disabling database serialization

There should be an option like use_postgres_serializer: false to disable postgres_ext-serializers on a case by case basis, eg. through default_serializer_options in the controller, or an option passed to render :json.

This would make it easier to work with projects where not all serializers are compatible with postgres_ext-serializers and ease migration towards full database serialization.

Empty data returns null instead of []

If there is no data the output JSON is:

{"tags":null, "notes":null}

Instead of:

{"tags":[], "notes":[]}

This is different behaviour than what the activemodel serializers normally produce and frameworks like ember throw errors.

PG::UndefinedParameter: ERROR: there is no parameter $1

Hi!

I have of next versions:

  • Rails 4.1.1
  • postgres_ext (2.4.1)
  • postgres_ext-serializers (0.0.3)
  • active_model_serializers (0.8.3)

my query is:

@model1.model2.includes(:model3).where('model3.state =?', 'done')

I have a error when run:

json = ActiveModel::ArraySerializer.new(ts, each_serializer: Api::V1::MySerializer).to_json

or

json = @model1.model2.includes(:model3).where('model3.state =?', 'done')
render json: json, each_serializer: Api::V1::MySerializer

Could you help me?

Thanks a lot!

DuplicateAlias error

If I serialize a class like:

class Offer < ActiveRecord::Base
  belongs_to :created_by, class_name: 'User', inverse_of: :offers
  belongs_to :reviewed_by, class_name: 'User', inverse_of: :reviewed_offers
end

Or

class Offer < ActiveRecord::Base
  belongs_to :user
  has_many :items
end

class Item < ActiveRecord::Base
  belongs_to :user 
end

Then it produces two CTE called users_attributes_filter which throws a PG DuplicateAlias error.

Support for {root: false}

First of all, thanks for an awesome project.


When I render with plain active model serializers with

render :json => @events, root: false

It produces an array of objects.

However when I put postgres_ext-serializers in place, it returns

{"events": [same array as before]}

In other words, the option of removing the root does not seem to be working.

serialize in service class will not trigger this extension

service class:

users = User.where(age: 22)
ActiveModel::ArraySerializer.new(users, each_serializer: UserSerializer, root: adults).as_json

This code does not trigger the extension to serialize json in postgres.
Currently I only see the extension is be used when
in controller:

render json: users

Is there a way to use this extension other then the controller?

Sideloading belongs_to produces invalid sql

class Tag < ActiveRecord::Base
  belongs_to :note
end

class TagSerializer < ActiveModel::Serializer
  attributes :id
  embed :ids, include: true
  has_one :note
end

class Note < ActiveRecord::Base
  has_many :tags
end

class NoteSerializer < ActiveModel::Serializer
  attributes :id, :content, :name
  has_many   :tags
  embed      :ids, include: true
end

If we serialize tag it produces:

WITH notes_ids AS 
( 
   SELECT "notes"."id" 
   FROM   "notes" 
   WHERE  "notes"."note_id" IN  ( SELECT "tags_ids"."id" FROM   "tags_ids")
)
...

Where tag_ids is defined afterwards so is not found in this query and note_id does not belong in notes table.

Support for active_model_serializers (>= 0.9.0)

Is there any chance you can add active_model_serializers (>= 0.9.0) support any time soon?
Being able to use "grape-active_model_serializers" combined with "postgres_ext-serializers" would be a major win for any rails api.
Thanks and regards,

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.