Giter VIP home page Giter VIP logo

stream_json_demo's Introduction

Stream JSON Demo

You can see the entire diff compared to a vanilla Rails 4.2.0 app. The app is also running on Heroku at http://stream-json-demo.herokuapp.com/random_numbers.

You can curl this demo with:

$ curl --compressed http://stream-json-demo.herokuapp.com/random_numbers

The app lazily generates a million random numbers and streams json to the client at the /random_numbers endpoint:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Encoding: deflate
Transfer-Encoding: chunked
[
  {
    "index": 0,
    "time": "2015-03-12T16:24:59.381Z",
    "number": 96712233280936019238290948377812066522
  },
  {
    "index": 1,
    "time": "2015-03-12T16:24:59.381Z",
    "number": 322627977204921440486826283319053625299
  },
  {
    "index": 2,
    "time": "2015-03-12T16:24:59.381Z",
    "number": 174374410324316249417283007292339235935
  }
]

To stream the JSON, the app uses ActionController::Live which allows us to send data to the client in chunks with HTTP chunked transfer encoding. The action also demonstrates compressing such a response with deflate. With this setup, you could (theoretically) stream an infinitely large response to the client without running out of memory.

The relevant file is app/controllers/random_numbers_controller.rb:

class RandomNumbersController < ApplicationController
  include ActionController::Live

  # Return a million random numbers.
  NUMBERS_COUNT = 1_000_000

  # Write out to client after every 2000 objects.
  FLUSH_EVERY = 2_000

  def index
    lazy_enum = random_number_objects_lazy.take(NUMBERS_COUNT)

    stream_json_array(lazy_enum)
  end

  private

  def random_numbers_lazy
    (0..Float::INFINITY).lazy.map { SecureRandom.random_number(2**128) }
  end

  def random_number_objects_lazy
    random_numbers_lazy.each_with_index.map do |random_number, i|
      {
        index:  i,
        time:   Time.current.xmlschema(3),
        number: random_number
      }
    end
  end

  # Curl requires the --compressed flag for this response to load correctly.
  def stream_json_array(enum)
    headers["Content-Disposition"] = "attachment" # Download response to file. It's big.
    headers["Content-Type"]        = "application/json"
    headers["Content-Encoding"]    = "deflate"

    deflate = Zlib::Deflate.new

    buffer = "[\n  "
    enum.each_with_index do |object, i|
      buffer << ",\n  " unless i == 0
      buffer << JSON.pretty_generate(object, depth: 1)

      if i % FLUSH_EVERY == 0
        write(deflate, buffer)
        buffer = ""
      end
    end
    buffer << "\n]\n"

    write(deflate, buffer)
    write(deflate, nil) # Flush deflate.
    response.stream.close
  end

  def write(deflate, data)
    deflated = deflate.deflate(data)
    response.stream.write(deflated)
  end
end

License

This Streaming JSON Demo is dedicated to the public domain by its author, Brian Hempel. No rights are reserved. No restrictions are placed on the use of this code example. That freedom also means, of course, that no warrenty of fitness is claimed; use code from here at your own risk.

This public domain dedication follows the the CC0 1.0 at https://creativecommons.org/publicdomain/zero/1.0/

stream_json_demo's People

Contributors

brianhempel avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.