You can see the entire diff compared to a vanilla Rails 4.2.0 app. The app is also running on Heroku at http://stream-json-demo.herokuapp.com/random_numbers.
You can curl this demo with:
$ curl --compressed http://stream-json-demo.herokuapp.com/random_numbers
The app lazily generates a million random numbers and streams json to the client at the /random_numbers
endpoint:
HTTP/1.1 200 OK
Content-Type: application/json
Content-Encoding: deflate
Transfer-Encoding: chunked
[
{
"index": 0,
"time": "2015-03-12T16:24:59.381Z",
"number": 96712233280936019238290948377812066522
},
{
"index": 1,
"time": "2015-03-12T16:24:59.381Z",
"number": 322627977204921440486826283319053625299
},
{
"index": 2,
"time": "2015-03-12T16:24:59.381Z",
"number": 174374410324316249417283007292339235935
}
]
To stream the JSON, the app uses ActionController::Live
which allows us to send data to the client in chunks with HTTP chunked transfer encoding. The action also demonstrates compressing such a response with deflate. With this setup, you could (theoretically) stream an infinitely large response to the client without running out of memory.
The relevant file is app/controllers/random_numbers_controller.rb
:
class RandomNumbersController < ApplicationController
include ActionController::Live
# Return a million random numbers.
NUMBERS_COUNT = 1_000_000
# Write out to client after every 2000 objects.
FLUSH_EVERY = 2_000
def index
lazy_enum = random_number_objects_lazy.take(NUMBERS_COUNT)
stream_json_array(lazy_enum)
end
private
def random_numbers_lazy
(0..Float::INFINITY).lazy.map { SecureRandom.random_number(2**128) }
end
def random_number_objects_lazy
random_numbers_lazy.each_with_index.map do |random_number, i|
{
index: i,
time: Time.current.xmlschema(3),
number: random_number
}
end
end
# Curl requires the --compressed flag for this response to load correctly.
def stream_json_array(enum)
headers["Content-Disposition"] = "attachment" # Download response to file. It's big.
headers["Content-Type"] = "application/json"
headers["Content-Encoding"] = "deflate"
deflate = Zlib::Deflate.new
buffer = "[\n "
enum.each_with_index do |object, i|
buffer << ",\n " unless i == 0
buffer << JSON.pretty_generate(object, depth: 1)
if i % FLUSH_EVERY == 0
write(deflate, buffer)
buffer = ""
end
end
buffer << "\n]\n"
write(deflate, buffer)
write(deflate, nil) # Flush deflate.
response.stream.close
end
def write(deflate, data)
deflated = deflate.deflate(data)
response.stream.write(deflated)
end
end
This Streaming JSON Demo is dedicated to the public domain by its author, Brian Hempel. No rights are reserved. No restrictions are placed on the use of this code example. That freedom also means, of course, that no warrenty of fitness is claimed; use code from here at your own risk.
This public domain dedication follows the the CC0 1.0 at https://creativecommons.org/publicdomain/zero/1.0/