Giter VIP home page Giter VIP logo

puffing-billy's Introduction

Puffing Billy Gem Version Build Status

A rewriting web proxy for testing interactions between your browser and external sites. Works with ruby + rspec.

Puffing Billy is like webmock or VCR, but for your browser.

Overview

Billy spawns an EventMachine-based proxy server, which it uses to intercept requests sent by your browser. It has a simple API for configuring which requests need stubbing and what they should return.

Billy lets you test against known, repeatable data. It also allows you to test for failure cases. Does your twitter (or facebook/google/etc) integration degrade gracefully when the API starts returning 500s? Well now you can test it!

it 'should stub google' do
  proxy.stub('http://www.google.com/').and_return(text: "I'm not Google!")
  visit 'http://www.google.com/'
  expect(page).to have_content("I'm not Google!")
end

You can also record HTTP interactions and replay them later. See caching below.

Installation

Add this line to your application's Gemfile:

gem 'puffing-billy', group: :test

And then execute:

$ bundle

Or install it yourself as:

$ gem install puffing-billy

Setup for Capybara

In your rails_helper.rb:

require 'billy/capybara/rspec'

# select a driver for your chosen browser environment
Capybara.javascript_driver = :selenium_billy # Uses Firefox
# Capybara.javascript_driver = :selenium_headless_billy # Uses Firefox in headless mode
# Capybara.javascript_driver = :selenium_chrome_billy
# Capybara.javascript_driver = :selenium_chrome_headless_billy
# Capybara.javascript_driver = :apparition_billy
# Capybara.javascript_driver = :webkit_billy
# Capybara.javascript_driver = :poltergeist_billy
# Capybara.javascript_driver = :cuprite_billy

Note: :poltergeist_billy doesn't support proxying any localhosts, so you must use :webkit_billy, :apparition_billy, or a custom headless selenium registration for headless specs when using puffing-billy for other local rack apps. See this phantomjs issue for any updates.

Setup for Watir

In your rails_helper.rb:

require 'billy/watir/rspec'

# select a driver for your chosen browser environment
@browser = Billy::Browsers::Watir.new :firefox
# @browser = Billy::Browsers::Watir.new = :chrome
# @browser = Billy::Browsers::Watir.new = :phantomjs

Setup for Cucumber

An example feature:

Feature: Stubbing via billy

  @javascript @billy
  Scenario: Test billy
    And a stub for google

Setup for Cucumber + Capybara

In your features/support/env.rb:

require 'billy/capybara/cucumber'

After do
  Capybara.use_default_driver
end

And in steps:

Before('@billy') do
  Capybara.current_driver = :poltergeist_billy
end

And /^a stub for google$/ do
  proxy.stub('http://www.google.com/').and_return(text: "I'm not Google!")
  visit 'http://www.google.com/'
  expect(page).to have_content("I'm not Google!")
end

It's good practice to reset the driver after each scenario, so having an @billy tag switches the drivers on for a given scenario. Also note that stubs are reset after each step, so any usage of a stub should be in the same step that it was created in.

Setup for Cucumber + Watir

In your features/support/env.rb:

require 'billy/watir/cucumber'

After do
  @browser.close
end

And in steps:

Before('@billy') do
  @browser = Billy::Browsers::Watir.new :firefox
end

And /^a stub for google$/ do
  proxy.stub('http://www.google.com/').and_return(text: "I'm not Google!")
  @browser.goto 'http://www.google.com/'
  expect(@browser.text).to eq("I'm not Google!")
end

Setup remote Chrome

In the case you are using a Chrome instance, running on another machine, or in another Docker container, you need to :

  • Fix the Billy proxy host and port
  • Passes the --proxy-server=<billy host>:<billy port>

WebSockets

Puffing billy doesn't support websockets, so if you are using them, or ActionCable for the Ruby On Rails developers, you can tell Chrome to bypass the proxy for websockets by adding the flag --proxy-bypass-list=ws://* to your remote chrome intance or Docker container.

Minitest Usage

Please see this link for details and report back to Issue #49 if you get it fully working.

Examples

# Stub and return text, json, jsonp (or anything else)
proxy.stub('http://example.com/text/').and_return(text: 'Foobar')
proxy.stub('http://example.com/json/').and_return(json: { foo: 'bar' })
proxy.stub('http://example.com/jsonp/').and_return(jsonp: { foo: 'bar' })
proxy.stub('http://example.com/headers/').and_return(
  headers: { 'Access-Control-Allow-Origin' => '*' },
  json: { foo: 'bar' }
)
proxy.stub('http://example.com/wtf/').and_return(body: 'WTF!?', content_type: 'text/wtf')

# Stub redirections and other return codes
proxy.stub('http://example.com/redirect/').and_return(redirect_to: 'http://example.com/other')
proxy.stub('http://example.com/missing/').and_return(code: 404, body: 'Not found')

# Even stub HTTPS!
proxy.stub('https://example.com:443/secure/').and_return(text: 'secrets!!1!')

# Pass a Proc (or Proc-style object) to create dynamic responses.
#
# The proc will be called with the following arguments:
#   params:  Query string parameters hash, CGI::escape-style
#   headers: Headers hash
#   body:    Request body string
#   url:     The actual URL which was requested
#   method:  The HTTP verb which was requested
proxy.stub('https://example.com/proc/').and_return(Proc.new { |params, headers, body, url, method|
  {
    code: 200,
    text: "Hello, #{params['name'][0]}"
  }
})

# You can also use Puffing Billy to intercept requests and responses. Just pass
# a Proc and use the pass_request method. You can manipulate the request
# (headers, URL, HTTP method, etc) and also the response from the upstream
# server. The scope of the delivered callable is the user scope where
# it was defined. Setting method to 'all' will intercept requests regardless of
# the method.
proxy.stub('http://example.com/', method: 'all').and_return(Proc.new { |*args|
  response = Billy.pass_request(*args)
  response[:headers]['Content-Type'] = 'text/plain'
  response[:body] = 'Hello World!'
  response[:code] = 200
  response
})

# Stub out a POST. Don't forget to allow a CORS request and set the method to 'post'
proxy.stub('http://example.com/api', method: 'post').and_return(
  headers: { 'Access-Control-Allow-Origin' => '*' },
  code: 201
)

# Stub out an OPTIONS request. Set the headers to the values you require.
proxy.stub('http://example.com/api', method: 'options').and_return(
  headers: {
    'Access-Control-Allow-Methods' => 'GET, PATCH, POST, PUT, OPTIONS',
    'Access-Control-Allow-Headers' => 'X-Requested-With, X-Prototype-Version, Content-Type',
    'Access-Control-Allow-Origin'  => '*'
  },
  code: 200
)

Stubs are reset between tests. Any requests that are not stubbed will be proxied to the remote server.

If for any reason you'd need to reset stubs manually you can do it in two ways:

# reset a single stub
example_stub = proxy.stub('http://example.com/text/').and_return(text: 'Foobar')
proxy.unstub example_stub

# reset all stubs
proxy.reset

Caching

Requests routed through the external proxy are cached.

By default, all requests to localhost or 127.0.0.1 will not be cached. If you're running your test server with a different hostname, you'll need to add that host to puffing-billy's whitelist.

In your rails_helper.rb:

Billy.configure do |c|
  c.whitelist = ['test.host', 'localhost', '127.0.0.1'] # To replace the default whitelist, OR
  c.whitelist << 'mynewhost.local' # to append a host without overriding the defaults.
end

If you would like to cache other local rack apps, you must whitelist only the specific port for the application that is executing tests. If you are using Capybara, this can be accomplished by adding this in your rails_helper.rb:

server = Capybara.current_session.server
Billy.config.whitelist = ["#{server.host}:#{server.port}"]

If you would like to cache whitelisted URLs, you can define them in c.cache_whitelist. This is useful for scenarios where you may want to set c.non_whitelisted_requests_disabled to true to only allow whitelisted URLs to be accessed, but still allow specific URLs to be treated as if they were non-whitelisted.

If you want to use puffing-billy like you would VCR you can turn on cache persistence. This way you don't have to manually mock out everything as requests are automatically recorded and played back. With cache persistence you can take tests completely offline.

The cache works with all types of requests and will distinguish between different POST requests to the same URL.

Params

Billy.configure do |c|
  c.cache = true
  c.cache_request_headers = false
  c.ignore_params = ["http://www.google-analytics.com/__utm.gif",
                     "https://r.twimg.com/jot",
                     "http://p.twitter.com/t.gif",
                     "http://p.twitter.com/f.gif",
                     "http://www.facebook.com/plugins/like.php",
                     "https://www.facebook.com/dialog/oauth",
                     "http://cdn.api.twitter.com/1/urls/count.json"]
  c.path_blacklist = []
  c.merge_cached_responses_whitelist = []
  c.persist_cache = true
  c.ignore_cache_port = true # defaults to true
  c.non_successful_cache_disabled = false
  c.non_successful_error_level = :warn
  c.non_whitelisted_requests_disabled = false
  c.cache_path = 'spec/req_cache/'
  c.certs_path = 'spec/req_certs/'
  c.proxy_host = 'example.com' # defaults to localhost
  c.proxy_port = 12345 # defaults to random
  c.proxied_request_host = nil
  c.proxied_request_port = 80
  c.cache_whitelist = []
  c.record_requests = true # defaults to false
  c.cache_request_body_methods = ['post', 'patch', 'put'] # defaults to ['post']
end
  • c.cache_request_headers is used to store the outgoing request headers in the cache. It is also saved to yml if persist_cache is enabled. This additional information is useful for debugging (for example: viewing the referer of the request).

  • c.ignore_params is used to ignore parameters of certain requests when caching. You should mostly use this for analytics and various social buttons as they use cache avoidance techniques, but return practically the same response that most often does not affect your test results.

  • c.path_blacklist = [] is used to always cache specific paths on any hostnames, including whitelisted ones. This is useful if your AUT has routes that get data from external services, such as /api where the ajax request is a local URL but the actual data is coming from a different application that you want to cache.

  • c.merge_cached_responses_whitelist = [] is used to group together the cached responses for specific uri regexes that match any part of the url. This is useful for ensuring that any kind of analytics and various social buttons that have slightly different urls each time can be recorded once and reused nicely. Note that the request body is ignored for requests that contain a body.

  • c.ignore_cache_port is used to strip the port from the URL if it exists. This is useful when caching local paths (via path_blacklist) or other local rack apps that are running on random ports.

  • c.non_successful_cache_disabled is used to not cache responses without 200-series or 304 status codes. This prevents unauthorized or internal server errors from being cached and used for future test runs.

  • c.non_successful_error_level is used to log when non-successful responses are received. By default, it just writes to the log file, but when set to :error it throws an error with the URL and status code received for easier debugging.

  • c.non_whitelisted_requests_disabled is used to disable hitting new URLs when no cache file exists. Only whitelisted URLs (on non-blacklisted paths) are allowed, all others will throw an error with the URL attempted to be accessed. This is useful for debugging issues in isolated environments (ie. continuous integration).

  • c.cache_path can be used to locate the cache directory to a different place other than system temp directory/puffing-billy.

  • c.certs_path can be used to locate the directory for dynamically generated SSL certificates to a different place other than system temp directory/puffing-billy/certs.

  • c.proxy_host and c.proxy_port are used for the Billy proxy itself which runs locally.

  • c.proxied_request_host and c.proxied_request_port are used if an internal proxy server is required to access the internet. Most common in larger companies.

  • c.allow_params is used to allow parameters of certain requests when caching. This is best used when a site has a large number of analytics and social buttons. c.allow_params is the opposite of c.ignore_params, a whitelist to a blacklist. In order to toggle between using one or the other, use c.use_ignore_params.

  • c.strip_query_params is used to strip query parameters when you stub some requests with query parameters. Default value is true. For example, proxy.stub('http://myapi.com/user/?country=FOO') is considered the same as: proxy.stub('http://myapi.com/user/?anything=FOO') and generally the same as: proxy.stub('http://myapi.com/user/'). When you need to distinguish between all these requests, you may set this config value to false.

  • c.dynamic_jsonp is used to rewrite the body of JSONP responses based on the callback parameter. For example, if a request to http://example.com/foo?callback=bar returns bar({"some": "json"}); and is recorded, then a later request to http://example.com/foo?callback=baz will be a cache hit and respond with baz({"some": "json"}); This is useful because most JSONP implementations base the callback name off of a timestamp or something else dynamic.

  • c.dynamic_jsonp_keys is used to configure which parameters to ignore when using c.dynamic_jsonp. This is helpful when JSONP APIs use cache-busting parameters. For example, if you want http://example.com/foo?callback=bar&id=1&cache_bust=12345 and http://example.com/foo?callback=baz&id=1&cache_bust=98765 to be cache hits for each other, you would set c.dynamic_jsonp_keys = ['callback', 'cache_bust'] to ignore both params. Note that in this example the id param would still be considered important.

  • c.dynamic_jsonp_callback_name is used to configure the name of the JSONP callback parameter. The default is callback.

  • c.record_requests can be used to record all requests that puffing billy proxied. This can be useful for debugging purposes, for instance if you are unsure why your stubbed requests are not being successfully proxied.

  • c.cache_request_body_methods is used to specify HTTP methods of requests that you would like to cache separately based on the contents of the request body. The default is ['post'].

  • c.use_ignore_params is used to choose whether to use the ignore_params blacklist or the allow_params whitelist. Set to true to use c.ignore_params, false to use c.allow_params

  • c.before_handle_request is used to modify method, url, headers, body before handle request by stubs, cache or proxy. Method accept 4 argumens and must return array of this arguments:

    c.before_handle_request = proc { |method, url, headers, body|
      filtered_body = JSON.dump(filter_secret_data(JSON.load(body)))
      [method, url, headers, filtered_body]
    }
  • c.cache_simulates_network_delays is used to add some delay before cache returns response. When set to true, cached requests will wait from configured delay time before responding. This allows to catch various race conditions in asynchronous front-end requests. The default is false.

  • c.cache_simulates_network_delay_time is used to configure time (in seconds) to wait until responding from cache. The default is 0.1.

  • c.after_cache_handles_request is used to configure a callback that can operate on the response after it has been retrieved from the cache but before it is returned. The callback receives the request and response as arguments, with a request object like: { method: method, url: url, headers: headers, body: body }. An example usage would be manipulating the Access-Control-Allow-Origin header so that your test server doesn't always have to run on the same port in order to accept cached responses to CORS requests:

    Billy.configure do |c|
      ...
      fix_cors_header = proc do |_request, response|
        allowed_origins = response[:headers]['Access-Control-Allow-Origin']
        if allowed_origins.present?
          localhost_port_pattern = %r{(?<=http://127\.0\.0\.1:)(\d+)}
          allowed_origins.sub!(
            localhost_port_pattern, Capybara.current_session.server.port.to_s
          )
        end
      end
      c.after_cache_handles_request = fix_cors_header
      ...
    end

Example usage:

require 'table_print' # Add this dependency to your gemfile

Billy.configure do |c|
  c.record_requests = true
end

RSpec.configure do |config|
  config.prepend_after(:example, type: :feature) do
    puts "Requests received via Puffing Billy Proxy:"

    puts TablePrint::Printer.table_print(Billy.proxy.requests, [
      :status,
      :handler,
      :method,
      { url: { width: 100 } },
      :headers,
      :body
    ])
  end
end

This will generate a human readable list of all requests after each test run:

Requests received via Puffing Billy Proxy:
STATUS   | HANDLER | METHOD  | URL                                     | HEADERS                        | BODY
---------|---------|---------|-----------------------------------------|--------------------------------|-----
complete | proxy   | GET     | http://127.0.0.1:56692/                 | {"Accept"=>"text/html,appli... |
complete | proxy   | GET     | http://127.0.0.1:56692/assets/appl...   | {"Accept"=>"text/css,*/*;q=... |
complete | proxy   | GET     | http://127.0.0.1:56692/assets/app...    | {"Accept"=>"*/*", "Referer"... |
complete | proxy   | GET     | http://127.0.0.1:56692/javascript/index | {"Accept"=>"text/html,appli... |
complete | stubs   | OPTIONS | https://api.github.com:443/             | {"Access-Control-Request-Me... |
complete | stubs   | GET     | https://api.github.com:443/             | {"Accept"=>"*/*", "Referer"... |
inflight |         | GET     | http://127.0.0.1:56692/example          | {"Referer"=>"http://127.0.0... |
.

Finished in 1.98 seconds (files took 2.11 seconds to load)
1 example, 0 failures

The handler column indicates how Puffing Billy handled your request:

  • proxy: This request was successfully routed to the original target
  • stubs: This was handled via a stub
  • error: This request was not handled by a stub, and was not successfully handled
  • cache: This response was handled by a previous cache

If your status is set to inflight this request has not yet been handled fully. Either puffing billy crashed internally on this request, or your test ended before it could complete successfully.

Cache Scopes

If you need to cache different responses to the same HTTP request, you can use cache scoping.

For example, an index page may return zero or more items in a list, with or without pagination, depending on the number of entries in a database.

There are a few different ways to use cache scopes:

# If you do nothing, it uses the default cache scope:
it 'defaults to nil scope' do
  expect(proxy.cache.scope).to be_nil
end

# You can change context indefinitely to a specific cache scope:
context 'with a cache scope' do
  before do
    proxy.cache.scope_to "my_cache"
  end

  # Remember to set the cache scope back to the default in an after block
  # within the context it is used, and/or at the global rails_helper level!
  after do
    proxy.cache.use_default_scope
  end

  it 'uses the cache scope' do
    expect(proxy.cache.scope).to eq("my_cache")
  end

  it 'can be reset to the default scope' do
    proxy.cache.use_default_scope
    expect(proxy.cache.scope).to be_nil
  end

  # Or you can run a block within the context of a cache scope:
  # Note: When using scope blocks, be sure that both the action that triggers a
  #       request and the assertion that a response has been received are within the block
  it 'can execute a block against a named cache' do
    expect(proxy.cache.scope).to eq("my_cache")
    proxy.cache.with_scope "another_cache" do
      expect(proxy.cache.scope).to eq "another_cache"
    end
    # It
    expect(proxy.cache.scope).to eq("my_cache")
  end
end

If you use named caches it is highly recommend that you use a global hook to set the cache back to the default before or after each test.

In Rspec:

RSpec.configure do |config|
  config.before :each { proxy.cache.use_default_scope }
end

Separate Cache Directory for Each Test

If you want the cache for each test to be independent, i.e. have it's own directory where the cache files are stored, you can do so.

in Cucumber

use a Before tag:

Before('@javascript') do |scenario, block|
  Billy.configure do |c|
    feature_name = scenario.feature.name.underscore
    scenario_name = scenario.name.underscore
    c.cache_path = "features/support/fixtures/req_cache/#{feature_name}/#{scenario_name}/"
    FileUtils.mkdir_p(Billy.config.cache_path) unless File.exist?(Billy.config.cache_path)
  end
end

in Rspec

use a before(:each) block:

RSpec.configure do |config|
  base_cache_path = Billy.config.cache_path
  base_certs_path = Billy.config.certs_path
  config.before :each do |x|
    feature_name = x.metadata[:example_group][:description].underscore.gsub(' ', '_')
    scenario_name = x.metadata[:description].underscore.gsub(' ', '_')
    Billy.configure do |c|
      cache_scenario_folder_path = "#{base_cache_path}/#{feature_name}/#{scenario_name}/"
      FileUtils.mkdir_p(cache_scenario_folder_path) unless File.exist?(cache_scenario_folder_path)
      c.cache_path = cache_scenario_folder_path

      certs_scenario_folder_path = "#{base_certs_path}/#{feature_name}/#{scenario_name}/"
      FileUtils.mkdir_p(certs_scenario_folder_path) unless File.exist?(certs_scenario_folder_path)
      c.certs_path = certs_scenario_folder_path
    end
  end
end

Stub requests recording

If you want to record requests to stubbed URIs, set the following configuration option:

Billy.configure do |c|
  c.record_stub_requests = true
end

Example usage:

it 'should intercept a GET request' do
  stub = proxy.stub('http://example.com/')
  visit 'http://example.com/'
  expect(stub.has_requests?).to be true
  expect(stub.requests).not_to be_empty
end

Proxy timeouts

By default, the Puffing Billy proxy will use the EventMachine::HttpRequest timeouts of 5 seconds for connect and 10 seconds for inactivity when talking to downstream servers.

These can be configured as follows:

Billy.configure do |c|
  c.proxied_request_connect_timeout = 20
  c.proxied_request_inactivity_timeout = 20
end

Customising the javascript driver

If you use a customised Capybara driver, remember to set the proxy address and tell it to ignore SSL certificate warnings. See lib/billy.rb to see how Billy's default drivers are configured.

Working with VCR and Webmock

If you use VCR and Webmock elsewhere in your specs, you may need to disable them for your specs utilizing Puffing Billy. To do so, you can configure your rails_helper.rb as shown below:

RSpec.configure do |config|
  config.around(:each, type: :feature) do |example|
    WebMock.allow_net_connect!
    VCR.turned_off { example.run }
    WebMock.disable_net_connect!
  end
end

As an alternative if you're using VCR, you can ignore requests coming from the browser. One way of doing that is by adding to your rails_helper.rb the excerpt below:

VCR.configure do |config|
  config.ignore_request do |request|
    request.headers.include?('Referer')
  end
end

Note that this approach may cause unexpected behavior if your backend sends the Referer HTTP header (which is unlikely).

Raising errors from stubs

By default Puffing Billy suppresses errors from stub-blocks. To make it raise errors instead, add this test initializers:

EM.error_handler { |e| raise e }

SSL usage

Unfortunately we cannot setup the runtime certificate authority on your browser at time of configuring the Capybara driver. So you need to take care of this step yourself as a prepartion. A good point would be directly after configuring this gem.

Google Chrome Headless example

Google Chrome/Chromium is capable to run as a test browser with the new headless mode which is not able to handle the deprecated --ignore-certificate-errors flag. But the headless mode is capable of handling the user PKI certificate store. So you just need to import the runtime Puffing Billy certificate authority on your system store, or generate a new store for your current session. The following examples demonstrates the former variant:

# Install the fabulous `os` gem first
# See: https://rubygems.org/gems/os
# gem install os
#
# --

# Overwrite the local home directory for chrome. We use this
# to setup a custom SSL certificate store.
ENV['HOME'] = "#{Dir.tmpdir}/chrome-home-#{Time.now.to_i}"

# Clear and recreate the Chrome home directory.
FileUtils.rm_rf(ENV['HOME'])
FileUtils.mkdir_p(ENV['HOME'])

# Setup a new pki certificate database for Chrome
if OS.linux?
  system <<~SCRIPT
    cd "#{ENV['HOME']}"
    curl -s -k -o "cacert-root.crt" "http://www.cacert.org/certs/root.crt"
    curl -s -k -o "cacert-class3.crt" "http://www.cacert.org/certs/class3.crt"
    echo > .password
    mkdir -p .pki/nssdb
    CERT_DIR=sql:$HOME/.pki/nssdb
    certutil -N -d .pki/nssdb -f .password
    certutil -d ${CERT_DIR}  -A -t TC \
      -n "CAcert.org" -i cacert-root.crt
    certutil -d ${CERT_DIR} -A -t TC \
      -n "CAcert.org Class 3" -i cacert-class3.crt
    certutil -d sql:$HOME/.pki/nssdb -A \
      -n puffing-billy -t "CT,C,C" -i #{Billy.certificate_authority.cert_file}
  SCRIPT
end

# Setup the macOS certificate store
if OS.mac?
  prompt = 'Add Puffing Billy root certificate authority ' \
           'to system certificate store'
  system <<~SCRIPT
    sudo -p "# #{prompt}`echo $'\nPassword: '`" \
      security find-certificate -a -Z -c 'Puffing Billy' \
      | grep 'SHA-1 hash' | cut -d ':' -f2 | xargs -n1 \
      sudo security delete-certificate -Z >/dev/null 2>&1 || true
    sudo security add-trusted-cert -d -r trustRoot \
      -k /Library/Keychains/System.keychain \
      #{Billy.certificate_authority.cert_file}
  SCRIPT
end

Mind the reset of the HOME environment variable. Fortunately Chrome takes care of the users home, so we can setup a new temporary directory for the test run, without messing with potential user configurations.

The macOS support requires the input of your password to manipulate the system certificate store. If you are lazy you can turn off sudo password prompt for the security command, but it's strongly advised against. (You know passwordless security, is no security in this case) Further, the macOS handling here cleans up old Puffing Billy root certificate authorities and put the current one into the system store. So after a run of your the suite only one certificate will be left over. If this is not enough you can handling the cleanup again with a custom on-after hook.

TLS hostname validation

em-http-request was modified to emit a warning if being used without the TLS verify_peer option. Puffing Billy defaults to specifying verify_peer: false but you can now modify configuration to do peer verification. So if you've gone to the trouble of setting up your own certificate authority and self-signed certs you can enable it like so:

Billy.configure do |c|
  c.verify_peer = true
end

Resources

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Added some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

TODO

  1. Integration for test frameworks other than RSpec.
  2. Show errors from the EventMachine reactor loop in the test output.

puffing-billy's People

Contributors

ahharvey avatar alanfoster avatar algodave avatar bwilczek avatar cupakromer avatar eliotsykes avatar fohara avatar foxtree avatar gkopylov avatar gustavosobral avatar jack12816 avatar jonrowe avatar ksylvest avatar mackermans avatar michaelrkn avatar msjonker avatar oesmith avatar phillbaker avatar rimian avatar ronwsmith avatar rounders avatar ryansch avatar sonjapeterson avatar stwr667 avatar swizec avatar tansaku avatar tomlea avatar twalpole avatar vfonic avatar xtrasimplicity avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

puffing-billy's Issues

setting a proxy stub for regex urls

Currently, I can't seem to use regex to stub out requests to ad servers, such as

proxy.stub(/doubleclick.net/).and_return(nil)

I'm coming from a WebMock background where I could just do

stub_request(:any, /doubeclick.net/)

and if I did any kind of Net::HTTP.get('http://blablabla.doubleclick.net/blabla', '/')
it would return nil, b/c of the previous stub.

Any idea if this is possible w/ puffing-billy?

Control cache yml name

VCR let's you control and decide the cassette name for each test if you want - do you guys feel this would be desirable for the persistent cache setup in puffing-billy?

The reason I ask is that I am making four calls to Stripe and it caches the first one and doesn't understand the next three are separate tests with new data, so they fail when they replay the wrong interaction.

Trailing slash is required

This fails:

proxy.stub('http://www.facebook.com').and_return(:text => 'Foobar')
visit 'http://www.facebook.com'
page.should have_content "Foobar"

This succeeds:

proxy.stub('http://www.facebook.com/').and_return(:text => 'Foobar')
visit 'http://www.facebook.com/'
page.should have_content "Foobar"

A trailing slash is required, which is easy to miss.

Silencing handle_request errors due to not being in the whitelist

Hey guys, so ... I wanted to use puffing-billy in a way where I can simply set a whitelist, and have it only allow requests to those host:port's.

So, when I set
Billy.config.non_whitelisted_requests_disabled = true
This basically makes my cucumber output return all of the puffing-billy error's from the handle_request method, it doesn't mess with my test at all, but I was wondering if there's a way to supress that, or a setting that I'm missing?

I don't want to cache things, I'd prefer just not allowing external calls to be made, and ignoring the errors.

Basically the errors that show up are like this:

puffing-billy: Connection to http://fonts.googleapis.com/css?family=Open+Sans:300italic,400italic,700italic,400,300,700 not cached and new http connections are disabled
/Users/josiahanjos/.rvm/gems/ruby-2.1.0/bundler/gems/puffing-billy-47b7502bcb1e/lib/billy/proxy_connection.rb:65:in block (2 levels) in handle_request' /Users/josiahanjos/.rvm/gems/ruby-2.1.0/bundler/gems/puffing-billy-47b7502bcb1e/lib/billy/proxy_connection.rb:62:intap'
/Users/josiahanjos/.rvm/gems/ruby-2.1.0/bundler/gems/puffing-billy-47b7502bcb1e/lib/billy/proxy_connection.rb:62:in block in handle_request' /Users/josiahanjos/.rvm/gems/ruby-2.1.0/gems/em-synchrony-1.0.3/lib/em-synchrony.rb:29:inblock in synchrony'
RuntimeError

I read the description for the c.non_whitelisted_requests_disabled and you guys say it's intended for debugging, which makes sense.
But in my case, I think it'd be preferable to have a whitelist, instead of a blacklist, or a list of stubs to external requests.
Would it be possible to add a new config line with... c.suppressed_errors = false # default
Where if I set it to true, these errors aren't printed?

How to intercept all external calls from JavaScript, not just those initiated by a headless browser

I use the PhantomJS gem in my app.

https://github.com/colszowka/phantomjs-gem

I use it to execute PhantomJS code from Ruby, like this:

Phantomjs.run("./app/assets/javascripts/trigger.js", file.path).html_safe

This triggers a PhantomJS script that interacts with Google' Google Charts API.

Likewise, I use the Screencap gem.

https://github.com/maxwell/screencap

This triggers a PhantomJS script that grabs an entire webpage for screenshotting.

Is there any way to intercept these external calls within Rspec using puffing-billy?

The docs give the impression that this is solely for use within headless browsers driven by Capybara or similar.

Inconsistency between :poltergeist and :poltergeist_billy ?

Test suite runs fine with :poltergeist, with :poltergeist_billy (and no other billy config), I get this:

  1) User updates profile with valid gender
     Failure/Error: sign_in user
     Capybara::Poltergeist::JavascriptError:
       One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

       TypeError: 'undefined' is not a constructor (evaluating 'new google.maps.LatLng(52.5122, 13.4194)')
           at http://127.0.0.1:59829/assets/jquery/address-picker.js:176
...

Allowing pass through for certain URLs

Hi there,

Amazing product.

As far as I can see there is no way of doing a "partial" mock - only caching certain services and then passing through for other services. Billy.config.non_whitelisted_requests_disabled will cause an error to be thrown but the behaviour I want is for something like Billy.config.mock_request_paths and only things on that list will be cached - everything else will pass through.

This is in order to prevent stuff I don't really care about like images from being cached. Bonus round: smarter regex for excluding categories of things.

Willing to work on this, just wondered if anyone else was.

Content-length header is occasionally missing/multiple?

Intermittently while running our cucumber tests, we see a test fail for an undetermined reason. Digging deeper it looks like there are issues with the headers being sent from puffing-billy. The following is part of a chrome://net-internals report for a request that is made during the test. All of the requests matching this URL exhibit the behavior for the duration of the test.

t= 0 [st= 0] +REQUEST_ALIVE  [dt=15]
t= 1 [st= 1]    URL_REQUEST_DELEGATE  [dt=0]
t= 1 [st= 1]   +URL_REQUEST_START_JOB  [dt=13]
                --> load_flags = 335609856 (BYPASS_DATA_REDUCTION_PROXY | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
                --> method = "GET"
                --> priority = "LOWEST"
                --> url = "https://autocomplete-api.smartystreets.com/suggest?callback=jQuery1102012385960388928652_1421181402248&auth-id=ELIDED&prefix=This+is+a+fake+street+addres&city_filter=&state_filter=&prefer=&suggestions=10&geolocate=true&_=1421181402250"
t= 1 [st= 1]      URL_REQUEST_DELEGATE  [dt=0]
t= 1 [st= 1]      HTTP_CACHE_GET_BACKEND  [dt=0]
t= 1 [st= 1]      HTTP_CACHE_OPEN_ENTRY  [dt=0]
                  --> net_error = -2 (ERR_FAILED)
t= 1 [st= 1]      HTTP_CACHE_CREATE_ENTRY  [dt=0]
t= 1 [st= 1]      HTTP_CACHE_ADD_TO_ENTRY  [dt=0]
t= 1 [st= 1]      URL_REQUEST_DELEGATE  [dt=0]
t= 1 [st= 1]     +HTTP_STREAM_REQUEST  [dt=10]
t=11 [st=11]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                    --> source_dependency = 732 (HTTP_STREAM_JOB)
t=11 [st=11]     -HTTP_STREAM_REQUEST
t=11 [st=11]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=1]
t=11 [st=11]        HTTP_TRANSACTION_SEND_REQUEST_HEADERS
                    --> GET /suggest?callback=jQuery1102012385960388928652_1421181402248&auth-id=ELIDED&prefix=This+is+a+fake+street+addres&city_filter=&state_filter=&prefer=&suggestions=10&geolocate=true&_=1421181402250 HTTP/1.1
                        Host: autocomplete-api.smartystreets.com
                        Connection: keep-alive
                        Accept: */*
                        User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36
                        Referer: http://localhost:3000/admin/orders/1/sidecar_jobs/new
                        Accept-Encoding: gzip, deflate, sdch
                        Accept-Language: en-US,en;q=0.8
t=12 [st=12]     -HTTP_TRANSACTION_SEND_REQUEST
t=12 [st=12]     +HTTP_TRANSACTION_READ_HEADERS  [dt=2]
t=12 [st=12]        HTTP_STREAM_PARSER_READ_HEADERS  [dt=2]
                    --> net_error = -346 (ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_LENGTH)
t=14 [st=14]     -HTTP_TRANSACTION_READ_HEADERS
                  --> net_error = -346 (ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_LENGTH)
t=14 [st=14]   -URL_REQUEST_START_JOB
                --> net_error = -346 (ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_LENGTH)
t=15 [st=15]    URL_REQUEST_DELEGATE  [dt=0]
t=15 [st=15] -REQUEST_ALIVE
              --> net_error = -346 (ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_LENGTH)

Notice the ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_LENGTH message. The following is a successful request:

t= 0 [st= 0] +REQUEST_ALIVE  [dt=15]
t= 0 [st= 0]    URL_REQUEST_DELEGATE  [dt=0]
t= 0 [st= 0]   +URL_REQUEST_START_JOB  [dt=14]
                --> load_flags = 335609856 (BYPASS_DATA_REDUCTION_PROXY | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
                --> method = "GET"
                --> priority = "LOWEST"
                --> url = "https://autocomplete-api.smartystreets.com/suggest?callback=jQuery110207462107751052827_1421180739735&auth-id=ELIDED&prefix=This+is+a+fake+street+addres&city_filter=&state_filter=&prefer=&suggestions=10&geolocate=true&_=1421180739737"
t= 1 [st= 1]      URL_REQUEST_DELEGATE  [dt=0]
t= 1 [st= 1]      HTTP_CACHE_GET_BACKEND  [dt=0]
t= 1 [st= 1]      HTTP_CACHE_OPEN_ENTRY  [dt=0]
                  --> net_error = -2 (ERR_FAILED)
t= 1 [st= 1]      HTTP_CACHE_CREATE_ENTRY  [dt=0]
t= 1 [st= 1]      HTTP_CACHE_ADD_TO_ENTRY  [dt=0]
t= 1 [st= 1]      URL_REQUEST_DELEGATE  [dt=0]
t= 1 [st= 1]     +HTTP_STREAM_REQUEST  [dt=10]
t=11 [st=11]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                    --> source_dependency = 702 (HTTP_STREAM_JOB)
t=11 [st=11]     -HTTP_STREAM_REQUEST
t=11 [st=11]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=1]
t=11 [st=11]        HTTP_TRANSACTION_SEND_REQUEST_HEADERS
                    --> GET /suggest?callback=jQuery110207462107751052827_1421180739735&auth-id=ELIDED&prefix=This+is+a+fake+street+addres&city_filter=&state_filter=&prefer=&suggestions=10&geolocate=true&_=1421180739737 HTTP/1.1
                        Host: autocomplete-api.smartystreets.com
                        Connection: keep-alive
                        Accept: */*
                        User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36
                        Referer: http://localhost:3000/admin/orders/1/sidecar_jobs/new
                        Accept-Encoding: gzip, deflate, sdch
                        Accept-Language: en-US,en;q=0.8
t=12 [st=12]     -HTTP_TRANSACTION_SEND_REQUEST
t=12 [st=12]     +HTTP_TRANSACTION_READ_HEADERS  [dt=2]
t=12 [st=12]        HTTP_STREAM_PARSER_READ_HEADERS  [dt=2]
t=14 [st=14]        HTTP_TRANSACTION_READ_RESPONSE_HEADERS
                    --> HTTP/1.1 200 ...
                        Access-Control-Allow-Origin: *
                        Connection: close
                        Content-Length: 351
                        Content-Type: application/json
                        Content-length: 351
                        Date: Thu, 08 Jan 2015 18:31:15 GMT
                        Server: nginx/1.7.6
t=14 [st=14]     -HTTP_TRANSACTION_READ_HEADERS
t=14 [st=14]      HTTP_CACHE_WRITE_INFO  [dt=0]
t=14 [st=14]      URL_REQUEST_DELEGATE  [dt=0]
t=14 [st=14]   -URL_REQUEST_START_JOB
t=14 [st=14]    URL_REQUEST_DELEGATE  [dt=1]
t=15 [st=15]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=15 [st=15]    HTTP_TRANSACTION_READ_BODY  [dt=0]
t=15 [st=15] -REQUEST_ALIVE

Notice the Content-length header is present here. There was nothing changed about the test.

Looking at the response at this point (

res.content = response[:content]
) it appeared that there weren't any content-length headers set. I saw that the evma_httpserver HTTPResponse class implements a #fixup_headers method (https://github.com/eventmachine/evma_httpserver/blob/231d2bd10a397e73025150b3fc346ebcd4305f9e/lib/evma_httpserver/response.rb#L194). I'm not sure if that has anything to do with it.

I'd appreciate any help that can be provided, it's frustrating when the CI builds fail intermittently.

HTTPS must have port specified

This fails:

proxy.stub('https://www.facebook.com/').and_return(:text => 'Foobar')
visit 'https://www.facebook.com/'
page.should have_content "Foobar"

This succeeds:

proxy.stub('https://www.facebook.com:443/').and_return(:text => 'Foobar')
visit 'https://www.facebook.com/'
page.should have_content "Foobar"

HTTPS requires the ':443' port part of the URL. This is correct but could be mentioned in an example to ease adoption.

Start puffing-billy with the same port

Hey,

I'm using puffing-billy to test a frontend applications fetching informations from an API. I've CORS activated in the server side.

The following header is saved in the cassettes: Access-Control-Allow-Origin: http://127.0.0.1:59533
So I imagine it's because puffing-billy starts with this port.

If I run the spec another time, I've the following error: XMLHttpRequest cannot load http://mon-domain.com:9292/auth/github. Origin http://127.0.0.1:59880 is not allowed by Access-Control-Allow-Origin. The port is now 59880.

There is a way to fix the port of puffing-billy? I tried to use the following configuration: config.proxy_port = 59533 but it doesn't change anything.

Thanks for your help :)

Rake tasks are not working

I noticed there are a few rake tasks that are included by this gem, which I can see when I run bundle exec rake -T, but I cannot use them for whatever reason. Here is the error message I get:

 rake aborted!
 Don't know how to build task 'cache:find_non_successful'
 /Users/newuser/.rvm/gems/ruby-2.1.1@global/gems/rake-10.3.2/lib/rake/task_manager.rb:62:in `[]'

I'm using Ruby 2.1.1 and Rails 4.1.5

Built-in SSL certificate already expired

I can't get pass this screen when testing with Selenium (even after adding it to security exception). Facebook and Google sites seem not to be affected though. I guess the certificate in lib/billy/mitm.crt needs to be updated. Thanks.

firefox

Tested on Firefox v28.0.

Performance issues with puffing-billy

Hey guys, just had a quick question about your experience with puffing-billy... so... I had hoped that using puffing-billy with our testing stack would have sped things up, since we wouldn't be making external requests any more, nor waiting for those to load... Currently those requests are not being made, I sort of disabled cache in a branch of my own, not just setting it to false, but completely keeping it from running, so there would be no extra code running that I wasn't using.

I was just wondering if you guys experience that type of slowness as well, is it due to all the requests being proxied through EM and not just directly to the app?

My runs without puffing-billy used to be ~1minute 20seconds, now with it, they're closer to 4 minutes.

I am running my tests with garbage collection on, which is necessary because if we defer it til the end, our travis runs blow up and the process gets killed.

However; I was seeing this slow down even previously to turning GC on.

Do you guys have any insight into this?

Unable to load URL: http://127.0.0.1:XXXXX

I'm a newcomer to puffing-billy so bear with me! I suspect there's something obvious there, but cannot find a way to make this work.

I attempted to enable puffing-billy on a capybara-webkit-enabled rspec feature. Note that I had to revert to an older RSpec version (see #19) to get that far.

As soon as I run the following spec (extract):

require 'spec_helper'

feature 'Sign-up' do
  scenario 'as a regular user', focus: true, js: true, driver: :webkit_billy do
    visit '/'
    # ...
  end
end

I get the following stack trace:

Failure/Error: visit '/'
     Capybara::Webkit::InvalidResponseError:
       Unable to load URL: http://127.0.0.1:58597/ because of error loading http://127.0.0.1:58597/: Connection refused
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/capybara-webkit-0.14.2/lib/capybara/webkit/browser.rb:205:in `check'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/capybara-webkit-0.14.2/lib/capybara/webkit/browser.rb:142:in `command'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/capybara-webkit-0.14.2/lib/capybara/webkit/browser.rb:18:in `visit'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/capybara-webkit-0.14.2/lib/capybara/webkit/driver.rb:29:in `visit'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/capybara-2.0.3/lib/capybara/session.rb:183:in `visit'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/capybara-2.0.3/lib/capybara/dsl.rb:51:in `block (2 levels) in <module:DSL>'
     # ./spec/features/sign_up_spec.rb:9:in `block (2 levels) in <top (required)>'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/example.rb:113:in `instance_eval'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/example.rb:113:in `block in run'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/example.rb:253:in `with_around_each_hooks'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/example.rb:110:in `run'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/example_group.rb:378:in `block in run_examples'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/example_group.rb:374:in `map'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/example_group.rb:374:in `run_examples'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/example_group.rb:360:in `run'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/command_line.rb:28:in `block (2 levels) in run'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/command_line.rb:28:in `map'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/command_line.rb:28:in `block in run'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/reporter.rb:34:in `report'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/command_line.rb:25:in `run'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/runner.rb:69:in `run'
     # /Users/thbar/.rvm/gems/ruby-1.9.3-p392-railsexpress@myapp/gems/rspec-core-2.11.1/lib/rspec/core/runner.rb:8:in `block in autorun'

Is there anyone with an idea here?

Thanks!

Cannot load such file -- rspec

I followed the instructions in the README and encounter this error when I try to run my specs:
Require: cannot load such file -- rspec (LoadError)

The error comes from /lib/billy/rspec.rb:1

My spec_helper.rb requires the following:

require 'rubygems'
require 'spork'
require 'capybara/rspec'
require 'capybara/poltergeist'
require 'billy/rspec'

Any advice?

Nested attributes with dynamic_jsonp_keys

I'm using the Balanced Payments API in a project. A typical request has parameters that look like this (converted to JSON for readability):

"data": {
  "name": "Example Brown 1",
  "number": "4111111111111111",
  "expiration_month": "12",
  "expiration_year": "2020",
  "cvv": "123",
  "address": {
    "postal_code": "11211"
  },
  "meta": {
    "capabilities_system_timezone": -8,
    "capabilities_user_agent": "Mozilla\/5.0 (Macintosh; PPC Mac OS X) AppleWebKit\/534.34 (KHTML, like Gecko) PhantomJS\/1.9.7 Safari\/534.34",
    "capabilities_language": "en-US",
    "capabilities_kp": 45,
    "capabilities_cli": 1,
    "capabilities_loaded": 1418668507849,
    "capabilities_screen_width": 2560,
    "capabilities_screen_length": 1440,
    "capabilities_hist": 1,
    "capabilities_cookie": "1418668507849.7760353146586567.1!0",
    "capabilities_cl": false,
    "capabilities_submitted": 1418668507899,
    "capabilities_scrollX": 0,
    "capabilities_scrollY": 0
  }
}

Because all of the keys are nested within the "data" key, using dynamic_jsonp_keys is very blunt - I have to either ignore everything or nothing. It would be nice to be able to specify nested keys to ignore, e.g.:

Billy.configure do |c|
  c.dynamic_jsonp_keys = ["callback", "data"["meta"]]
end

or something along those lines. I'm not exactly sure what the right interface would be.

The relevant code is at https://github.com/oesmith/puffing-billy/blob/master/lib/billy/cache.rb#L99.

Stack Trace in cache lookup

I get the following stack trace...

The http parser returns the uri part, so the cache code get a null when it tries to construct the full url path.
@header_data in proxy connection shows this:
GET /v1/someobjects?id=1b0d1735-96ba-4d8f-9da7-71e03f5e5fbe HTTP/1.1\r\nHost: 10.0.0.2:9999\r\nConnection: keep-alive\r\nAccept: /\r\nUser-Agent: BLAH/1.0\r\n\r\n"

Stack Trace seen:

NoMethodError
undefined method +' for nil:NilClass /Users/someuser/.rvm/gems/ruby-1.9.3-p392/gems/puffing-billy-0.2.1/lib/billy/cache.rb:74:inkey'
/Users/someuser/.rvm/gems/ruby-1.9.3-p392/gems/puffing-billy-0.2.1/lib/billy/cache.rb:21:in cached?' /Users/someuser/.rvm/gems/ruby-1.9.3-p392/gems/puffing-billy-0.2.1/lib/billy/proxy_connection.rb:83:inhandle_request'
/Users/someuser/.rvm/gems/ruby-1.9.3-p392/gems/puffing-billy-0.2.1/lib/billy/proxy_connection.rb:55:in on_message_complete' /Users/someuser/.rvm/gems/ruby-1.9.3-p392/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:inrun_machine'
/Users/someuser/.rvm/gems/ruby-1.9.3-p392/gems/puffing-billy-0.2.1/lib/billy/proxy.rb:68:in `main_loop'

Ignoring google maps and cache.

Hi,

I have a question. I'm trying to use the cache but I'm having some issues with google maps. There are some files that always get regenerated and I'm not exactly sure how to ignore them. I tried to add https://maps.google.ca/maps to the ignore list. But it didn't work.

Have you tried to ignore google maps parameters in the past? Do you have some insight here?

Thanks in advanced for your help.

When deleting the cache, all tests hang in Cucumber

Hey guys, whenever I delete my cache and run all my tests, all of my @puffing-billy Cucumber tests hang up for a very long time and fail. I'm using the cache with VCR setup.

Here's my config:

Billy.configure do |c|
  c.cache = true
  c.cache_request_headers = false
  c.ignore_params = ["http://www.google-analytics.com/__utm.gif",
                     "https://r.twimg.com/jot",
                     "http://p.twitter.com/t.gif",
                     "http://p.twitter.com/f.gif",
                     "http://www.facebook.com/plugins/like.php",
                     "https://www.facebook.com/dialog/oauth",
                     "http://cdn.api.twitter.com/1/urls/count.json",
                     "http://get_maps.gstatic.com",
                     "http://get_fonts/googleapis.com",
                     "http://get_maps.googleapis.com"]
  c.path_blacklist = []
  c.persist_cache = true
  c.ignore_cache_port = true # defaults to true
  c.non_successful_cache_disabled = false
  c.non_successful_error_level = :warn
  c.non_whitelisted_requests_disabled = false
  c.cache_path = 'features/support/request_cache/'
end

I added this line before the config to try and get Billy to reset itself: Billy.proxy.reset_cache

I get this error:

Timed out waiting for response to {"name":"visit","args":["http://127.0.0.1:61200/employment_emails/new"]}. It's possible that this happened because something took a very long time (for example a page load was slow). If so, setting the Poltergeist :timeout option to a higher value will help (see the docs for details). If increasing the timeout does not help, this is probably a bug in Poltergeist - please report it to the issue tracker. (Capybara::Poltergeist::TimeoutError)

The only way to get it to run properly is to uninstall and reinstall puffiing-billy,vcr, and webmock. Any thoughts?

Caching stubbed responses

We are running a separate api server(via nginx) from our app and making requests with a /api/v1/endpoint/query url. When running Capybara, it won't recognize that endpoint because it is running on a different server. We found PuffingBilly to stub out all of those queries... great so far.

The caching of external api requests seems like a great option, but since our api endpoint is on the same host, capybara can't find it and we can't cache the real data. We found a sneaky way of getting data by using the proxy.stub(http://127.0.0.1:9887/api/v1/analytics/query').and_return(Proc.new { |params, headers, body| to proxy any api requests and then making a Net::HTTP request to a live server and returning the actual json data.

The problem with this is that PuffingBilly will not cache the response. Any thoughts on handling this or forcing PB to cache the stubbed response? Or maybe just using the proxy without the stub?

Cache persistance not working

We're having some issues when trying to integrate puffing-billy with a rails 3.2 app and we haven't been able to figure out if it's because I'm using the lib wrong or if it's an actual bug.

We're using puffing-billy on a scenario that visits a page with a form that uses Google Maps JS API's autocomplete component, fills the fields and selects one of the options from the autocomplete, and submits the form. Our goal would be to be able to:

  • Retrieve the JS library from Google, to be able to detect any breaking changes on the library on the integration tests.
  • Persist the cache of a real call, since stubbing Google's responses would be too complex.

We're using Capybara+RSpec+Webkit (headless)+WebMock for our integration tests, and our current configuration looks like this:

RSpec.configure do |config|
   ...
   # Switch the drivers when a feature or scenario is marked with 'mock_js'
   config.around :each do |example|
    if example.metadata[:mock_js]
      Capybara.javascript_driver = :webkit_billy
      Capybara.current_driver = :webkit_billy

      example.run

      Capybara.current_driver = :webkit
      Capybara.javascript_driver = :webkit
    else
      example.run
    end
  end
end

Billy.configure do |c|
  c.cache = true
  c.cache_request_headers = false
  c.path_blacklist = []
  c.persist_cache = true
  c.ignore_cache_port = true # defaults to true
  c.non_successful_cache_disabled = false
  c.non_successful_error_level = :warn
  c.non_whitelisted_requests_disabled = false
  c.cache_path = 'scenarios/puffing-billy/req_cache/'
end

Capybara.javascript_driver = :webkit_ignore_ssl

When running the scenario, the cache for the Google Maps API call is not persisted.
If we don't stub the call to Google Maps API endpoint, it fails to retrieve some assets (though strangely it works for others). Below is a snippet from the test.log file

puffing-billy: CACHE KEY for 'http://127.0.0.1:64858/assets/events.css' is 'get_127.0.0.1_2eb19e129e8b4d150b9934508d1a60792d9be7cc'
puffing-billy: PROXY GET for 'http://127.0.0.1:64858/assets/events.css'
puffing-billy: Request failed: http://127.0.0.1:64858/assets/events.css

If we include the stub for Google Maps API JS call, all assets are retrieved correctly from localhost. I guess we could manually stub the responses from Google, but we wouldn't be able to accomplish the goals listed above... The log for the run with the stubbed Google call regarding the failing assets from the previous example is:

puffing-billy: CACHE KEY for 'http://127.0.0.1:65337/assets/events.css' is 'get_127.0.0.1_2eb19e129e8b4d150b9934508d1a60792d9be7cc'
puffing-billy: PROXY GET for 'http://127.0.0.1:65337/assets/events.css

In neither of the examples the cache is persisted on the defined directory (though I think it's expected in the latter case given that we're stubbing all calls to our only external dependency).

URI::InvalidURIError bad URI(is not URI?)

I am receiving the following error when running tests with puffing-billy:

URI::InvalidURIError bad URI(is not URI?): https://fonts.googleapis.com:443/css?family=Cabin+Sketch:400,700|Love+Ya+Like+A+Sister

This error is occurring in /lib/billy.cache.rb line 90 format_url method. The line url = URI(url) raises the exception because of the '|' [pipe] character in the URL. This is valid, but causes the exception.

By changing the line to url = URI.parse(URI.encode(url.strip)) a valid URI object is returned.

In the meantime, I have encoded the '|' within my html file as: <link href='https://fonts.googleapis.com/css?family=Cabin+Sketch:400,700%7CLove+Ya+Like+A+Sister' rel='stylesheet' type='text/css'>

Modifying cached responses

Hello,
I have been working on creating specs for a couple different scenarios my scraper may come upon. To emulate these situations I had planned to change the content of a page cached with puffing-billy. However it seems like I receive an error from my scraper whenever I change the content. How can I change the content of the cache without failure?

javascript driver segfault

Hi guys, just started using puffing-billy, and really liked it when I had it working, but at some point it just stopped working and I got a segfault:

/.rvm/gems/ruby-1.9.3-p545/gems/puffing-billy-0.2.3/lib/billy/proxy_connection.rb:20: [BUG] Segmentation fault
ruby 1.9.3p545 (2014-02-24 revision 45159) [x86_64-darwin13.1.0]

I'm trying to just use the selenium_billy driver, and am using mostly the default configs for caching. (A couple custom path_blacklists.) Any thoughts on why this is happening and how I can fix? I'm happy to provide whatever else information might be useful. I'm pretty new to javascript testing so any advice/guidance would be very much appreciated.

Thanks so much!

WebMock and puffing-billy

I'm trying to set up puffing-billy on my local repo, and I'm running into a few issues. I am basically registering the webkit driver like this:

Capybara.register_driver :webkit do |app|
Capybara::Webkit::Driver.new(app)
end

And then I am adding the @billy tag to my scenarios, and I've got a Before hook just like the one you guys have in the readme, which sets the current_driver and javascript_drivers to be :webkit_billy

Now, my issue is that if I visit a page when the driver is just :webkit
I take a screenshot, and everything looks proper, all the assets are there, it looks good.
However; When I run it tagged with @billy, or if I set the current_driver and javascript_driver's to be :webkit_billy, all of the assets are not rendered...

I suspect it might be related to nginx, since the asset host is http://localhost:8089

I added this into a billy.rb file which is in features/support/

server = Capybara.current_session.server
Billy.config.whitelist = ["#{server.host}:#{server.port}/", 'http://localhost:8089']

Doesn't seem to help with my issue though. Should I add anything else beyond that? Am I missing some config settings to get this running properly with capybara-webkit?

just noticed @oesmith was active 20 min ago so I'm hoping to get someone's attention!

dynamic_jsonp regex is brittle

I'm using the Balanced Payments API, whose response looks like this:

  /**/ balanced_jsonp_6773175620473921(
  {"status":201,"header":{"X-Balanced-Guru":"OHMd564adc2855811e4a13102a1fe52a36c","Content-Type":"application\/json","x-balanced-host":"bapi-live-prod-7nf6hx-10-3-5-34","x-balanced-software-build":"1.12.16","Content-Length":170,"access-control-allow-origin":"*","access-control-allow-headers":"Content-Type","x-balanced-revision":"1.1","access-control-allow-methods":"POST, OPTIONS","X-Midlr-Version":"2","x-newrelic-app-data":"PxQFWFNXCQYTVVhWAwQDVUYdFhE1AwE2QgNWEVlbQFtcCxYxSBVbDQoZVA4IF0pcXAgEEBhTVggPbldQAQkWDEQRFgFKXVVGVkcVQQFNE1JKBgJWVlQAAgJXUVsFBwJXVwYaE1pSWk4QQF0ECg0ABlAEWFYEWABUU1cVTUYFWV9DATw="},"body":"{\n  \"bank_accounts\": [\n    {\n      \"href\": \"\/bank_accounts\/BA6uHpU4zgaOP4afGaAoalqq\",\n      \"id\": \"BA6uHpU4zgaOP4afGaAoalqq\",\n      \"links\": {}\n    }\n  ],\n  \"links\": {}\n}"}
  );

The regex that Puffing Billy uses to replace the dynamic component doesn't work here, because it looks only at the first word, and looks for ({ without a newline between the characters.

In my project, I've monkey-patched the method to change the regex to /\w+\(/. Do you see any problem with this approach? If not, I'd be happy to open a pull request (or feel free to just change it if that's easier).

The relevant code is at https://github.com/oesmith/puffing-billy/blob/master/lib/billy/handlers/cache_handler.rb#L41.

Thanks!

can't load stripe.js when running tests with capybara-webkit on linux

Hey everyone,

I love puffing-billy! But I'm having a pretty critical problem. I am doing an end to end test and on the checkout page, we load stripe from js.stripe.com like so:

javascript_include_tag 'https://js.stripe.com/v1/'

We then of course use that library to get a payment token.

When I run the test on my mac, the payment succeeds and the test passes. on ubuntu (and circle CI) however, it does not. The test fails and stripe (ruby) renders:

Empty string given for card. You should supply either a token or full card details. To learn more about collecting payment information as a token, see: https://stripe.com/docs/tutorials/cards

Turns out that is because the Stripe (js) library isn't loaded and therefore my server is receiving an empty stripe_card_token.
To confirm: If I interrupt the test and I run page.driver.console_messages I get:

ReferenceError: Can't find variable: Stripe

But this only happens on linux! Not sure what's up. Happy to post whatever info you need. I am running ubuntu 12.04, I am not using the caching feature of puffing-billy, and I have WebMock installed but it is set to allow_net_connect!

Let me know if you have any ideas or debugging steps.

Missing slash causes visit to skip proxy

The following skips the BIlly proxy on visit:

proxy.stub('http://example.com').and_return(:text => 'Foobar')`
visit 'http://example.com'

But if I use "http://example.com/" it will return 'Foobar'. Perhaps this an issue in phantomjs. It seems like either form should work the same.

Caching not working

I'm not sure if this is a bug or me implementing it wrong but my HTTP calls aren't being cached by Bully.

I'm using Ruby 2.2, Rails 4.2.0.
I have model spec which is testing a method that loads a website and parses some data from it using Phantom JS + Nokogiri. I want to use Bully to cache this interaction.

At the moment the spec passes fine, but Bully isn't caching. In fact I'm not sure if Bully is being used at all, because when I set non_whitelisted_requests_disabled = true the spec still passes.

In rails_helper.rb I have the following:

require 'billy/rspec'

Capybara.javascript_driver = :poltergeist_billy

Billy.configure do |c|
  c.cache = true
  c.cache_request_headers = false
  c.ignore_params = ["http://www.google-analytics.com/__utm.gif",
                     "https://r.twimg.com/jot",
                     "http://p.twitter.com/t.gif",
                     "http://p.twitter.com/f.gif",
                     "http://www.facebook.com/plugins/like.php",
                     "https://www.facebook.com/dialog/oauth",
                     "http://cdn.api.twitter.com/1/urls/count.json"]
  c.path_blacklist = []
  c.persist_cache = true
  c.ignore_cache_port = true # defaults to true
  c.non_successful_cache_disabled = false
  c.non_successful_error_level = :warn
  c.non_whitelisted_requests_disabled = false
  c.cache_path = 'spec/req_cache/'
end

My spec is simple, as I'm trying to get caching working with the minimum amount of code to start:

it 'should attempt to parse the data', :js => true  do
    subject
end

What am I doing wrong here?

`c.ignore_urls` is not working

It cannot ignore a url with a dynamic parameter, which is often the case with trackers. For example:

https://api.mixpanel.com/track/?ip=1&img=1&data=eyJldmVudCI6ImNoZWNrb3V0Lm9wZW4iLCJwcm9wZXJ0aWVzIjp7InRva2VuIjoiZDc1MTNiMTZkNTY0OTE3MWYwNDVlY2Q5YTBkZTVjMWIiLCJ1c2VyQWdlbnQiOiJNb3ppbGxhLzUuMCAoTWFjaW50b3NoOyBJbnRlbCBNYWMgT1MgWCAxMC45OyBydjozMS4wKSBHZWNrby8yMDEwMDEwMSBGaXJlZm94LzMxLjAiLCJvcHRpb24tcmVmZXJyZXIiOiJodHRwczovLzEyNy4wLjAuMTozMDAxLyIsIm9wdGlvbi11cmwiOiJodHRwczovLzEyNy4wLjAuMTozMDAxLz9wbGFuPWdvbGQiLCJvcHRpb24tdGltZUxvYWRlZCI6MTQwOTY5Mzg5NCwib3B0aW9uLWtleSI6InBrX0tUeXM1YjdvMkVvTFlaNkJrUnhVRDFRZXFoRXZmIiwib3B0aW9uLW5hbWUiOiJSb2JvU3dlZXBlciIsIm9wdGlvbi1kZXNjcmlwdGlvbiI6IiQ0Ljk5L21vbnRoLiBGcmVlIGZvciBmaXJzdCAzMCBkYXlzLiIsIm9wdGlvbi1wYW5lbExhYmVsIjoiU3Vic2NyaWJlIGZvciIsIm9wdGlvbi1lbWFpbCI6InNwb2Nrc3BsYW5ldEBnbWFpbC5jb20iLCJvcHRpb24tYW1vdW50Ijo0OTksImRpc3RpbmN0X2lkIjoiNGQ5NWQzMDMtNzFkOS05NDYwLWE5NTItN2QwZDY2ODk2MzNmIiwic3QtdWktdHlwZSI6ImRlc2t0b3AiLCJzdC12YXJpYW50LXRyYWNpbmciOiJleGNsdWRlZCIsInN0LXZhcmlhbnQtcmVtZW1iZXJNZSI6ImV4Y2x1ZGVkIiwic3QtdmFyaWFudC1hdXRvY29tcGxldGUiOiJleGNsdWRlZCIsInN0LXZhcmlhbnQtcGhvbmVWZXJpZmljYXRpb24iOiJleGNsdWRlZCIsInN0LXZhcmlhbnQtcGhvbmVWZXJpZmljYXRpb25Db3B5IjoiZXhjbHVkZWQiLCJzdC12YXJpYW50LXBob25lVmVyaWZpY2F0aW9uUGF5IjoiZXhjbHVkZWQiLCJzdC12YXJpYW50LWFkZHJlc3MiOiJleGNsdWRlZCIsInN0LXZhcmlhbnQtanN1bml0IjoiZXhjbHVkZWQiLCJzdC1hY2NvdW50cy1lbmFibGVkIjp0cnVlLCJzdC1yZW1lbWJlci1tZS1lbmFibGVkIjp0cnVlLCJzdC1yZW1lbWJlci1tZS1jaGVja2VkIjpmYWxzZSwic3QtYWNjb3VudC1jcmVhdGVkIjpmYWxzZSwic3QtbG9nZ2VkLWluIjpmYWxzZSwic3QtdWktaW50ZWdyYXRpb24iOiJpZnJhbWUiLCJoIjo5MDAsInciOjE0NDB9fQ==

undefined method `dynamic_jsonp='

For some reason I cannot fathom, I have been getting an error that suggests that dynamic_jsonp and dynamic_jsonp_key methods are not defined in the configuration class. I confirmed that they aren't defined by prying into the code during execution, but the gem source clearly shows they are being made available by attr_accessor method. The specs of the gem are passing for that particular call, and this is only happening when the configuration is being set up. Does anyone have any ideas?

Inconsistency between :webkit and :webkit_billy drivers (failing scenario)

I am using latest capybara-webkit, cucumber and puffing-billy.

The following is green with :webkit:

I am using non-declarative steps for demo purposes.

@javascript
Feature: Test

  Scenario:
    When I go to the "https://www.tumblr.com/login"
    And I fill in "signup_email" with "[email protected]"
    And I fill in "signup_password" with "mypassword"
    And I click "Log in"
    Then I should see 'Log out'

But when I switch to :webkit_billy by adding @billy tag to this scenario, it gets red:

  Scenario:                                                                 # features/test.feature:4
    When I go to the "https://www.tumblr.com/login"                         # features/step_definitions/test_steps.rb:1
    And I fill in "signup_email" with "[email protected]" # features/step_definitions/test_steps.rb:5
    And I fill in "signup_password" with "mypassword"            # features/step_definitions/test_steps.rb:5
    And I click "Log in"                                                    # features/step_definitions/test_steps.rb:9
    Then I should see 'Log out'                                             # features/step_definitions/test_steps.rb:13
      expected to find text "Log out" in "Tumblr Follow the world’s creators.Cancel Sign up How old are you? I have read, understand, and agree to the Tumblr Terms of Service. RefreshAudioSubmit Log in Password help?" (RSpec::Expectations::ExpectationNotMetError)
      ./features/step_definitions/test_steps.rb:15:in `/^I should see 'Log out'$/'
      features/test.feature:9:in `Then I should see 'Log out''

When I am doing page.save_and_open_page at the end of this scenario, I see the same login form without errors. It looks like 'Login' button press is simply skipped for some reason.

I also can't see the CACHE POST log message. Log output is here:

Proxy listening on http://localhost:55770
PROXY GET https://www.tumblr.com:443/login
CACHE GET https://secure.assets.tumblr.com:443/assets/styles/global.css?1a5d529729ac8e35d70ecc5a05d13158
CACHE GET https://secure.assets.tumblr.com:443/languages/strings/en_US.js?1092
CACHE GET https://secure.assets.tumblr.com:443/assets/styles/onboarding.css?8910a8d326a35357ba2bf8df040513d4
CACHE GET https://secure.assets.tumblr.com:443/assets/styles/dashboard/posts.css?1bc8595e6c032fcc9722a3c34b75a3c4
CACHE GET https://secure.assets.tumblr.com:443/assets/styles/dashboard.css?e7c5c3562fe6219b509387b6f4ee7ed9
CACHE GET https://secure.assets.tumblr.com:443/assets/scripts/jquery_with_plugins.js?63b5e280f51e0b2a9ef0615b5fa0b120
CACHE GET https://www.google.com:443/recaptcha/api/js/recaptcha_ajax.js
CACHE GET https://secure.assets.tumblr.com:443/javascript/recaptcha.js?20eb36846dbb6511fe475409cdd3398e
CACHE GET https://secure.assets.tumblr.com:443/javascript/prototype_and_effects.js?1a5f6626c3500e52e1c72620862d34ac
CACHE GET https://secure.assets.tumblr.com:443/assets/scripts/dashboard.js?a3b1d30024d2ea02b438d662d103755d
CACHE GET https://secure.assets.tumblr.com:443/assets/scripts/audio_player.js?5b1e8e3698f3de66bafa5f966b781d5f
CACHE GET https://secure.assets.tumblr.com:443/assets/scripts/onboarding.js?4d6aa4fdcb311a392a16610c0789bda8
CACHE GET https://secure.assets.tumblr.com:443/images/background.png?5
CACHE GET https://secure.assets.tumblr.com:443/images/logo.png?5
CACHE GET https://secure.assets.tumblr.com:443/images/captcha_sprite.png?2
CACHE GET https://ssl.google-analytics.com:443/ga.js
CACHE GET https://secure.quantserve.com:443/quant.js
CACHE GET https://sb.scorecardresearch.com:443/beacon.js
PROXY GET https://www.tumblr.com:443/impixu?T=1367837819&J=eyJ0eXBlIjoidXJsIiwidXJsIjoiaHR0cDpcL1wvd3d3LnR1bWJsci5jb21cL2xvZ2luIiwicmVxdHlwZSI6MH0=&U=JDMOAFFAIC&K=733b9db932df710ed6fe20792133d95b0064b76dad05600cbbe41380a82a8212&R=
CACHE GET https://secure.assets.tumblr.com:443/images/white_80.png
CACHE GET https://secure.assets.tumblr.com:443/images/loading_reblog.gif
CACHE GET https://secure.assets.tumblr.com:443/images/black_75.png
CACHE GET https://secure.assets.tumblr.com:443/images/inline_photo_loading.gif
PROXY GET https://pixel.quantserve.com:443/pixel;r=1736364804;a=p-19UtqE8ngoZbM;fpan=1;fpa=P0-509773980-1367837820862;ns=0;ce=1;cm=;je=0;sr=1280x800x32;enc=n;dst=0;et=1367837820862;tzo=-240;ref=;url=https%3A%2F%2Fwww.tumblr.com%2Flogin;ogl=
PROXY GET https://ssl.google-analytics.com:443/__utm.gif?utmwv=5.4.1&utms=1&utmn=1686698563&utmhn=www.tumblr.com&utmcs=UTF-8&utmsr=1280x800&utmvp=1680x1050&utmsc=32-bit&utmul=ru-ru&utmje=0&utmfl=-&utmdt=Sign%20up%20%7C%20Tumblr&utmhid=607740979&utmr=-&utmp=%2Flogin&utmht=1367837820886&utmac=UA-97144-8&utmcc=__utma%3D189990958.2090029988.1367837821.1367837821.1367837821.1%3B%2B__utmz%3D189990958.1367837821.1.1.utmcsr%3D(direct)%7Cutmccn%3D(direct)%7Cutmcmd%3D(none)%3B&utmu=qhC~

I've also attached screenshot after clicking 'Login' button and sleeping for 2 seconds. It is taken with page.driver.browser.render

screenshot

Cache doesn't work on CircleCI

I'm using puffing-billy for stubbing a form which directly uploads a file to Amazon S3 . This works perfectly fine on my localhost , 2 .yml files are generated ( 1 for preflight Options method & 1 for Post method ).

But when I checked-in & pushed those 2 .yml files to CircleCi and run the specs, somehow it doesn't work . Do I miss any configuration to make the Cache work on CircleCI ?

Any help or suggestion are appreciated.

Best regards.

Cut a release

Just saw the updates for dynamic_jsonp - this is great! Can you cut a release so that we can get this from RubyGems? Thanks a bunch.

VCR config.ignore_localhost = true

I've been using puffing-billy in a project that also uses VCR+WebMock. The following config avoided any problems:

VCR.configure do |config|
  config.ignore_localhost = true
end

This is different to the config suggested in the README. Would a pull-request with this be welcome? If yes should this config be in addition or instead of the existing VCR+WebMock config given in the README?

Compatibility with VCR

I have just tried to install Puffing Billy, but VCR is intercepting the calls that the proxy is making when I use it for a Cucumber scenario. Do VCR/Webmock need to be disabled?

Stub matches fail if stub created with URL params in the supplied URL

If you create a stub using a URL with query parameters, then the stub on the exact same URL/query string fails:

proxy.stub('http://example.test/index?some=params')
proxy.find_stub('GET','http://example.test/index?some=params') #=> returns nil, no stub found

The reason this occurs is because when a stub is created, it stores the full URL passed in to the initializer as @url:

https://github.com/oesmith/puffing-billy/blob/master/lib/billy/proxy_request_stub.rb#L5-L10

    def initialize(url, options = {})
      @options = {:method => :get}.merge(options)
      @method = @options[:method].to_s.upcase
      @url = url
      @response = {code: 204, headers: {}, text: ""}
    end

However, when ProxyRequestStub#matches? substrings the passed in URL to strip off the query parameters and then tries to match the first part of the resulting URL sans parameters with the stub:

https://github.com/oesmith/puffing-billy/blob/master/lib/billy/proxy_request_stub.rb#L55-L63

    def matches?(method, url)
      if method == @method
        if @url.is_a?(Regexp)
          url.match(@url)
        else
          url.split('?')[0] == @url
        end
      end
    end

If the original intent of this is correct, then the stub should probably strip off the query parameters when it stores the URL in the initializer.

Support callback in non-mocked JSONP responses

We are trying to test against Stripe.js using puffing-billy. We want to record real responses and cache them.

The problem is that Stripe.js uses JSONP and the callback param changes each time we run our test.

We'd love to be able to set things up so that when we hit a URL like

http://example.com/jsonp_endpoint?foo=bar&callback=my_callback_12345

and the response looks something like

my_callback_12345({
  message: "success!"
});

then when we run another test later on that hits

http://example.com/jsonp_endpoint?foo=bar&callback=my_callback_98765

it detects this and returns (from the cache):

my_callback_98765({
  message: "success!"
});

Thus, it would actually reach in and rewrite the response.

That way, we can record what the live site returns. Right now it doesn't work because the callback param is set using a browser timestamp, which is different on each run.

Puffing billy proxy not working during sequential rspec examples in webkit

I was trying to use puffing-billy to proxy web requests and noticed that my third party ajax service requests were only being proxied for the first rspec example:

rspec spec/features/examples/tumblr_api_spec.rb -- succeeds
rspec spec/features/examples/facebook_api_spec.rb -- succeeds
rspec spec/features/examples -- fails randomly on either tumblr or facebook (depending on which comes first)

This is when the driver being tested is webkit_billy

Proxy not hit when using capybara test server ?

Hi !

I've got a problem with running capybara/selenium tests with billy. My setup is basically as follows:

Capybara.run_server = true
Capybara.default_selector = :css
Capybara.register_driver :selenium_billy do |app|
Capybara::Selenium::Driver.new(app, :browser => :firefox)
end

Capybara.default_driver = :selenium_billy

Set up like above, then any proxies I set up will be ignored :(

The same code, only with the test server switched off and using a development instance running, works fine. (I.e. I start rails s in a different console and point the tests at localhost:3000 instead of at "/").

Anyone have any ideas why the cache is not hit ?

page.driver.debug not working

I'm using this with Poltergeist and Capybara, and the proxy seems to be working. However, first, after I make a request, I cannot do Billy.proxy.port anymore, I get:

RuntimeError: eventmachine not initialized: evma_get_sockname from /x/gems/puffing-billy-0.2.3/lib/billy/proxy.rb:32:in get_sockname

Then, when I use page.driver.debug, it will open my browser as normal but it is then not able to connect to the server and immediately shows the could not connect page.

It works fine if I don't set the proxy for Poltergeist.

Option not to save headers, method, content as binary?

I noticed everything is being saved as binary. This is really hard to edit. Sometimes you make requests on an API and want to control the response for multiple test cases.

I think I might understand the value in using binary for the content, but the url, headers and method should be plain text. This is just obscuring very useful data for debugging.

If content wasn't force encoded to binary, JSON responses could be modified ever so slightly to test edge cases more difficult to get through a real request.

Basically, I'd like better control over the cached requests / responses.

Push new gem to rubygems

There have been a number of changes since the gem was last pushed in February. Could we get a new version up on RubyGems?

Thanks!
Tony

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.