Lots of Net::HTTP / Net::Protocol errors with Sidekiq (threadsafe issue?) about pusher-http-ruby HOT 12 CLOSED

I wouldn't have thought this is a threading issue - net/http should be fine. Can you maybe post some of the other backtraces? For example DecodeError - I can basically guarantee that Pusher will never return a 200 with invalid json - is there a proxy in the way? I can't see where the readline error could come from - it slightly suggests that your environment may be broken?

from pusher-http-ruby.

Post your code that configures and uses the Pusher API. The gem looks pretty thread-safe.

from pusher-http-ruby.

@mloughran DecodeError issue:

vendor/bundle/ruby/1.9.1/gems/multi_json-1.5.0/lib/multi_json/adapters/oj.rb:17:in `load'
vendor/bundle/ruby/1.9.1/gems/multi_json-1.5.0/lib/multi_json/adapters/oj.rb:17:in `load'
vendor/bundle/ruby/1.9.1/gems/multi_json-1.5.0/lib/multi_json.rb:96:in `load'
vendor/bundle/ruby/1.9.1/gems/pusher-0.11.1/lib/pusher/request.rb:93:in `handle_response'
vendor/bundle/ruby/1.9.1/gems/pusher-0.11.1/lib/pusher/request.rb:52:in `send_sync'
vendor/bundle/ruby/1.9.1/gems/pusher-0.11.1/lib/pusher/resource.rb:18:in `post'
vendor/bundle/ruby/1.9.1/gems/pusher-0.11.1/lib/pusher/client.rb:134:in `post'
vendor/bundle/ruby/1.9.1/gems/pusher-0.11.1/lib/pusher/client.rb:216:in `trigger'
lib/pusher_wrapper.rb:30:in `trigger'

@mperham our code:

PusherWrapper code:

require_dependency 'configurator'

class PusherWrapper
  include Configurator

  config_file     'pusher.yml'
  config_accessor :url

  def self.key
    url.match(/^http:\/\/(\w*):.*$/)[1]
  end

  def self.authenticated_response(channel_name, custom_data)
    Pusher[channel_name].authenticate(custom_data)
  end

  def self.handle_webhook(webhook)
    webhook.events.each do |event|
      case event["name"]
      when 'channel_occupied'
        Sidekiq.redis { |con| con.sadd("pusher:channels", event["channel"]) }
      when 'channel_vacated'
        Sidekiq.redis { |con| con.srem("pusher:channels", event["channel"]) }
      end
    end
  end

  def self.trigger(channel_name, event_name, data)
    if Sidekiq.redis { |con| con.sismember("pusher:channels", channel_name) }
      Pusher.trigger(channel_name, event_name, data)
      true
    else
      false
    end
  rescue Pusher::Error
    false
  end

end

Configurator

require 'active_support/core_ext'

module Configurator
  extend ActiveSupport::Concern

  included do
    class << self
      attr_accessor :config_path, :prefix
    end
  end

  module ClassMethods

    def config_file(filename, options = {})
      @config_path = Rails.root.join('config', filename)
      @config_file_options = { rails_env: true }.merge(options)
      @prefix = File.basename(filename, '.yml').upcase
    end

    def config_accessor(*attributes)
      @config_attributes = attributes
    end

    def method_missing(*args)
      method_name = args.shift.to_sym

      if @config_attributes && @config_attributes.include?(method_name)
        yml_options[method_name] == 'env_var' ? ENV["#{@prefix}_#{method_name.to_s.upcase}"] : yml_options[method_name]
      else
        yml_options[method_name].nil? ? super(method_name, *args) : yml_options[method_name]
      end
    end

    def respond_to?(*args)
      method_name = args.shift.to_sym

      (@config_attributes || []).include?(method_name) || yml_options[method_name] || super(method_name, args)
    end

    def yml_options
      if Rails.env == 'test'
        calculate_yml_options
      else
        @yml_options ||= calculate_yml_options
      end
    end

    def calculate_yml_options
      yml_hash = if @config_file_options[:rails_env]
        YAML.load_file(@config_path)[Rails.env.to_s]
      else
        YAML.load_file(@config_path)
      end
      HashWithIndifferentAccess.new(yml_hash)
    end

  end
end

Pusher initializer

require_dependency 'pusher_wrapper'

Pusher.url = PusherWrapper.url

It would be great if you could see anything suspicious in our code! Thanks!

from pusher-http-ruby.

That code looks fine.

The errors you are seeing are symptomatic of multiple threads sharing an unprotected socket. net/http should be thread-safe though so I'm unsure of the issue.

from pusher-http-ruby.

You're absolutely right, this is a threading issue. You can replicate with something as simple as

(1..100).map do |i|
  Thread.new do
    puts Pusher['channel'].trigger!('event', {})
  end
end.each { |t| t.join }

I'm slightly amazed that nobody has reported this before; threading must still be quite rare in your average rails app.. You can workaround by removing the memoization in the Pusher::Client#net_http_client method. I'll try to take a look at this properly tomorrow to see if there's a better solution. Thanks guys.

from pusher-http-ruby.

Yeah that's crazy, I was quite sure to find another issues like mine.
I have forked and removed memorization and it seems to going fine now.

Thanks a lot for your support and your gems @mloughran & @mperham!

from pusher-http-ruby.

Same errors and same problems here. It worked using the fork done by @thibaudgg .

from pusher-http-ruby.

I've released a new gem version (0.11.2) which just removes the memoization.

I also got round to integrating net-http-persistent which actually reuses net/http connections (in a threadsafe way) - if you fancy being a gunea pig for this please try the https://github.com/pusher/pusher-gem/tree/persistent_net_http branch. I'm still debating whether this should be the default and whether plain vanilla net/http should be configurable. Thoughts appreciated :)

from pusher-http-ruby.

I just have to say.. i literally started getting this error this morning and found this issue.. I am also using sidekiq and i just upgraded from 0.10.0 to 0.11.1. Could this have been introduced in 0.11.X?

thx!

from pusher-http-ruby.

@kbaum yes you're right, pretty sure this was a regression in 0.11.0 caused by 3425798

from pusher-http-ruby.

Great, I'll give a try to the persistent_net_http branch after my holiday. Thanks!

from pusher-http-ruby.

For those coming here with similar issues, it appears to be fixed for us by v0.12.0

from pusher-http-ruby.