Comments (9)
what I did for now is modify agent.prepare_request to look at the host_headers passed and if its a hash, then use that hash to set the passed header(s), if its not leave the current functionality
unless @host_headers.empty?
@host_headers.each do |name,header|
if host.match(name)
if header.is_a(Hash)
header.each do |header_name, header_value|
headers[header_name] = header_value
end
else
headers['Host'] = header
end
end
end
end
I think the polite thing for me to do is somehow propose this as a submit request? I'm not sure how to do that (took me a few minutes to figure out how to create/test a local gem to make sure this worked) - so let me know if you would like me to figure out how to submit this as a change
from spidr.
Going to need a little more info to isolate the root cause. I'm not sure whether it's the site or Spidr::Agent which is not following HTTP/1.1.
from spidr.
I've run my code on ~85 sites - only had the issue on https://www.logility.com/
Patching Agent.prepare_request with the above code, and then passing in "Accept-Encoding: plain" did work
So my guess is that the server is doing something wrong with its deflate/gzip, so telling it to not encode works. Since I can't change the server, I needed to address on the scanning side.
from spidr.
I tried a number of them and they all triggered it (can't say for sure its
every url, but I didn't come across any that didn't)
On Wed, Nov 18, 2015 at 8:18 PM, Postmodern [email protected]
wrote:
Is it every URL on logility.com or a specific URL that triggers it?
—
Reply to this email directly or view it on GitHub
#43 (comment).
from spidr.
Probably is related to ruby's default headers:
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
User-Agent: Ruby
from spidr.
Right, by default ruby accepts gzip and deflate. Its the decompression
that is failing.
Setting the accepted encoding to only plain, means no compression.
The problem was there is no way in spidr to send the change to the header
(no way to override the ruby default) - the code I shared exposes the
ability to set that request header manually.
On Wed, Nov 18, 2015 at 8:30 PM, Postmodern [email protected]
wrote:
Probably is related to ruby's default headers:
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: /
User-Agent: Ruby—
Reply to this email directly or view it on GitHub
#43 (comment).
from spidr.
I wonder if this is a bug in ruby. I may be open to adding another callback to allow setting custom headers. Although, I don't want to change too much just to workaround a bug that may be in Ruby or www.logility.com.
from spidr.
In searching for that error it has occurred to other libraries as well - so
its sort of a bug in ruby I guess. That said, exposing a request header
makes sense as there could be a lot of reasons you want to force a specific
set.
On Wed, Nov 18, 2015 at 8:39 PM, Postmodern [email protected]
wrote:
I wonder if this is a bug in ruby. I may be open to adding another
callback to allow setting custom headers. Although, I don't want to change
too much just to workaround a bug that may be in Ruby or www.logility.com.—
Reply to this email directly or view it on GitHub
#43 (comment).
from spidr.
Spidr 0.6.0 added Agent#default_headers
which is just a Hash of headers that gets added to every request. Setting agent.default_headers['Accept-Encoding'] = 'plain'
or passing in default_headers: {'Accept-Encoding' => 'plain'}
would fix this.
from spidr.
Related Issues (20)
- Limit crawl to links matching pattern HOT 3
- unable to ignore links HOT 4
- Add ignore_paths and ignore_paths_like
- Add low-level HTTP request methods
- `ignore_links` not working. HOT 4
- path conflicts with opaque (URI::InvalidURIError) HOT 3
- Thank you HOT 1
- Add Logging
- Figure out why specs are failing only on JRuby? HOT 1
- Switch to Ruby 2.x keyword arguments
- Switch to using async-http HOT 1
- Add spec for `Spidr::Agent.start_at`
- Add spec for `Spidr::Agent.site`
- Add spec for `Spidr::Agent.host`
- Write specs for Agent.domain HOT 1
- Support passing a URI as a proxy setting
- How to control the depth of crawling? HOT 1
- Switch to `Addressable::URI` for URI parsing
- Switch to using `require_relative` to improve load-times
- Add `# frozen_string_literal: true` comments to all files
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spidr.