lorien / awesome-web-scraping Goto Github PK

In Java,

There is:

• Heritrix
  http://www.crawler.archive.org/index.html
  which is hard to understand and seldomly supported.
• Norconex Collector
  https://www.norconex.com/enterprise-search-software/open-source-crawlers/
  which is actively developped and the main dev is very supportive.

Hey, guys!

What section I need use to put fake-useragent library for python
This lib provides generation user agent strings based on the world browsers statistic.
fake-useragent

I'm not sure, but I've made pull request

If this satisfies to guidelines, please merge that.

Thank's!

Awesome gem to check equivalency between xml documents or nodes. Has built in Rspec matchers so you can write specs like this:

expect(node_1).to be_equivalent_to(node_2)

I made a Diffbot PHP client, it's now the official client for PHP.

I didn't know which category to put it in, so I'll just put it here: https://github.com/Swader/diffbot-php-client/ and here are some tutorials about it: http://www.sitepoint.com/tag/diffbot

Scrapy supports Python3 now.

just a suggestion, having a russian flag at the top of the page might not be the best idea?

i'm using querypath for html/xml reading