IGScraperKit
Create dynamic web scraper in Objective-C or Ruby.
Usage
Create a scraper:
#import "IGScraperKit.h"
IGScraper* scraper = [IGScraper scraperWithBlock:^id(IGXMLNode* node, NSString* url) {
return [[[node queryWithXPath:@"//p"] firstObject] text];
}];Then scrape HTML with scraper:
[scraper scrape:@"<html><p>Hello World</p></html>" URL:nil];
// => @"Hello World"#import "IGScraperKit.h"
IGScraperRecipe* recipe = [[IGScraperRecipe alloc] init];
[recipe addURLPattern:[NSRegularExpression regularExpressionWithPattern:@"https://www\.google\.com/search\?q=.+" options:0 error:nil]
withScraperBlock:^id(IGXMLNode *node, NSString *url) {
// handling for the page ...
return data;
}];
[recipe addURLPattern:[NSRegularExpression regularExpressionWithPattern:@"https://www\.google\.com/" options:0 error:nil]
withScraperBlock:^id(IGXMLNode *node, NSString *url) {
// handling for the page ...
return data;
}];
[recipe scrapeWithHTML:html URL:URL];Write Scraper With Ruby
If you want something more dynamic, you can define a Recipe in Ruby:
# A recipe scrape page based on URL
class GoogleRecipe < ScraperKit::Recipe
title "Google Search"
# define a HTML scraper by `on url`, where url can be a string or a Regexp
on %r{https://www\.google\.com/search\?q=.+} do
# doc is a HTMLDoc object represent the document
doc.xpath('//h3/a').collect {|node| node.text }
end
# if the page you need to parse is not HTML
# use `on_text url`
on_text %r{https://www\.google\.com/search\.json.+} do
# doc is a string of the document
JSON.parse(doc)
end
endThen load the recipe into IGRecipeRegistry and parse the page:
#import "IGScraperKit.h"
// load the recipe
NSString* path = [[NSBundle mainBundle] pathForResource:@"google" ofType:@"rb"];
NSString* recipe = [[NSString alloc] initWithContentsOfFile:path encoding:NSUTF8StringEncoding error:nil];
IGRecipeRegistry* registry = [[IGRecipeRegistry alloc] init];
[registry loadRecipe:Recipe(@"walmart")];
NSArray* result = [registry scrapeWithHTML:html url:@"https://www.google.com/search?q=doughnuts"];
// => <__NSArrayM 0xed85590>(
// Doughnut - Wikipedia, the free encyclopedia,
// Home - Krispy Kreme Doughnuts and Coffee,
// Voodoo Doughnut - The Magic is in the Hole!!!,
//【超人氣甜甜圈】Krispy Kreme Doughnuts 、台北店 ... - Yam天空部落,
// Doughnut Recipes - Allrecipes.com,
// Doughnut Plant,
// Revolution Doughnuts,
// Sidecar Doughnuts & Coffee,
// Top Pot Hand-Forged Doughnuts,
// Lucky's Doughnuts
// )
To use this, you will need to include JavaScriptCore framework (iOS 7, OS X 10.9) and define IGSCRAPERKIT_ENABLE_SCRIPTING before import IGScraperKit.h.
Installation
To install IGScraperKit throught CocoaPods, add following lines to your Podfile:
pod "IGScraperKit", '0.3.1'Or with Ruby supports:
pod "IGScraperKit/Scripting", '0.3.1'Dependencies
- IGScraperKit use IGHTMLQuery for HTML processing. Check the ruby wrappers if you need to use the Ruby interface.
- IGScraperKit use JavaScriptCore in iOS 7 and Opal for JavaScript support.
- Use phantomjs to run Opal tests.
Development
- Install gems: In the project folder, run the command:
bundle install - Install cocoapods: Run the command:
pod install
License
MIT License. Check LICENSE.txt.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent