amberishsingh / abot Goto Github PK

Spread the word

Merge branch 1.0 into trunk, create bin zip for downloads section

Verify can run on .net 3.5, 4.0, 4.5

Add fatal crawl errors to CrawlResult

Change Abot.Console to Abot.Demo

setup google groups and add link to homepage.

Contact graphic designer to have logo created

Use ILMerge to create a single Abot.dll with all dependent dlls

Consider using AutoResetEven instead of busy wait

Make Abot check its version an if less than the latest "featured" version log a 
message suggesting an update

Handle relative link parsing when html Base tag present

Add PageCrawlDisallowed & PageLinksCrawlDisallowed events so the user can 
detect when a page was either not crawled due to a crawl decision returning 
false.

Add maxpagestocrawl check in crawldecisionmaker

Update all assemblies to 4.5

Console fails when crawling site wvtesting2.com

Add constructor to webcrawler that takes only ICrawlDecisionMaker and both 
ICrawlDecisionMaker and CrawlConfiguration

Add abot version dynamically to user agent string

Create crawl configuration object and a generic provider that can be overridden.

Implement crawl depth

Add crawl recovery that reloads pages that were crawled, pages to crawl and 
other context. This allows the crawl to pick up where it left off. May also 
need to add a stop for this work properly

Implement use of downloadableContentTypes config value

Consider using CsQuery as the parser. It boasts speeds of x times faster than 
hap.

https://github.com/jamietre/CsQuery

Create Nuget installer

Consider accepting lambda expressions for crawl decisions.

Pros: 
-Allows users to determine crawl behavior on the fly
-No classes or interfaces to implement and plugin

Cons:
-Not easy to test compound crawl behaviors
-Users must set these values on every instance (lots of copy and paste)
-Hard to group related or behaviors that are grouped together

-Think about moving unique uri crawling check/logic to IScheduler so a 
DistributedScheduler would be the only change to give distributed scheduling.

Hook up google analytics

Currently ignoring test...

Crawl_PageLinksCrawlDisallowedSubscriberThrowsExceptions_DoesNotCrash

Search for "//TODO This test only fails when run under NCOVER"

Add integration tests for at least the following sites...

sitesimulator
wvtesting.com
sethgodin.com

Add MaxTimeToCrawl configuration. Crawl times out after this limit is reached.

Verify site crawls are returning expected number of pages...

1: sitesimulator
2: wvtesting2.com
3: sethgodin.com

CrawlContext should be reset on every crawl() call

Use Vs fakes to raise code coverage on untestable code

Use concurrent collections for Scheduler and CrawlContext.CrawledUris now that 
abot is targeting 4.5.

Create a PoliteWebCrawler.

-Add throttling
-Add manual crawl delay
-Add respect robots crawl delay
-Add respect robots disallow directive
-Add respect meta robots no index no follow

Create monitoring plugin that leverages abot

Implement manual crawl delay

Implement crawl timeout where the crawler will stop after x seconds

-Add Func<PageToCrawl, CrawlDecision> for ShouldCrawlPage
-Add Func<CrawledPage, CrawlDecision> for ShouldCrawlPageLinks
-Add Func<CrawledPage, CrawlDecision> for ShouldDownloadPageContent

Add crawltimeout where crawl ends if the timeout time has elapsed.

Add disallow reason when firing PageCrawlDisallowed and 
PageLinksCrawlDisallowed events

Create WebCrawler that uses a list of rules for its crawl descisions

Add page for custom crawler work by hour

Make configuration like MaxThreads and UserAgentString easy from crawler 
object. Maybe make a BasicWebCrawler : WebCrawler that gives this functionality.

Implement use of isUriRecrawlingEnabled config value.

Verify is able to run on mono by using MOMA

-Link to the latest stable instead of making them go to the downloads tab
-Add fiddler .saz file to replay
-Add Abot vs Arachnode vs NCrawler section
-Add faqs page
-Split up quickstart onto its own page
-Add more detail to running the tests w/fiddler etc..

Add lic text from http://www.apache.org/licenses/LICENSE-2.0

Setup Paypal donate button, and possibly a consulting block purchase page.

[deleted issue]

Verify nuget solution restore