@marg624, I'm having some trouble storing numerical words in the lexicon for some reason. Do you think this might be due to the SHA encoding?

Determine if and which node needs to be taken down and brought up again with a fresh NodeID

Possibly using Java BlockingQueue interface.

We need to create a URL feeder for a newly started crawler.

It either reads from file or loads the status of the crawler from the database.

Will need to setup ring and handle crawler DHT caching.

Messages (and urls) do not implement comparable. As a result, currently we cannot use priority queue as a blocking queue implementation.

• Can this be done via script to ensure the TestSuites are always running all the JUnit TestCases?

Create backup scripts for the database on EC2; since even home is not persistent, other scripts must be also included (that generate keys and push to each devs account).

Create a class for robots.txt handling

Create ant scripts for production environment. These should probably be in a separate build.xml

The project currently won't build without at least method stubs in the Lexicon

Some examples of missing methods from Indexer.java

• lexicon.getLexicon();
• lexicon.addListToLexicon(stemlist);
• Lexicon lexicon = new Lexicon();

212fe86

• This doesn't return the next doc. It returns a cursor.

Contemplate which parts of the indexer need DHT storage. How to implement this? How to share nodes/message nicely with Crawler?

Once FreePastry is up and running, start testing crawler and indexer there!

Options include:

1. Porter Stemmer Algorithm - http://tartarus.org/~martin/PorterStemmer/
2. Stanford NLP Tools (Lemmatizer) - http://nlp.stanford.edu/software/corenlp.shtml

Should we use the lemmatizer library if Andreas allows it? Presumably it will greatly improve query results.

Merge HttpResponse to HttpModule and implement the Http client

Create ant script that runs all Unit tests and writes output (apart from System.out, it could be some form machine-readable form of xml or csv, whatever is easier to interpret)

Add LinkExtracror and Configuration classes

Consider design implications of queries using quotation marks.

Call the appropriate processing methods (i.e. parser, extractor, stemmer, etc.) from the indexer and test output.

We need to add granular per-package logging; plus a global logger that gets a copy of everything (so we can validate relative logical time-stamps).

Currently relying heavily on System.out.println to check for appropriate output. Should convert this to proper JUnit tests.

• Inside catch - use logger.warn("Exception", exceptionObject)

We need a DUE right before Frontier. We need to finalize duplicate elimination strategy (is it going to keep a different version? Update url status as "more important"? Discard? Maybe no duplicate elimination?)

Here's a list of words our search engine should generally ignore:

Stop Words

The exception here is when words are in quotes, in which case we take the query in its entirety.

Call the appropriate processing methods (i.e. parser, extractor, stemmer, etc.) from the indexer and test output.

We need to create a global entry point for the backend, in order to initialize everything correctly. This can't be part of existing modules (such as the crawler or the indexer); it must be either a new package or in util.

Note that this is closely linked with #2

• In a file?
• How many URLs to start? - does this depend on the number of nodes? threads? neither?
• Top Alexa Sites could be used as a source of ideas.
• CNN
• tumblr
• yahoo
• wordpress
• wikipedia?
• reddit (lots of links to various parts of the interne, but also a lot of imgur links)

And why does it have type URL?

I though the RobotsModule was to be a facility for processes to determine if a URL was allowed to be crawled. I didn't think it would have state, besides maybe some LRU caching of recently accessed RobotsTxt objects.

It seems that adding new Hits to the Indexer is a bit slow at the moment. Is there a good way to still maintain synchronicity and batch new word lexicon additions? Maybe this is where the DumpLexicon comes in?

sends some large number of messages and reports on how many each node received.

Create a Configuration Singleton class so we can

• share project configuration properties
• make use of test and production configurations

Words like I'm => I am, I've => I have, etc.

Hey @nvasilakis, this is probably simple, but how do we get rid of these warnings? This one came up when I ran RobotsManagerTest.java

log4j:WARN No appenders could be found for logger (com.yippee.db.crawler.RobotsManager).
log4j:WARN Please initialize the log4j system properly.

Possibly using Java BlockingQueue interface.

• Should deploy a few nodes

Use Stanford NLP Lemmatizer to assist in interpreting queries.

• Managers should not have to call setup and close.

http://wordlist.sourceforge.net/ - wordlist andreas gave
have local log of new words seen, so can add then to lexicon
http://piazza.com/class#spring2012/cis455555/617 - piazza post where he explains it

• Its a bit more complicated than appending the relative URL to the URL of the page