This project is established for UCI CS121 final assignment

src
├── app.py
├── build.py
├── config.ini
├── controller.py
├── model.py
├── packages
│   └── requirements.txt
├── static
│   ├── index.css
│   ├── logo.png
│   └── search.css
├── templates
│   ├── index.html
│   └── search.html
└── util
    ├── indexBuilder.py
    ├── simhashIndex.py
    └── textProcessor.py

• Implement a inverted index
• posting include tf-idf score

• multi query

• token stemming
• weight of words in title,<strong>, and heading
• less than 300ms response
• a complete search engine

• duplicate and near duplicate pages
• page rank
• 2-gram/3-gram indexing
• posting with word position
• anchor words
• Web or GUI interface

Download the processed data files from here, unzip to the same directory where you place this project. The file structure should be look like this:

sparrowlite
├── src
├── DATA <--drop it here
└── README.MD

You need to specify where the source data directory and where to put processed data in src/config.ini

[DATABASE]
WEBSITES_DIR = ../DEV/ # dir where your raw source files locate, if you use built data file I provided, no need to change this one
DATABASE_DIR = ../DATA/    # output dir of your data   
MERGE_CHUNK  = YES         # do not change this line

Install dependencies:

Python 2

pip install -r src/package/requirements.txt

Python 3

pip3 install -r src/package/requirements.txt

To run the application you are required to set your FLASK_APP environment variable.

For Windows:

cd src
set FLASK_APP=main.py

For macos or Unix-like:

cd src
export FLASK_APP=main.py

Start flask application

flask run --host=127.0.0.1 --port=3000

The program will listen and serve the application at localhost on port 3000. Visit http://localhost:3000 in your browser to see the web page.

• need to rewrite the function to merge partial data files using heapq
• ugly retrieval result styling
• stemming
• use config.ini to set crawled website directory and output directory
• implement forword idx of inverted idx for retrieval docuements
• return just url for stage 2, not path
• flx disk seek difference on windows and mac/unix-like
• implement complex tuple intersection algorithm
• slow or false implementation of cosine similarity
• use Simhash to remove near duplicates
• fix "," ValueError when parsing docid csv
• implement inward and outward link graph