This project is established for UCI CS121 final assignment
src
├── app.py
├── build.py
├── config.ini
├── controller.py
├── model.py
├── packages
│ └── requirements.txt
├── static
│ ├── index.css
│ ├── logo.png
│ └── search.css
├── templates
│ ├── index.html
│ └── search.html
└── util
├── indexBuilder.py
├── simhashIndex.py
└── textProcessor.py
- Implement a inverted index
- posting include tf-idf score
- multi query
- token stemming
- weight of words in title,<strong>, and heading
- less than 300ms response
- a complete search engine
- duplicate and near duplicate pages
- page rank
- 2-gram/3-gram indexing
- posting with word position
- anchor words
- Web or GUI interface
Download the processed data files from here, unzip to the same directory where you place this project. The file structure should be look like this:
sparrowlite
├── src
├── DATA <--drop it here
└── README.MD
You need to specify where the source data directory and where to put processed data in src/config.ini
[DATABASE]
WEBSITES_DIR = ../DEV/ # dir where your raw source files locate, if you use built data file I provided, no need to change this one
DATABASE_DIR = ../DATA/ # output dir of your data
MERGE_CHUNK = YES # do not change this line
Install dependencies:
Python 2
pip install -r src/package/requirements.txt
Python 3
pip3 install -r src/package/requirements.txt
To run the application you are required to set your FLASK_APP
environment variable.
For Windows:
cd src
set FLASK_APP=main.py
For macos or Unix-like:
cd src
export FLASK_APP=main.py
Start flask application
flask run --host=127.0.0.1 --port=3000
The program will listen and serve the application at localhost on port 3000. Visit http://localhost:3000 in your browser to see the web page.
- need to rewrite the function to merge partial data files using heapq
- ugly retrieval result styling
- stemming
- use config.ini to set crawled website directory and output directory
- implement forword idx of inverted idx for retrieval docuements
- return just url for stage 2, not path
- flx disk seek difference on windows and mac/unix-like
- implement complex tuple intersection algorithm
- slow or false implementation of cosine similarity
- use Simhash to remove near duplicates
- fix "," ValueError when parsing docid csv
- implement inward and outward link graph