Giter VIP home page Giter VIP logo

searchengine's Introduction

INFO 344 PA 2

A search engine built off of crawling CNN.com and BleacherReport.com. A special thank you to CK and Chris for all of the help.

Website Link

Live Search Website feel free to visit ./dashboard to view up to date stats about crawling and errors.

Screenshots

Windows Azure Dashboard

Windows Azure Dashboard

Amazon AWS Dashboard

Amazon AWS Dashboard

Search App index.html

Search Index

Write up

End-to-End experience.

  • PA1: I had to edit my index.php file so that it only returned results when I had an exact match. I also had to add in special javascript in order to handle the JSONP received from the AWS instance.
  • PA2: I actually changed my PA2 the most. I implemented a hybrid list trie that was able to fit the entire wikipedia title dataset in about 400mb. This was fascinating how it effected the speed SO much. Adding the results to the search page just required me using a datalist.
  • PA3: I changed the partition key to be every word in the CNN or BleacherReport title. This essentially made my Azure Table an inverted index. When a user searches, each word in the search gets queried against the Azure table. After this is complete I use a linq statement to order by most common link and then by date. This allows for nice relevant search results at incredible speeds.
  • PA4: I was able to implement a cache for 100 of the most recent results, and utilized Chris Oh's adsense script to add monetization to my search engine.

Extra credit write up

  1. I was able to complete the Hybrid List Trie after many hours and attempts.
  2. I also made my page somewhat pretty with a custom logo, images for the NBA players, and a clean UI for search.

searchengine's People

Contributors

campbellsean avatar

Stargazers

 avatar

Watchers

James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.