miguelramosfdz / norch Goto Github PK

Table of Contents generated with DocToc

• Installation
• Operation
  • Start your Norch.js server
    • Startup options
• Crawling
  • norch-fetch
  • norch-document-processor
  • norch-indexer
• Indexing API
  • Document Format
  • HTTP Interface
  • Norch-indexer
  • Indexing parameters
    • filterOn
• Replication API
  • Snapshot
  • Empty
  • Replicate
• Search API
  • Get document by ID
  • Search parameters
    • q
    • offset
    • pagesize
    • (Search using the search-index.js API)
• Matcher API
  • Connecting to a matcher
• About Norch
• License

#Installation

Confirm that node.js is set up correctly, then install Norch like so:

$ npm install -g norch

Notes:

1. You need admin priviledges to install globally- on mac and linux use sudo: $ sudo npm install -g norch.
2. Norch can also be installed into the current working directory by dropping the -g flag: $ npm install norch. Norch now exists in the node_modules directory. From the directory you ran the install command, type: $ node_modules/norch/bin/norch to start norch.

If everything went to plan- Norch should now be installed on your machine

#Operation

Note: for the purposes of accessability, this doc assumes that Norch is being installed locally on your own computer (localhost). Once Norch is rolled out on to remote servers, the hostname on all URLs should be updated accordingly. Command line commands are denoted by the prefix $ which should not be typed in

##Start your Norch.js server

Type

$ norch

Hurrah! Norch is now running locally on your machine. Head over to http://localhost:3030/ and marvel. The default port of 3030 can be modified if required.

Startup options

$ norch --help

  Usage: norch [options]

  Options:

      -h, --help         output usage information
      -V, --version      output the version number
      -p, --port <port>  specify the port, defaults to 3030
      -n, --hostname <hostname> specify the hostname, defaults to 0.0.0.0 (IADDR_ANY)

###Using Norch in a node-program

Norch can be instantiated like

var norch = require('norch')(); 

or by giving it an instance of search-index

var si = require('search-index')();
var norch - require('norch')({si: si});

#Crawling Norch has command line tools for spidering, fetching, processing and indexing webpages that can be installed seperately

##norch-fetch Get your webpages with norch-fetch

Install:

$ npm install -g norch-fetch

Help:

$ norch-fetch -h or read the docs

##norch-document-processor Turn your fetched webpages into JSON with norch-document-processor

Install:

$ npm install -g norch-document-processor

Help:

$ norch-document-processor -h or read the docs

##norch-indexer Index your JSONified webpages with norch-indexer

Install:

$ npm install -g norch-indexer

Help:

$ norch-indexer -h or read the docs

#Indexing API

##Document Format

Norch indexes data that is in the format below. Field values can be either strings or simple arrays. Arrays can be used to create filters and facets. An ID can optionally be specified, if no ID is specified, an unique ID will be assigned by Norch

[
  {
    "id":"1",
    "title":"A really interesting document",
    "body":"This is a really interesting document",
    "metadata":["red", "potato"]
  },
  {
    "id":"2",
    "title":"Another interesting document",
    "body":"This is another really interesting document that is a bit different",
    "metadata":["yellow", "potato"]
  }
]

##HTTP Interface

If the above was in a file called data.json, it could be indexed using a command like

curl --form [email protected] http://localhost:3030/indexer --form filterOn=metedata

There is some test data in the test/testdata folder of the norch.js package. It can be indexed like so:

curl --form [email protected] http://localhost:3030/indexer --form filterOn=places,topics,organisations

You can also put the data to be indexed in the URL like this (note that single quotes go on the outside and double on the inside of the JSON object):

curl --form document='[{"title":"A really interesting document","body":"This is a really interesting document"}]' http://localhost:3030/indexer

##Norch-indexer

Norch can optionally be indexed using the norch-indexer node app.

##Indexing parameters

###filterOn

Example

 --form filterOn=places,topics,organisations

filterOn is an array of fields that can be used to filter search results. Each defined field must be an array field in the document. filterOn will not work with string fields.

#Replication API

##Snapshot

Create a snapshot of the search index by doing this:

curl http://localhost:3030/snapshot -o snapshot.gz

##Empty

Empty an index by using the 'empty' endpoint (or alternatively you can just delete the data directory):

curl http://localhost:3030/empty

##Replicate

Replicate into an empty index from a snapshot file by doing this:

curl -X POST http://localhost:3030/replicate --data-binary @snapshot.gz -H "Content-Type: tion/gzip"

#Search API

##Get document by ID

It is possible to get a document and associated index entries by ID by calling http://localhost.com:3030/getDoc?docID= followed by the document's ID

##Search parameters

Search is available on http://localhost.com:3030/search

###q (Required) For "query". The search term. Asterisk (*) returns everything.

Usage:

q=<query term>

http://localhost:3030/search?q=moscow

To search on one or more fields, do something like:

q[title]=reagan&q[body]=intelligence contra

http://localhost:3030/search?q[title]=reagan&q[body]=intelligence%20contra

###offset

(Optional) The index in the resultSet that the server returns. Userful for paging.

Usage:

offset=<start index>

http://localhost:3030/search?q=moscow&facets=topics&filter[topics][]=grain&offset=5

###pagesize

(Optional) defines the size of the resultset (defaults to 20)

Usage:

pagesize=<size of resultset>

http://localhost:3030/search?q=moscow&facets=topics&filter[topics][]=grain&offset=5&pagesize=5

###(Search using the search-index.js API)

For more complex queries, including those involving filters and facets, you can search using the search-index api by sepcifying a JSONq parameter like so:

http://localhost:3030/search?JSONq={"query":{"*":["usa"]}}

#Matcher API

Norch comes with a matcher that can be used to create autosuggest functionality. The matcher is derived from the content of the reverse index. At the moment Norch ships with one matcher, there is a desire to abstract this out into a framework that can accomodate mulitiple pluggable matchers.

##Connecting to a matcher

Using something like Typeahead or JQuery autocomplete the matcher can be called by using this URL:

http://localhost:3030/matcher?beginsWith=<matcher term>

#About Norch

Norch.js is an experimental search engine built with Node.js and search-index featuring

• Full text search
• Stopword removal
• Faceting
• Filtering
• Matching (Autosuggest)
• Fielded search
• Field weighting
• Relevance weighting (tf-idf)
• Paging (offset and resultset length)
• Virtualisation (see virtual-norch)

Github: https://github.com/fergiemcdowall/norch

Mailing list: [email protected] - subscribe by sending an email to [email protected]

Build Status Master Branch:

#License

Norch.js is released under the MIT license:

Copyright (c) 2013 Fergus McDowall

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.