package name... about search HOT 5 CLOSED

I'm in the process of working on a few changes to this same stuff along with speeding querying up a lot. I'm pretty new to Node and any suggestions would be a great help. I'm open to changing the name, how about node-search?

from search.

sounds good...i'll keep an eye on the repo for when your changes land...i committed a small reorganization a bit ago if you want to pull that

i'll poke around some more and see if there's anything that jumps out at me...the most obvious thing is the apparent buffering of the entire data set -- is that something you're working on? the nodish thing would be operating on the readStream you would get from nStore bit by bit...

of course, this would be problematic if you want to stay within the realm of commonjs -- the alternative is to use forEachables (e.g. LazyArray [1]) for streaming...this is what perstore does [2] -- it allows consumers to operate on result sets like they were arrays but asynchronously and without buffering, so it's a pretty nice abstraction for this kind of use case (for example note how in this commit i adapted your indexer function to accept a lazy array as well as a normal array [3])

speaking of node-specific -- is that the direction you plan on going? if so, that's fine -- and node-search makes sense...nsearch seems like a common convention for node modules these days too (and at least to me doesn't carry the same implication that the package is only usable in node)

[1] http://github.com/kriszyp/promised-io/blob/master/lib/lazy-array.js
[2] http://github.com/kriszyp/perstore
[3] http://github.com/deanlandolt/Search/commit/c277236f3a2e9aaae47d009c1e4e22414464a571

from search.

Thanks for the info, I really appreciate it. I'm not sure what you mean by buffering the data set but if you mean creating the array of documents to search over every time we make a query then yes. I've taken this out and the search speed is a small fraction of what it was before but I'm really concerned about storage of the index.

I would prefer not creating dependencies of specific data stores, and if I was to choose one data store to use I'm tempted to choose redis over nstore. What do you think? I think the choice really depends on what you are making.

I will check out these projects and try to take in the ideas in the changes that you have made. Thanks for the support.

from search.

I've made lots of changes with this stuff in mind but one of the biggest changes is that you can now index one document at a time instead of needing to do everything at once. This makes if so that the data can be streamed in as it is loaded.

But I have to admit I'm having difficulty really understanding how asynchronous code works in Node. I've posted a question to http://groups.google.com/group/nodejs/browse_thread/thread/ab08a21cd50d9839 to try to get some insite. I would like to do this stuff the right way but there is so few people writing about it because so much of it is so self explanatory I guess. Rather than use someones library I would like to understand how these things are made. If you have any pointers to the basics, they would really be of interest to me.

from search.

I've made the changes I was looking to make in this round and have posted to the new project name if you are interested. I'd be interested to know what you think of this direction. http://github.com/talltyler/node-search

from search.