I'm not sure if this still holds but I have a post-it note saying that if a ring event (yz_events) occurs while the Solr process is down then bad things happen. Investigate this behavior and fix if there is an issue.

It should be possible to perform a distributed search across buckets (indexes) that share a common schema by Solr's distributed search shards parameter. This would be useful when we wish to primarily store and search a subset of data within a bucket (e.g. a single customer's data) but at times we wish to search across a larger set of data (e.g. all customers).

Solr's distributed search shards parameter could be overloaded, so that when a client makes a query against the Riak cluster it passes a shards parameter that contains a list of buckets/indexes to query. Yokozuna could then replace this shards parameter with a list of the actual internal shards to use for the distributed search query.

While testing issue #104 I discovered that starting Yokozuna on-the-fly can cause the yz_cover server to crash because the other nodes aren't up and thus the coverage plan returns insufficient_vnodes_available. The cover server should be modified to deal with this return value and store undefined for the plan. That way a plan will be created during the query request and the query requests will fail (not the cover server) until a plan can be made and cache.

Verify that the Riak Java Client is able to run search requests against Yokozuna like it does Riak Search. This is vital for users that are upgrading from RS.

• Verify both HTTP and PB.

Create an interface for Yokozuna similar to RiakSearch. Any client communicating with RiakSearch should be able to point to this interface instead and expect everything to work the same. Unimplemented features (eg. updates) should be noops.

The biggest part of this issue will be implementing a webmachine resource, riak_solr_searcher_wm, which will query on /solr/INDEX/select

riak_solr_indexer_wm will exist, but be inoperative: /solr/INDEX/update

Update to Solr 4.3.0.

http://lucene.apache.org/solr/4_3_0/changes/Changes.html

• Lookout for logging changes - make sure logging works.
• Update codec/version in config.
• Take close look at solr.xml changes. Don't think anything has to change for this upgrade but there will be changes for Solr 4.4 and 5.x.

The list of TODOs need to be squashed. Most of these should probably be handled in a dedicated PR but going to use this top-level issue to track them. Mark as DONE when squashed.

• yz_doc.erl:186: %% TODO: I don't like having X-Riak-Last-Modified in here. Add
• yz_doc.erl:204: %% TODO: do this in KV vnode and pass to hook
• yz_entropy.erl:79: %% TODO: rename to_iso8601
• yz_entropy_mgr.erl:462: %% TODO: check for not_registered
• yz_entropy_mgr.erl:464: %% TODO: even though ownership change is checked this
• yz_entropy_mgr.erl:469: %% TODO: use async version in case vnode is backed up
• yz_entropy_mgr.erl:471: %% TODO: hashtree_pid can return {error, wrong_node}
• yz_entropy_mgr.erl:479: %% TODO: add timestamp so we know how long ago
• yz_events.erl:88: %% TODO: tick and ring_event should be merged, actions taken in
• yz_index.erl:84: %% TODO: Allow data dir to be changed
• yz_index_hashtree.erl:281: %% TODO: return _yz_fp from iterator and use that for
• yz_index_hashtree.erl:369: %% TODO: We should probably remove these warnings before final
• yz_index_hashtree.erl:433: %% TODO: handle non-existent tree
• yz_index_hashtree.erl:455: %% TODO: OMG cache this with entry in proc dict, use _yz_fp as Index
• yz_index_hashtree_sup.erl:17: %% TODO: should shutdown be longer to account for leveldb?
• yz_solr.erl:252: %% TODO: Encoding functions copied from esolr, redo this.
• yz_wm_search.erl:67: %% TODO: this isn't always XML, user can pass wt
• yz_xml_extractor.erl:60: %% TODO: warn if there is leftover data?

Currently extractors have access to a single value of a riak object. It would be useful for extractor developers to have access to the entire riak object in order to extract metadata. For instance, it would be useful to extract Link headers.

Verify that the Riak Python Client is able to run search requests against Yokozuna like it does Riak Search. This is vital for users that are upgrading from RS.

• Verify only PB, since HTTP is deprecated

Rename the app and sup modules to be yz_app and yz_sup to be consistent with the others. Leave yokozuna.app.src and yokozuna.erl alone. This follows the idiom discussed in "OTP in Action".

The default schema was copied from Solr's example and has a lot of extra cruft.

Yokozuna uses "doc-based" partition. This requires it to perform a "covering" query which means the larger the cluster gets the more nodes that must be contacted. Benchmarks should be performed for a few different cluster sizes ranging from 5-100. Perhaps 5/10/25/50/75/100. Scripts should be used to provision the nodes, run the tests, collect the raw results and log data, and create visualizations. This way the tests can be run quickly and may be reproduced by others.

I think expanding the tools under misc/bench to deploy on Joyent's cloud is the easiest path forward.

Lager 2.0 has a new feature that allows you to pretty print erlang records. This makes it much easier to read the logs. Go through all the logging calls and convert to this new style.

https://github.com/basho/lager/pull/100

Provide methods and documentation for migrating from Riak Search.

Checklist

• Add function to flip query service from RS to Yokozuna (basho/riak_api#31)
• Add riak-admin cmd to flip query service?
• Add YZ AAE status reporting (#154)
• Add YZ AAE status to raik-admin aae-status
• Add command for immediate re-index
• Test that AAE migration works
• Test that command migration works (i.e. some command to immediately re-index under Yokozuna)
• Document migration steps
• Document migration of schema
• Create example Solr schema that represents approximation of default RS schema

Migration by AAE

A nice benefit of active-anti-entropy (AAE) is that it provides a
"free" upgrade path from Riak Search to Yokozuna. After upgrading a
node queries would still go to Riak Search but new data would
temporarily be indexed by both Riak Search and Yokozuna. Over time
AAE would repair the missing data from Yokozuna, completing it's
index. After AAE has repaired everything the Riak Search hook could
be removed and queries could be redirected to Yokozuna. This simple
description hand-waves over a few things.

• How to determine when AAE has repaired all the data?
• What does it mean to "redirect the query"? Would it require the
  client switching the HTTP resource it points to? Would it be a
  dynamic re-binding in Riak so that the Riak Search Solr interface
  now points to Yokozuna?
• Riak Search tries it's best to look and act like Solr but at the end
  of the day it isn't Solr. The results, in terms of ranking and
  format, will probably change. How will this affect clients? Will
  the Riak Search client work against Yokozuna? Should there be a
  translation? Should we just force the client to use a real Solr
  client? If so, how do you properly integrate existing Solr clients
  with Riak clients? E.g. a client runs a Solr query for the keys but
  then runs a Riak query for the object data. Are there easy ways to make
  this one operation?

Currently the HTTP search resource (yz_wm_search) doesn't pass the incoming headers to Solr nor does it return the headers received from Solr. In order to have the best chance of working with Solr clients these headers should be passed verbatim.

tl;dr: If data is written while nodes are down then when those nodes come back up there will be a window where Yokozuna gives < 100% harvest on queries involving that data.

harvest = data available / complete data

While nodes are down data will be written to secondary vnodes. This is known as sloppy quorum. Also, Yokozuna will take the downed nodes out of query rotation.

When the primary vnodes come back up Yokozuna will put those nodes back in query rotation. In almost all cases this will happen BEFORE that secondary vnodes have handed the new data to the primaries via hinted handoff. If Yokozuna decides to use those nodes/vnodes for a query then it will have degraded harvest. But there is no reason it has to be this way because there are other replicas available.

Solution: Yokozuna coverage needs to take hinted handoff into account. If a primary vnode/partition is waiting for data from a fallback then it must be taken out of the query rotation.

Solr Cell is the integration of Apache Tika with Solr.
It allows Solr to index rich document formats like HTML, PDF,
Microsoft Office documents. It does this by providing a request
handler resource, called ExtractingRequestHandler. It takes a
document as input, feeds it to Tika, and uses SAX to produce events
that are created into a Solr document suitable for indexing. Solr
provides several URL parameters to control this process such
as literal.<fieldname>=<value> which allows a field-value to be
added to the created Solr document.

It's not immediately obvious how to integrate Solr Cell. It's
different from the current index path in Yokozuna. A different HTTP
resource is used and Solr doc creation happens on the Solr side, not
in Yokozuna. But Yokozuna currently assumes that all data must be
extracted, turned into a Solr doc, and then sent as an
update message. Perhaps this is a sign that the extractor
abstraction is too narrow for all use cases? The main question to
answer is:

How should Yokozuna differentiate between data that should go
through the extraction process versus data that should be passed to
Solr Cell?

A few of the steps for deploying EC2 nodes could be replaced with a shell script.

• Setting node name
• Setting cookie (prompt user)
• Listen on external interface for HTTP/PB (prompt user to verify this is okay)
• Create data dirs

No json is actually required, but the header is. It'd be nice to only require a content-type if a body is sent.

Add a Riak Test that verifies the behavior of the HTTP admin API.

Schema Admin

• Verify creating schema
• Verify 404 if no schema given
• Verify failure if bad content-type given
• Verify retrieving schema (make sure content type is application/xml)

Index Admin

• Verify listing indexes
• Verify creating an index with no schema given
• Verify creating an index with a custom schema
• Verify deleting an index

While building on FreeBSD 9.1-STABLE, the following scripts were unable to execute: priv/build-jar.sh and priv/grab-solr.sh.

I was hitting this issue when I was working on Riak - maybe @jaredmorrow knows this issue better than me.

The quick hack for FreeBSD is changing #!/bin/bash to #!/usr/local/bin/bash, but I don't recommend this because it includes an absolute path. Maybe #!/usr/bin/env bash? (Correct me if I'm wrong)

Currently Yokozuna calculates the coverage plan from scratch for EVERY query. Instead it would be good to cache the plans and round-robin them. If it's not too hard baking this into the riak_core coverage code would be ideal as other coverage operations would benefit. As always, verify the results via benchmark and analysis. I did some preliminary fprof analysis and it seemed to help a lot but make sure to copy results into this issue.

I have a branch with an initial stab at this cache, but it only caches one plan and thus might not distribute as well as it should.

rzezeski/yokozuna@master...rz-cache-plan

While looking at this it would be good to talk to @kellymclaughlin about adding the coverage code to reduce the number of unique nodes contacted and other work that he presented during our internal RICON talks.

Allow Yokozuna query results to be used as input to map-reduce
operations.

Currently Yokozuna indexes fallback data. This shouldn't be done for the following reasons:

• Queries are never run against fallback data. This is by design as fallback data, by virtue, is always a subset of data for a given partition. It is only written to when the primary is not available. If queries were to run against fallbacks than they would be neglecting the primary data set.
• Yokozuna doesn't cleanup fallback indexes. During ownership handoff Yokozuna makes sure to clean up the old partition's owner indexes after ownership has been completed. There are no checks to cleanup fallback data because it never should have been written in the first place.
• KV is the authoratative source for data. Yokozuna is merely an index for that data. Let KV handle the fallback cases and index that missing data when hinted handoff occurs.
• It uses up extra CPU/disk for no reason.

This issue should be fairly straightforward to test and fix.

• Write a riak_test that proves fallback data is being indexed. This should be easy enough by downing a node, writing some data, fetching the preflist, and then querying against the fallback node + partition combo. If that query returns > 0 then fallback data is being indexed.
• Change the yz_kv:index function to check P against the PrimaryPL. If it is not a member then don't index.

https://github.com/rzezeski/yokozuna/blob/master/src/yz_kv.erl#L216

The quorum values are hardcoded. It should use symbolic values instead.

Yokozuna uses Solr filter queries in order to eliminate overlapping data which would cause incorrect results to be returned. While running benchmarks on an 8-node baremetal cloud cluster (see issue #44) I found that the filter query can be built in a smarter manner and reduce the amount of work done per query.

Currently YZ builds a fq like this ('p' stands for partition):

(<nodeA> AND <p1>) OR (<nodeB> AND <p4>) OR (<nodeC> AND <p7>) OR (<nodeA> AND <p10>) OR ...

Notice how <nodeA> is repeated. Each unique node/partition combo gets its own group query. This can be compressed down like so:

(<nodeA> AND (<p1> OR <p10> OR ..)) OR (<nodeA> AND (<p4> OR <p13> OR ...)) OR ...

This reduces the amount of work Solr needs to do for each query. It can be reduced even further as the node filtering is only needed if an ownership change is in play.

<p1> OR <p4> OR <p7> OR ...

The first level of compression is an easy change to make. The second one will take a little more work as it requires keeping tabs on ownership changes, but it should be doable.

Finally, if the moons and stars align, you can remove the filter query altogether. This depends on several factors:

• ring size
• number of nodes
• ownership claim (e..g. v2 vs. v3)
• is cluster currently under ownership transfer

Removing the filter query gives a substantial boost in query performance. It should be easy enough to detect when conditions are right to remove the filter query. But I might leave that work to wait for another ticket. I just thought I'd mention it here so that the idea doesn't get lost.

First stage: #76

Change the ?YZ_DEFAULT_SOLR_VM_ARGS to default to []. We shouldn't force a default set of JVM options if the user removes the solr_vm_args from app.config.

Data stored in Riak is opaque to Riak. This means that Riak doesn't
know anything about the structure of the data being stored in it.
Whether it be JSON or JPEG, it's all the same to Riak.

On the contrary, the application storing the data often understands
the structure of the data. The application may want to tag the data
with attributes that give additional context. For example, tagging a
picture with information such as who uploaded it and when it was
taken.

The Riak Object contains metadata which is a set of key-value pairs.
This lends itself nicely to indexing because Solr expects a document
which is a set of field-value pairs. It's a matter of mapping the
metadata key to a field name.

In Riak the object metadata is an Erlang dict which is created
directly in the case of protobuffs or translated from HTTP headers in
the case of HTTP. This feature should work the same no matter the
method of creation.

API and Implementation Notes

TODO: UTF-8 in field names?

• The object's metadata may contain a yz-tags key. The value should
  be a CSV specifying the fields to tag. A comma must be used as
  delimiter and whitespace is ignored.
• For HTTP the header x-yz-tags may be used. Yokozuna will check
  for both fields.
• If a field listed in yz-tags doesn't exist it will not be
  considered an error but will be silently ignored.
• Tag names should match Solr field names declared in the schema.
• If a tag field name doesn't match any fields defined in the Solr
  schema then the result will depend on the schema config. In most
  cases I imagine this will cause an error.
• The value of a tag is always passed verbatim to Solr.
• Any metadata key may be tagged.

Example

The following is an example of HTTP headers conforming with the rules
above. The data is a fictitious picture uploaded by myapp.

yz-tags: myapp-where_loc, myapp-user_s, myapp-description_t
myapp-where_loc: Baltimore
myapp-user_s: rzezeski
myapp-description_t: Federal Hill at dusk.

The schema for this index might contain the following entries.

<dynamicField name="*_loc" type="location" indexed="true" stored="true"/>
<dynamicField name="*_s" type="string" indexed="true" stored="true" />
<dynamicField name="*_t" type="text_general" indexed="true" stored="false"/>

This example uses dynamic fields for each tag but a direct mapping
could be used as well.

yz-tags: myapp-description
myapp-description: Federal Hill at dusk.

<dynamicField name="myapp-description" type="text_general" indexed="true" stored="false"/>

An example of including metadata not specific to myapp.

yz-tags: content-type

Thought Process

Following is a summary of the thought process for this feature.

Mapping Keys to Fields

The first thing to decide is how to map a metadata key to a Solr
field. A naive approach might look like the 2i interface which relies
on a prefix and a suffix. The prefix signals the the key should be
indexed, e.g. x-riak-index. The suffix indicates the type of the
value, e.g. _bin. It's then a matter of deciding what the prefix
should be and how to map suffix to Solr fields.

In fact, why not use the 2i interface since it is already defined and
supported by Riak clients? There are several problems with this
approach.

• There is only support for binary (_bin) and int (_int) types.
  Solr supports many more types.
• What Solr type should _bin map to? It could reasonably map to
  different types like text, text_general, string, etc.
• What if _bin should be mapped to a different type depending on the
  data? What is the convention for specifying the Solr type?
• What happens when leveldb is being used? Should both 2i and Solr
  indexing occur? What if the user only wants one of the systems to
  index the data? How does the user tell the system what to do?
• The 2i interface is now performing double duty. Any change to its
  API needs to take both systems into account.

Yokozuna could have a dedicated prefix/suffix convention but then
there is an additional layer of translation between Yokozuna and Solr.
This adds complexity in the system and causes additional burden to the
users.

Solr already has a mapping from field name to type. This should be
exploited. If a user adds a summary tag then there should be a
summary field in the Solr schema. If the user adds a summary_t
tag then it should match the *_t dynamic field. This is obvious and
requires no translation layer. That handles the suffix issue, but
what about the prefix? How does Yokozuna know when a metadata key
is a tag?

A prefix like tag- could be used. That's not so bad but it causes
metadata to be polluted with prefixes. Also, what if the user wants
to tag keys like content-type, which already exist? Should a
duplicate value be made with a tagged key? Can the prefix be avoided
altogether?

Why not be explicit about which metadata keys are tags? Add a special
metadata key named tz-tags containing a CSV of the metadata keys
to tag. On the downside it requires an additional metadata entry but
on the upside it's explicit about what is being tagged, leaving the
user in full control. It doesn't require remembering a set of rules
about when metadata is tagged. This seems like the best possible
solution.

Tags With CSV

In most cases it is obvious that the tag value should be sent to Solr
verbatim. What about a comma separated value, CSV? Solr has the
notion of multi-valued fields and it might make sense to treat a CSV
as such a field. Once again the problem is discovering what semantic
the user wants. A prefix of suffix could be used. Perhaps a special
syntax for the yz-tags value, e.g. yz-tags: keywords(csv) would
indicate a CSV. This could work but once again requires translation
and user education. I think this should be avoided at all costs.

Instead, the tag value should be passed to Solr verbatim in all cases.
If a CSV should be converted into a multi-valued field then the user
should update the Solr config or schema to interpret it as such.

I think this could be achieved by either "poly fields" or a custom
Field Mutating Update Processor.

A Note on X- Headers

Notice the examples above are not using the X- convention for custom
HTTP headers. The IETF deprecated this style in RFC6648 because
it causes problems for existing systems when a custom header becomes
part of the standard. Also, using a prefix specific to your
application is sufficient.

A nice feature of the proposed plan above is that the user is in
complete control of the header names. Either style may be used.

https://gist.github.com/4207723

Currently there are two json extractors. One of them adds field name suffixes based on the type so that users can store JSON data using default schema without adding the suffixes. There is really no reason for the suffix extractor to be stand-alone. Add an option to the default JSON extractor that tells it to auto-add the suffixes. E.g.

yz_json_extractor:extract(Value, [{auto_suffix, true}])

Would be good to basically support all the content types that are currently supported by riak_search

A noop extractor creates no fields. It has several uses.

• If the data is opaque, such as JPEG of PDF, then there is no easy
  way for Yokozuna to extract fields from the content. Tags can be
  added via object metadata and SolrCell/Tika can extract fields from
  the content.
• An extractor might exist for the data, such as XML, but perhaps the
  user wants to only tag the data or let SolrCell extract fields.
• If the content type has no registered extractor it is best not to
  assume anything about the data.

In both cases above I mentioned SolrCell. I'm not sure yet how
Yokozuna interacts with that. The point is that a noop extractor
should exist so that a user can tell Yokozuna not to extract fields.

Specification

• The module name should be yz_noop_extractor.
• It should support the extractor API, see the extract function in the
  XML extractor.
• It should ignore the value and return [].
• The default map should be changed to use noop as the default.

Go through EVERY LINE in default_schema.xml, understand what it does, and remove it if it doesn't make sense to be part of the default schema for Yokozuna.

Verify that the Erlang Riak Client is able to run search requests against Yokozuna like it does Riak Search. This is vital for users that are upgrading from RS.

This PR lives here: basho/riak-erlang-client#112

Upgrade to Solr 4.2.0.

• update luceneMatchVersion
• revert back to using binary package for Solr, create script for building custom binary package and upload to s3
• run a few of the fruit query benchmarks and see if perf improves at all
• run the riak_test tests

• Update build process to pull the 4.0 final release, stop tracking branch
• Compare the latest exampleapp schema with current default schema
• Compare other supporting files such as stopwords, etc.
• Run riak test to verify main functionality still works

Investigate if the schema name and version fields are used for anything in Solr. Should Yokozuna use them for anything? At minimum, perhaps the default schema should have it's name changed to default.

https://github.com/rzezeski/yokozuna/blob/master/priv/default_schema.xml#L48

If a bad schema (e.g. malformed XML) is written and an index is created which uses that schema then it will go into an endless loop of failing to create the index.

• See if it is possible to verify the schema during creation, if Solr doesn't provide anything at least make sure xmerl can parse it. (Think about using a temporary Solr Core to verify the schema).
• In case the previous step doesn't exist or is some how made ineffective, check for this condition and automatically cancel the index creation.
• Perhaps the schema should stay in KV (i.e. don't delete it) but be marked as "bad" so it can't be used in future index creation until it is fixed.

As we push closer to a production release, we should have a Riak client or two which can manage/query YZ. I lean toward Ruby via HTTP, but any will do.

In order to more easily work with existing Solr and Riak Search clients Yokozuna should listen on the canonical Solr search URL.

solr/<index>/select

This will cause issues if Riak Search is also enabled but that's okay for now. Smarter logic will be added later to deal with upgrade scenarios.

If the Erlang VM crashes then the shell wrapper for Solr
will consume 100% CPU until the Java process is killed. I'm hoping
this is just an oversight in the script like handling a signal. I
don't particularly like that Yokozuna uses this wrapper but when I
wrote this piece months ago I couldn't find a way for Erlang ports to
send signals to processes. I'm open to alternative approaches.

When the index flag is modified actions must taken to invalidate the current hashtrees or else AAE will think data is convergent when it isn't.

Enabling Index Content Flag

• Delete by query all documents in default index for the bucket.
• Clear all yokozuna hashtrees

When writing data without the flag set Yokozuna will do minimal indexing. Basically just the special fields. It will write these documents to the default index which is in turn read by the AAE process. This must be done or else the YZ AAE and KV AAE trees won't match up. This means that when the flag is set AAE will say everything is good but in reality it isn't because none of the previously written objects have their content indexed when they should. Therefore the default index data for this bucket must be deleted so that future AAE hashtrees don't read it and all the hashtrees must be cleared so that they can rebuild and discover that the data for the bucket needs to be re-indexed.

This will allow AAE to re-index in the background but it could take a while. In this scenario Yokozuna should also do a pre-emptive re-index of this data which it could probably do in a more efficient manner than full-on AAE. E.g. you could use yokozuna itself to iteratively list keys for the bucket and then feed them into a riak_pipe flow for re-indexing.

Disabling Index Content Flag

• Drop the Solr core for the bucket.
• Clear all Yokozuna hashtrees.

This is pretty much the same as the first scenario except you can just drop the entire index/core. The same issue arises in that you want to move the indexing from one core to another. In this case you now want to drop full indexing in the dedicated index and do minimal re-indexing in the default index.

There are many reasons why a JVM may continue running after it's parent erlang process has died, from core dumps to running older Erlang prior to R15B02.

The solr JVM should run a thread which occasionally checks if its PPID=1. If so, the erlang parent process is crashed, and the JVM process should System.exit() itself.

Note that there are two parts to this task:

1. get the solr web service to launch a low priority background thread that runs every few seconds.
2. having that thread check for a ppid (which will require some external ps call, or better yet, just writing a script which checks).

Note that the JVM does not have any built-in method for getting a PPID, and many of the "get the PID this way" hacks found out on the interwebs are unreliable (eg. ManagementFactory.getRuntimeMXBean().getName() is not guaranteed).

Riak Branches

rz-yokozuna

Contains changes need to include Yokozuna with the Riak release.

yz-merge-1.3.0

All commits needed to build Yokozuna against Riak 1.3.0, cherry picked from above branches. Additional modification to rebar.config, local to this branch, to pull in the yz-merge-1.3.0 branch of KV.

git cherry-pick -x origin/rz-yokozuna

Riak KV Branches

rz-yz-index-objects

Code to send object updates to Yokozuna so they may be indexed.

rz-yz-aae-status

Code to make AAE status work with Yokozuna.

yz-merge-1.3.0

All commits needed to integrate above branches with KV version 1.3.0.

git cherry-pick -x rz-yz-index-objects~1 rz-yz-index-objects
git cherry-pick -x ..rz-yz-aae-status

There were a few last minute changes to AAE for Riak 1.3 that should be read over and potentially ported to Yokozuna.

• basho/riak_kv#483
• basho/riak_kv#486

Yokozuna should have the ability to create and modify schemas
remotely, HTTP for this specific issue.

There are still a lot of questions regarding this issue. The
fundamental ones are:

• Can a running system cope with schema changes? If so, how can it be
  done safely?
• Can schemas be modified piecemeal or must it be all-or-nothing?
• Can JSON be used to read/modify/write the schema?
• Should concurrent writers/siblings be accounted for? I'm a little
  less worried about concurrent writers in a healthy cluster and more
  worried about partitioned writes. Would PW/PR/DW/W/R=N be good
  enough?

Specification

• The resource: <host-port>/yokozuna/schema/<schema-name>

GET

• Return the schema with content type of application/xml.
• TODO: allow to pick-out subset of schema to return, e.g. a list of
  fields?
• TODO: allow to return in JSON format?

PUT

• Accepts text/xml or application/xml.
• The body is a properly formed Solr schema. See the
  example schema.
• If the schema name already exists then don't replace the current
  one. Instead return an error to user stating it already exists. Need
  to be able to overwrite a schema in case a bad schema is uploaded.
• TODO: Think about adding param overwrite=true to bypass the
  previous check allowing the user to overwrite the current schema
  definition. This has to be thought about carefully because changing
  schemas could cause issues.

POST

TODO: Think about allowing POSTs to add to or modify a subset of a
schema. E.g. adding a new field without read/modify/write of entire
schema.

DELETE

TODO: Do we allow deletes of schemas?

Since yokozuna piggy-backs off the KV vnode there is no need to have a vnode in yokozuna. Removing the vnode will cause the yokozuna service to never be marked as up. I'm not sure that matters but even so a call can be made directly to riak_core_node_watcher:service_up.

Why is it bad?

Collocated replicas cause incorrect and non-deterministic query
results.

When can it happen?

Any time the number of active nodes is less than the target_n_val.
Given the default value of 4 this could happen with a cluster of 3
or less. It could happen in a 5 node cluster when 2 nodes go down and
new or existing data is written. It could happen in a 10 node cluster
where a partition occurs leaving two cluster of 7 and 3. The 7 node
partition won't have collocated replicas because it has more than
target_n_val nodes but the 3 node side doesn't.

What is the cause?

The fundamental cause is the assumption that each partition has a
dedicated backend instance. Normally, every partition has it's own
instance of bitcask, leveldb, etc. Given a preflist where two
replicas are on the same node there are still two distinct copies of
that object on two different instances of the backend. Partitions do
not share backend instances.

Yokozuna DOES share one Solr instance and one Core instance for
all partitions on the node. By default, each Solr document must
have a unique identifier. When a document is written with an existing
id then it overwrites the old one. Yokozuna writes documents using
the Riak key as the unique id and a field that lists the single
owning partition. The following is an example.

<doc>
  <id>riak_key</id>
  <_pn>2<_pn>
  <text>I'll take bolth, please.</text>
</doc>

Querying will work as long as this is the only replica of key
riak_key on this node. But if another replica is written then it
will squash the previous value for _pn. This means that queries
which filter by partition 2 will come up empty when they shouldn't.

Verify that the Ruby Riak Client is able to run search requests against Yokozuna like it does Riak Search. This is vital for users that are upgrading from RS.

• Verify PB.

This PR lives here: basho/riak-ruby-client#113

@kenperkins mentioned wanting a mechanism to move from Search to YZ in the future. It would be a nice thing to provide.