Project Greenstack is intended to build a unified protocol all components of Couchbase may utilize to communicate with and within the cluster. The protocol is designed to address the shortcomings of the memcached binary protocol, and still be simple and efficient. See the one-pager for a full description of the rationale behind creating a new protocol.

The protocol is full duplex, meaning that both parties may send and receive packets at all times. This differs from the memcached binary protocol where we had a notion of a client and a server, where the client would send requests to the server and the server would only send responses to the requests. Being able to send notifications from memcached to clients/ns_server is something we’ve missed from the binary protocol. Examples for use cases could be:

• I'm starting to run out of memory, please slow down.. Today we're just accepting data until we hit a threshold and we start refusing stuff... we could have told the client earlier to back off..
• I got a new vbucket configuration map..
• I've initiated a shutdown of the bucket.. expect it to go away..
• I'm currently doing warmup, I’m done doing warmup
• Send messages back to ns_server for things to pop up in the UI (ex: we’ve had n number of incorrect logins the last minute, is the app misconfigured or is are we under “attack”)

Note: This means that client authors needs to be prepared for receiving other packets than a response for their request when they try to fetch the next frame off the network (and handle the command; which could be nothing more than sending not supported/unknown command)

Another difference from the memcached binary protocol is that the end receiving the commands may process and send responses out of order (as long as the fence bit isn’t set).

Everything that flows on the wire belongs to a frame, which is built up by a header, an optional extra header (referred to as Flex header) and a body. The first header describes the packet, and contains all of the information a simple proxy should have to look at. In the mandatory header there is also an optional flex header that may be utilized to build extended features or carry extra information. Finally there is a body where the payload for the command should go. There is a design decision to keep the header and the flex header within our own format, and just use Google FlatBuffers/Protobuffers in the payload. This allows proxies or command dispatcher to transparently do work with the frame without having to decode the frame.

All values in the protocol are specified in network byte order.

All data visible on the network belongs to a frame with the following layout:

The frame body consist of the following layout.

The minimum size of a packet is 7 bytes for a request and 9 bytes for a response (11 bytes and 13 including the frame header).

The opaque field is an array of 4 bytes the “client” in a request may use as a personal reference to identify the request in the response. The “server” for a request must provide the same value in the response.

The opcode is the actual action requested. It is defined per component:

All opcodes should be defined in a document with a description of the opcode and its stability tag (volatile, uncommitted, committed). See http://www.lehman.cuny.edu/cgi-bin/man-cgi?attributes+5 for a description of volatile, uncommitted and committed (TODO we need to adapt those terms to our own definition and update the document with that)

The flag section of the frame contains fields that is needed by protocol parsers that otherwise don’t need to decode the flex header. The flag section is defined in a future extensible way by allowing additional flag bytes to be defined. There should however be a justification for adding features as flags compared to adding it to the flex header.

The status code is a two byte value. It is defined per component with the following range:

The current set of status codes is defined in include/libgreenstack/Status.h<

The flex header length field is only present if the bit for the presence of a flex header is set in the flags section. The flex header length contains the number of bytes in the flex header.

The flex header allows for a future extensible way to pass arbitrary information to each command.

Each entry in the flex header contains the following three mandatory attributes (the value may however be of 0 length).

The following keys are currently defined. The length field in the table below defines the legal value for the length and must be present even if it is specified as a fixed width. No knowledge of the keys should be necessary in order to parse the flex header to pick out a certain field.

NOTE: We won't implement all of these initially, they're added here when we thought of them and they may be dropped or changed

After connecting to the advertised port the actor connecting to the port must start by sending the HELLO command to the other end to identify itself. Note that you may receive commands from the other end before (or instead of) the HELLO reply in the case the other end have other information it needs to notify you about (e.g. out of resources, not ready to accept clients at this time etc).

After a successful HELLO exchange the you should normally authenticate to the other end if applicable.

After you've identified yourself to memcached with the HELLO section you're not connected to any bucket, and have to run SELECT BUCKET in order to associate the connection with a bucket. By default you only have access to the "default" bucket, but if you authenticate to the server you may gain access to more buckets. This differs from the memcached binary protocol where running SASL AUTH authenticates and select the bucket.

The following section defines all commands that are considered generic and may be implemented by multiple components.

I'll be using the term client and server in the following chapters. A client is the party that initiates the connect, and the server is the party that the client connects to. It may very well be two servers communicating with each other.

The following section defines all commands memcached provides.

Mutations in Greenstack differs from the memcached binary protocol in the way that they're all implemented through a "mutation" command with an extra field in the command specifying the actual operation to perform. The motivation for doing this is that they all share the exact same code path within the memcached core, except for when the object is inserted into the underlying hash table. It makes it easier to extend the support for new kinds of mutation support if it means that we just have to update one location rather than updating the entire state machinery with a new opcode etc.

List buckets will only list the buckets you have access to

In order to track progress and make it easier for external parties to integrate with Greenstack, the development of the server follows the following plan.

It is a bit hard to set dates for some of the milestones at this time. As part of moving to Greenstack we'll be creating a detailed documentation of the new commands; may have to change the engine API and write unit tests. A rough estimate would be 1 1/2 day per command in average. When I've added support for a few of them its easier to predict the future (and the work involved..)

Greenstack is enabled by using protocol=greenstack for the interface entry in memcached.json.

ns_server needs to enable Greenstack protocol for a new ports (plain and SSL). This is targeted for Milestone 3.

try_read_command needs to be aware of the Greenstack protocol and dispatch the opcodes to the right underlying protocol handler. Initially we don't bother to try to be smart with respect to buffer handling (that's planned for milestone 7 with a potential move to bufferevents in libevent)

We need to refactor the current executor pattern in the memcached core so that both protocol reuse the same internal functions to implement commands.

• For encode I should allow for an iovector (so I don't have to copy the payload twice)

mkdir build
cd build
cmake ..
gmake all test install

mkdir build
cd build
cmake -G "NMake Makefiles" ..
nmake all test install

Please send feedback to [email protected]