stuhood / gaffe Goto Github PK

This may involve polishing the disk format: it's definitely not finished, and the column oriented layout described in http://www.google.com/buzz/goog.research.buzz/WsARqxc7d7R/Dremel-Interactive-Analysis-of-Web-Scale-Datasets is very interesting.

Iterating over all of the adjacencies should be sufficient initially.

The worst case is that a query wants absolutely everything stored in a generation. Therefore:

• A MemoryGen's thread should collect the entire resultset for a query atomically (and quickly), since the worst case only creates a 2nd copy of all references in a MemoryGen, which we know fits in memory.
• A PersistedGen on the other hand should have a pool of threads (each with their own reader state/buffers), since the results might not fit in memory.

Read queries will consult the generations in stack order, but may perform writes back to a 'cache' generation (between the head and tail of the stack) to cache a path, or to ask a hot path to be persisted.

Will probably require an overhaul of the memory layout to match the disk layout more closely (ugh, "fool me once...").

Structural queries involve 'fill in the blank' edges and vertices, anchored on at least one known vertex.

Generations should be queried in sorted order. Vertices in newer generations may refer to their locations in previous generations, or indicate that they are definitive (somehow), such that no previous generations need to be consulted.