Compress the index for persistence about lofi-dx HOT 5 CLOSED

Before:

{
  "national": {
    "1": {
      "frequency": 7,
      "postings": [9,32,1039,1078,1189,1276,1310]
    },
    "2": {
      "frequency": 7,
      "postings": [9,23,530,574,611,761,830]
    },
    "3": {
      "frequency": 9,
      "postings": [15,42,426,492,659,717,813,855,1008]
    }
  },

  "other": {}
}

After:

{
  "national": {
    "1": "7/9,32,1039,1078,1189,1276,1310",
    "2": "7/9,23,530,574,611,761,830",
    "3": "9/15,42,426,492,659,717,813,855,1008",
  },

  "other": {}
}

Decreased the index size and memory allocation by about 50%. See scripts/index-stats.js

Additional compression can be left to users.

from lofi-dx.

Before: { size: '117 kB', memory: '162 kB' }
After: { size: '54 kB', memory: '63.8 kB' }

from lofi-dx.

Perform difference encoding for additional reduction in bytes. Example:

from lofi-dx.

Added delta encoding

from lofi-dx.

Added base36 encoding on top of delta encoding, which further reduces byte and memory size.

• Original: '19,42,157,472,685,709,1025,1051,1477,1571,1670,1837,1922,2038,2063,2084,2168' –> 76 bytes
• Delta encoded: '19,23,115,315,213,24,316,26,426,94,99,167,85,116,25,21,84' –> 57 bytes
• Base36 encoded: 'j,n,37,8r,5x,o,8s,q,bu,2m,2r,4n,2d,38,p,l,2c' –> 44 bytes

from lofi-dx.