binary attachments on an enhanced DeployData message about rchip-proposals HOT 6 OPEN

From the discussion in tech gov today considering compression of hex and conversion to binary, along with the inevitable size limits suggesting chunking is necessity and streaming will necessarily be done in rholang using linked lists of up to about 10 meg chunks each deployed separately. Compressing the rholang deploys on chain is under consideration in which case the hex compresses well and that was a simpler more general way to save space at this time..

The issue of retention was raised, perhaps using a timestamp or block height for expiration and maybe a deployId or some unforgeable to extend the expiration.

from rchip-proposals.

As @jimscarver mentioned, chunking can now be done on the deploy level with hex encoded string.
Using one deploy with large amount of data makes difficulty with gRPC because it has max message limit except used in streaming mode. Also validation of this kind of large deploy is becoming more complex.

I've tested the difference (or impact) of storing binary as hex and conversion.

[1] Storing hex string on a channel

new return(`rho:rchain:deployId`), x in {
  x!("<bytes>") |
  for(@a <- x) {
    return!(a)
  }
}

[2] Storing binary on a channel

new return(`rho:rchain:deployId`), x in {
  x!("<bytes>".hexToBytes()) |
  for(@a <- x) {
    return!(a)
  }
}

Cost in phlogiston of storing hex string with or without conversion to binary.

Calling hexToBytes has constant overhead but it has lower cost per byte.

from rchip-proposals.

As part of REV vault changes @Isaac-DeFrain created this example of linked list which can be used to store chunks of binary data.

new
  empty,
  cons,
  print,
  stdout(`rho:io:stdout`)
in {
  // adds an element to the head of an existing linked list
  contract cons(@value, pointer, ret) = {
    new elem in {
      elem!(value, *pointer) |
      ret!(*elem)
    }
  } |
  // prints all elements in the list from head to tail
  contract print(elem, ret) = {
    for (@value, @next_elem <- elem) {
      if (value != Nil) {
        ret!(value) |
        print!(next_elem, *ret)
      }
    }
  } |
  // build a linked list and print the elements from head to tail
  new tmp in {
    empty!(Nil, Nil) |
    cons!(2, *empty, *tmp) |
    for (@elem <- tmp) {
      cons!(1, elem, *tmp) |
      for (@elem <- tmp) {
        cons!(0, elem, *tmp) |
        for (@elem <- tmp) {
          print!(elem, *stdout)
        }
      }
    }
  }
}

from rchip-proposals.

So clearly large amounts of data have to be split between deploys.

But still, for chunks of moderate size, the cost of hex-encoding seems fairly high. thanks for the measurements for the cost of evaluation, @tgrospic . We also have a charge for parsing the rholang source code containing the hex string, before interpreting it, right?

from rchip-proposals.

Meanwhile, I gather there's a bittorrent connection in progress; surely that would render this moot. Anybody have a pointer handy?

from rchip-proposals.

So clearly large amounts of data have to be split between deploys.

But still, for chunks of moderate size, the cost of hex-encoding seems fairly high. thanks for the measurements for the cost of evaluation, @tgrospic . We also have a charge for parsing the rholang source code containing the hex string, before interpreting it, right?

My measurement showed that hexToBytes conversion has constant overhead which means when parsing is done all binary data is already converted and conversion from hex is not dependent on size of data.

Cost of parsing is 1 phlogiston per byte of source code.

from rchip-proposals.