mafintosh / append-tree Goto Github PK

View Code? Open in Web Editor NEW

54.0 54.0 13.0 80 KB

Model a tree structure on top off an append-only log.

License: MIT License

JavaScript 100.00%

append-tree's People

Contributors

Stargazers

Watchers

Forkers

watson blahah pfrazee d4tocchini andrewosh dcposch jasnell martinheidegger sirrujak

append-tree's Issues

README tweak

In the readme, you mention that the hypercore feed can be faked with:

any append-only log that supports .append() and .length

But in testing, I found more was needed. So I made a simple minimal interface.

// Fake hypercore feed for testing and learning
var values = []
var feed = {
  append (value, cb) {
    console.log('APPEND', {value, cb})
    values.push(value)
    nextTick(cb)
  },
  get (index, options, cb) {
    console.log('GET', {index, options, cb})
    nextTick(cb, null, values[index])
  },
  get length () {
    console.log('LENGTH')
    return values.length
  },
  ready (cb) {
    console.log('READY', {cb})
    nextTick(cb)
  }
}

Sandbox Breakout / Arbitrary Code Execution (brfs)

dat-node is affected by https://nodesecurity.io/advisories/548

brfs needs a patch browserify/brfs#83 after static-eval (and static-module) were patched.

Possible bug dealing with deletions

I'm writing a recursive delete algorithm, which looks like this:

function recurseDelete (archive, targetPath, st) {
  return co(function* () {
    // fetch stat if needed
    if (!st) {
      st = yield stat(archive, targetPath)
    }
    if (st.isFile()) {
      // delete file
      return new Promise((resolve, reject) => {
        console.log('unlink', targetPath)
        archive.unlink(targetPath, (err) => {
          if (err) reject(toBeakerError(err, 'unlink'))
          else resolve()
        })
      })
    } else if (st.isDirectory()) {
      // fetch children
      var children = yield readdir(archive, targetPath)
      // delete children
      yield Promise.all(children.map(childName => 
        recurseDelete(archive, path.join(targetPath, childName)))
      )
      // delete self
      return new Promise((resolve, reject) => {
        console.log('rmdir', targetPath)
        archive.rmdir(targetPath, err => {
          if (err) reject(toBeakerError(err, 'rmdir'))
          else resolve()
        })
      })
    } else {
      throw new Error('Unexpectedly encountered an entry which is neither a file or directory at', path)
    }
  })
}

That's being run against the following file-tree:

a
b/ (dir)
b/a/ (dir)
b/b
b/c
b/d/ (dir)
b/d/a
b/d/b
b/d/c/ (dir)
b/d/c/a
b/d/d
c/ (dir)
c/b/ (dir)

Specifically against the /b folder. A working log would look like this:

unlink b/b
unlink b/c
rmdir b/a
unlink b/d/d
unlink b/d/a
unlink b/d/b
unlink b/d/c/a
rmdir b/d/c
rmdir b/d
rmdir b

But the log I get is this:

unlink b/b
unlink b/c
rmdir b/a
unlink b/d/d
unlink b/d/a
unlink b/d/b
unlink b/d/c/a
rmdir b/d/c
Error: File not found

For some reason, rmdir('/b/d/c') is failing with File not found. If I run the entire method on the '/b/d' folder, no such error occurs!

I'm wondering if the record of the '/b/d/c' folder is getting lost somehow?

crash

[19871:0x261fab0]  9943111 ms: Mark-sweep 1425.6 (1470.6) -> 1425.6 (1470.6) MB, 2519.1 / 7.8 ms  last resort 

 <--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x10f09d0266a1 <JS Object>
    1: notFound [.../.nvm/versions/node/v7.10.0/lib/node_modules/dat/node_modules/append-tree/index.js:674] [pc=0x2ab92e0ec063](this=0x1f1940e08689 <JS Global Object>,names=0x259447e76411 <JS Array[3]>)
    2: /* anonymous */ [.../.nvm/versions/node/v7.10.0/lib/node_modules/dat/node_modules/append-tree/index.js:110] [pc=0x2ab92e0ebe8b](this=0x1f1940e086...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [dat]
 2: 0x12b82ac [dat]
 3: v8::Utils::ReportOOMFailure(char const*, bool) [dat]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [dat]
 5: v8::internal::Factory::NewTransitionArray(int) [dat]
 6: v8::internal::TransitionArray::Insert(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Map>, v8::internal::SimpleTransitionFlag) [dat]
 7: v8::internal::Map::ShareDescriptor(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::DescriptorArray>, v8::internal::Descriptor*) [dat]  8: v8::internal::Map::CopyAddDescriptor(v8::internal::Handle<v8::internal::Map>, v8::internal::Descriptor*, v8::internal::TransitionFlag) [dat]
 9: v8::internal::Map::CopyWithField(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::FieldType>, v8::internal::PropertyAttributes, v8::internal::Representation, v8::internal::TransitionFlag) [dat]
10: v8::internal::Map::TransitionToDataProperty(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [dat]
11: v8::internal::LookupIterator::PrepareTransitionToDataProperty(v8::internal::Handle<v8::internal::JSObject>, v8::internal::Handle<v8::internal::Object>, v8::internal::PropertyAttributes, v8::internal::Object::StoreFromKeyed) [dat]
12: v8::internal::StoreIC::LookupForWrite(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [dat]
13: v8::internal::StoreIC::UpdateCaches(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [dat]
14: v8::internal::StoreIC::Store(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) [dat]
15: v8::internal::Runtime_StoreIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) [dat]
16: 0x2ab92dc043a7
Aborted

should use `path.sep` instead of `/` for cross-OS compat

Right now paths are created using / to join and split paths - this makes the library not compatible with node fs on Windows. If all the / path construction and parsing was changed to use path.sep, it would be OS tolerant I think.

Typo in default opts

https://github.com/mafintosh/append-tree/blob/master/index.js#L653

Option for timestamps on operations.

I understand that timestamps are not-reliable, but going through the append-log by hand (if the user-interface is provided), doesn't give you a hint when a change happened. A simple option to store the timestamp could help when a human needs to interact with the data. Would you be okay with a PR that adds a timestamp option (default to false)?

Put a cache on split()

I found the split function was taking up a fair share of CPU in some workloads. I got an order of magnitude speed growth by adding a cache. Might be able to do something as simple as an object map of names to arrays that gets cleared every 15 seconds

mafintosh / append-tree Goto Github PK

append-tree's People

Contributors

Stargazers

Watchers

Forkers

append-tree's Issues

README tweak

Sandbox Breakout / Arbitrary Code Execution (brfs)

Possible bug dealing with deletions

crash

should use `path.sep` instead of `/` for cross-OS compat

Typo in default opts

Option for timestamps on operations.

Put a cache on split()

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent