beakerbrowser / webdb Goto Github PK

table.add and other methods will fail with 'Invalid URL' exception in node.js (not under Electron) because node's URL type cannot parse 'dat:' protocol.

Curious to know if 'WebDB' would be open to a pull request allowing for the ability to pass in a custom file parser. Specifically, I'd like to use smarkt, but it looks like anything that implements a stringify method that can consume JSON and a parse method that spits out JSON, should work correctly. Happy to help if it's a useful idea, no worries if not :)

i.e.

var db = new WebDB('dat:// 494898...', { parser: require('smarkt') })

For example, as Fritter users WebDB, how it indexes all users without needing a server?
Does UnwalledGarden standard uses WebDB?

Sorry if this is a rather silly question, but I'm trying out the examples - I've got a /data directory and I've ran dat create and dat share in it. But I get "QueryError: Unable to put(): the given archive is not owned by this user" when I run:

const DatArchive = require("node-dat-archive");
const WebDB = require("@beaker/webdb");
const assert = require("assert");
var webdb = new WebDB("./data", { DatArchive });

webdb.define("people", {
  // validate required attributes before indexing
  validate(record) {
    assert(record.firstName && typeof record.firstName === "string");
    assert(record.lastName && typeof record.lastName === "string");
    return true;
  },

  // secondary indexes for fast queries (optional)
  index: ["lastName", "lastName+firstName", "age"],

  // files to index
  filePattern: ["/person.json", "/people/*.json"]
});

const doStuff = async () => {
  await webdb.open();
  await webdb.indexArchive('dat://<urlFromDatShare>')

  await webdb.people.upsert(
    "dat://<UrlFromDatShare>/bob.json",
    {
      firstName: "Bob",
      lastName: "Roberts",
      age: 32
    }
  );

  var mrRoberts = await webdb.people.get("lastName", "Roberts");
};

doStuff();

How do I specify that I own this dat? What silly thing am I doing wrong?

My usecase: I'm adding some timestamps when serializing, and it would be nice to be able to return those from my API.

Should be a one-line change from what I can tell! :)

WebDB uses DatArchive's defaults and does not pass user options to it. I'd like to be able to pass options through in order to, for example, use custom discovery servers.

When indexing DBs, if you do:

async () => {
    await webdb.indexArchive('dat://232ac2ce8ad4ed80bd1b6de4cbea7d7b0cad1441fa62312c57a6088394717e41/');
console.log(await webdb.listSources());
}

You will see ["dat://232ac2ce8ad4ed80bd1b6de4cbea7d7b0cad1441fa62312c57a6088394717e41"] in your console (note, no trailing /).

If you do this:

...
console.log(webdb.isSource('dat://232ac2ce8ad4ed80bd1b6de4cbea7d7b0cad1441fa62312c57a6088394717e41/');
...

You will see false, because the / at the end is missing. It seems like isSource does an exact string match, when the thing that's important is the dat hash.

A little thing, but it was throwing me off for a bit :)

Any thoughts? It's simple enough, I might make another PR for this unless you have any objections...

I've been playing around with this trying to learn it (very exciting!!!), and was really struggling to get any records to show up in my test DB. As I drilled into things, I realized that I had a copy/paste error in my code:

await db.define('people', {
      // validate required attributes before indexing
      validate(record) {
        assert(record.name && typeof record.name === 'string')
        assert(record.bio && typeof record.bio === 'string')
        return true
      },
    ...

assert is not defined and thus a Reference Error was getting thrown, but the try/catch block in indexer.js just sets isValid = false; and ditches the error, so I had no idea what the problem was. What are your thoughts of throwing the error if it catches one? It'll be a simple enough fix, so you may see a PR from me later 😬 .

Here's the snippet from indexer.js I'm talking about:

async function readAndIndexFile (db, archive, filepath, version=false) {
     ...
    // index on the first matching table
    for (var i = 0; i < tables.length; i++) {
      let table = tables[i]
      if (table.isRecordFile(filepath)) {
        // validate
        let isValid = true
        if (table.schema.validate) {
          try { isValid = table.schema.validate(record) }
          catch (e) { isValid = false } <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
        }

Any place where an example is being used?

After WebDB has indexed a file, if that file is then removed (outside of webdb), a NotFoundError is thrown. The record for that file seems to remain in the database.

This may be a "works as designed" situation, as I may not be thinking of all use cases. However, it seems like it would be a good idea to handle this error and unindex the file, when that indexed file is removed from the archive. This would probably be a catch (maybe emitting a 'index-file-missing' event or something) on archive.download() in the following line:
archive.fileEvents.addEventListener('invalidated', ({path}) => archive.download(path))
https://github.com/beakerbrowser/webdb/blob/master/lib/indexer.js#L48

I'm happy to submit a PR, but thought I'd open an issue to be sure I'm not missing something.

In the api docs the constructor has the signature new WebDB([name]). However, in the initial nodejs example it has what looks to be the signature new WebDB(path[, options]). I see a mention of the path, but I don't see anything about the options. Also is the path still optional in node?

Side Note: these types of environment dependent dynamic signatures are kind of a pain in the ass to work with in typed pre-processors like Typescript & Flow (edit: specifically if name is optional and path is required in this specific case).

A design tradeoff for WebDB that I didn't consider enough was bundle size.

• In the most recent release, JSON-Schema was added for validation. Turns out my module choice (ajv) adds 200kb to the output bundle (unminified).
• The level.js choice probably adds another 300-400kb.

It'd be relatively easily to remove JSON Schema and replace it with a validation function. That function could even be implemented downstream using JSON Schema! But it'd be up to the app dev. EDIT done.

Removing level.js would be significantly harder.

I'm looking to add many records for a table, so people/*.json. Should I manually define the id, like:

import { v4 } from 'uuid';

await webdb.people.put(`dat://myaccount.hashbase.io/people/${v4()}.json`, {
  firstName: 'Bob',
  lastName: 'Roberts',
  age: 31
});

Or is there a better way to do that? Also, how do you automatically create the folder?

In the CRUD examples on bobsUrl, is another person's dat being modified? I assume not, but it's not clear to me.

Take for example:

await db.profiles.update(bobsUrl, {name: 'robert'})

What is this doing? Is it updating a profile json file on a Dat from another source?

When calling .delete() from the browser I recieve:

Error: Cannot .delete() databases outside of the browser environment. You should just delete the files manually.

I had a nagging feeling that I misconfigured something in my project (so level-browserify is pulling the wrong level), but after a bit of sleuthing, nothing obvious is standing out to me.

Hey!

What do you think about having live-updating DB queries, as a more user-friendly interface on top of the WebDBTable events put-record and del-record?

An example:

const query = webdb.people.where('lastName').equalsIgnoreCase('roberts')

query.on('put', console.log) // {url, origin, indexedAt, record}
query.on('del', console.log) // {url, origin, indexedAt}

As I understood it, WebDB instances emit indexes-updated events to let others know that they have updated records in all indexes from a single archive. I would love to have the same for all archives it currently keeps track of.

Currently a rebuild event (triggered by a table definition change) will cause all indexes to be rebuilt. We can update this to only rebuild the affected tables.

const WebDB = require('@beaker/webdb')
var webdb = new WebDB('webdb-example')

when i try to instansiate webdb inside an async function in order to use it like it is written in the example you published here, the debugger responds with: require is not defined. what i'm doing wrong here?

Do i have to import module.js ?
or how to import browserify?