oada / client Goto Github PK

Promise.map(['a', 'b', 'c'], async (i) => { 
    await CONNECTION.put({
        path: '/bookmarks/test/'+i,
        tree: {
            bookmarks: {_type: '...', _rev: '...', 
                test: {_type: '...', _rev: '...', 
                    '*': { _type: '...', _rev: '...'} 
                 }
             }
        },
        data: {foo: 'bar'}
    })
})

The result:
Only 'a' gets created as a resource under /bookmarks/test. The remainder are just plain objects.

The temporary fix to just get a tree created properly is to use Promise.each. However, because we're not actually protected against concurrent writes, this is only a solution for a single client trying to get a tree setup.

I spent some time going through the code and it seems a bit difficult to fix the way iteration is setup. This code is very clean and works well for single clients/non-concurrent writes. I like that it creates the entire "new" tree of data outside of the bookmarks tree then one final link joins this to the bookmarks tree. However...this is also its downfall: if when it reaches that final link into the main bookmarks graph it finds a link already there, the whole thing must essentially be thrown out as any common children resources are duplicates.

The solution unfortunately entails a bit more complexity. It requires that we first find the highest existing resource and work down from there (as originally implemented in oada-cache). Then, we can actually perform the proper if-matches as we create those links from parent to child.

More detailed problem walkthrough:

Because of the way linkObj is setup on the iteration prior to when it gets written/put, the if-match check will retry the current iteration, but it will still use the wrong linkObj setup in the previous iteration, clobbering 'test' resources.

In the example above, the first iteration creates the three /bookmarks/test/* resources, for example /bookmarks/test/a. Also prepared on this iteration is a linkObj of {_id: 'resources/a'} to be used on the next iteration, so that the parent can point to it.

On the second iteration, /bookmarks/test does not yet exist as a resource and so it is created once for each item mapped over, (e.g. resources/testa, resources/testb, resources/testc all get created). The body of those new resources is constructed from the child path and the linkObj, e.g., {a: {_id: 'resources/a'}}. The linkObj prepared for the next iteration is {_id: resources/testa}.

On iteration three, /bookmarks exists. Because bookmarks exists, a different code path executes the following put using the linkObj and an if-match on the bookmarks rev retrieved when we checked that it existed, e.g.,: {path: /bookmarks/test, body: {_id: resources/testa}, rev: 2}. The if-match check does its job and makes 'b' and 'c' retry this current iteration (because it was mapped, they keep pace with one another), but even when the correct rev is retrieved on /bookmarks, the put body (linkObj) is still wrong.

I was thinking this would be a good time make API changes if we want. We can maintain an external layer that presents an 'oada-cache' style API ... with the hope that a new API is cleaner (more typeable) and eventually wins.

Here is an example of the API that comes to mind. This is driven by my use case in 'oada-jobs'. Would love to have a discussion on this. If you are agreeable, I would be happy to help make it so.

import { connect, /* LRU */ } from '@oada/client'

// For now just WS, but could have others on day
const conn = new OadaWS("example.oada.com");

/* Maybe "wrap" in a cache layer?
 * conn = LRU(conn);
 */
// Note: type annotation is just for example clarity
// Note: oada1, oada2, ... live at the same time and they
//       need a way store the token separately 
const oada1: OadaClient = conn.connect(token1);  
const oada2: OadaClient = conn.connect(token2);

/*
 * all of these use the same WS socket from above but are
 * authed with the correct token
 */
oada1.get(path) // Return type? Should leverage @awlayton's oada-types as possible
oada2.put(path, body, options) // Return type? Should leverage @awlayton's oada-types as possible
oada1.watch(path).on('all|new|edit|delete', (change: OadaChane, oada: OadaClient) => {
	// where `oada` is the OadaClient that recieved the change
})

EDIT: Removing clone and setToken as they are not needed. We will make conn.connect() "cheap" instead.

We need some ability to await until a particular path or value shows up on something we're watching. This is something that jobs will likely implement as a core feature.
It should make it a simple one-liner to implement this feature and utilize this feature where the code will continue after the condition is met.

The client seems to send _type property when creating a link from a tree.

After making around 1000 requests concurrently, client's queue handling begins processing, but node exits prior to finishing. Code after the map does not execute.

await Promise.map(Object.keys(data), async (key, i) => {                              
  let r = await CONN.put({                       
    path: `/bookmarks/foo/bar/${key}`,
    data: {"foo": i}
  }).catch((err) => {                  
    console.log(err);
  })                            
}, {concurrency: 1000})
.catch(err => {
  console.log(err);
})```

Temporary fix is to keep node alive with a `setInterval`.

Now that the WebSockets auto reconnect, if you try to connect to an invalid domain client just hangs indefinitely while the socket repeatedly tries and fails to connect.

It would be good to actually get and error instead if the initial attempt fails.

Because, why not?

So you can't figure out what ksuid key was generated from your POST.

@awlayton's @oada/oadaify is pretty cool. It would be even cooler if @oada/client just oadaified things, so that the problems @oada/oadaify's solves are just no more.

put /bookmarks/services/target/jobs/
{ _id: 'resources/' }
2:30 PM what that does in oada/client is create a new resource and link it under
2:30 PM if you use the tree
2:30 PM i.e. the tree tells it that jobs/ should be a resource, and that resource doesn't exist
2:31 PM so it decides to make one, then put { _id: ... } to it which is basically a no-op
2:31 PM I believe we fixed this in oada-cache long ago, but new client hasn't had all the edge cases yet (edited)
2:31 PM so you can get around it by putting the jobkey in the body instead of just the link
2:31 PM but if you put to the jobkey in the url, tree put doesn't work

Client believes the content-type and returns a Buffer under the 'data' key when requesting the _meta document of a resource that is an application/pdf.

Current fix seems to be JSON.parse(response.data.toString())

I think the content-type of _meta documents need to be carefully considered.

Previous discussion found here: oats-center/isoblue#60

Essentially, we were running into an issue where the oada.put() call would never return. This often occurred immediately after the ISOBlues came out of suspend. We believe that the issue stems from the server closing the websocket due to lack of "keep alive"s coming from the client, however when the client resumes from suspend, it does not realize this and continues using the now defunct websocket.

We are able to handle this pretty coarsely however some ability to check the status of the socket or force it to reconnect it would be helpful

To replicate the issue (I am missing the type of connection):

this.oada = await connect({ domain: this.domain, token: this.token })
      .catch(e => {
        console.log("Error -> failed to connect to OADA. The error was ", e);
      });

I got this:

Error -> failed to connect to OADA. The error was  TypeError: emitter.once is not a function

A more descriptive error message is needed.