gritzko / swarm Goto Github PK

It's possible to insert text into string inside object as ShareJS?

{
  p: ["arr_obj", idx, "text_field", offset],
  si: "my new text"
}

Since swarm is op-based CRDT, is it possible to implement travel-in-time kind of feature, say let user sees an very old version of the content by clicking on different dates?

http://ppyr.us:8001/demo3/index.html

Steps to reproduce bug:

• select all Availability items in schedules being online
• toggle both schedules to offline
• select first two items on left side to Consistency
• select last two items on right side to Partition tolerance
• toggle right schedule online
• toggle left schedule online

Now data is inconsistent.
The order of last two steps is important.

Hi everyone!
I use swarm.js like caching mechanism in my TeX editor.
So, i would like to use the shifting of history (ctrl-z, ctrl-shift), but i don't understand how to switch a model versions in swarm.js :(

a few quick questions about how to use swarm for a typical webapp. thanks in advance!

is it possible for client replicas to be read only? for example, is there a flag in the host setup to set this.

for the available models, which would work best for an append only data type? should i use vector and only the .append function?

if i use a crdt, what is the recommended way to implement server side validation on a user's changes to the models? or should they just not be trusted and model changes be filtered through server side validation before being saved.

I noticed the Text type is missing from both Html5Client.js and NodeServer.js. Is there a particular reason for that or would you accept a pull request with that change?

Without a require for Text in NodeServer, in my environment (ES6+Babel+Webpack) the server-side code does not add the Text type to Syncable properly:

In server.js:

var Swarm = require('swarm')
Swarm.Text = require('swarm/lib/Text')
console.log('types are now:')
console.dir(_.keys(Swarm.Syncable.types))
// types are now:
// [ 'Syncable', 'Model', 'Set', 'Vector', 'Host' ]

and then later when receiving Text from a client:

error processing message [Error: type unknown: /Text#note] Error: type unknown: /Text#note

However, for some reason, this does work:

var Swarm = require('swarm/lib/NodeServer')
Swarm.Text = require('swarm/lib/Text')
console.log('types are now:')
console.dir(_.keys(Swarm.Syncable.types))
// types are now:
// [ 'Syncable', 'Model', 'Set', 'Vector', 'Host', 'Text' ]

The swarm package.json specifies lib/NodeServer.js as "main", so as I understand the code above should be equivalent. It might have something to do with my use of ES6 and Webpack.

Hi, I want to try to use Svarm.js. You can show an example of using the current version SwarmJs.
I've seen your usage examples, but they are all for version 0.3.
Help with examples!
Thank you)

Hi

I cannot managed to execute some remote method (called one client side but executed on server-side). Would be nice if you can provide a quick example. I looked on the doc, but the info are obsolete (the spec format).

I added my method in the remotes object but he doesn't seems to be parse for a websocket call unlike set.

My goal is to fetch data in DB on server with a client call like mySwarmModel.find(myId);

(1) IFRAME load races; conf/ doesn't sync over postMessage
(2) editor - block deletion from the middle

Timestamps are used in specifiers and that is critical for correctness.
Meanwhile, Web clients may have mad clocks. Need to negotiate that in the host-to-host handshake.
see http://swarmjs.github.io/articles/lamport/

The Spec object has an internal index used for parsing which causes calls to be non-idempotent. For example:

var someSpec = ...
console.log(spec.token('!'))
// prints Object {quant: "!", body: "2QXIl03+A0017r~0", bare: "2QXIl03", ext: "A0017r~0"}
console.log(spec.token('!'))
// prints undefined
someSpec.index = 0
console.log(spec.token('!'))
// prints Object {quant: "!", body: "2QXIl03+A0017r~0", bare: "2QXIl03", ext: "A0017r~0"}

Presumably index should be reset with Spec at the beginning of each parse method?

Hello,

As far as I can see a specific swarm host (client) will receive events only for objects in use, currently I'm building an app prototype using React, I have multiple documents, when I open first document host.get('/Text#1) client's swarm host will receive events only for this specific object, now I want to close this document and open a new one (without refreshing), host.get('/Text#2), now my swarm host will receive and will handle events for both objects (I can see this in browser's console). How can I make swarm host to not handle events for first object anymore?

host.unregister does not solve this.

Thanks

It looks like there is either a typo or unexpected side effect in Spec.Map.covers here...
https://github.com/gritzko/swarm/blob/master/lib/Spec.js#L222-L223
... since the lastIndex on the reQTokExt RegExp is set, but it's actually using the reTokExt RegExp.

Add example with SockJS.

Hi!

I'm trying to set up a swarm cluster on the backend but am having a couple of issues. Reading the code, in Pipe.js it says that a pipe is a channel to a remote swarm host. This works great when the connection is from client -> server, but not so much when doing server -> server. I am able to connect servers to each other but there does not appear to be any replication going on between different instances.

My issue is that we need more then one machine on the back end and since users that share models could be connected to different physical hosts we need a way to ensure that each server stays up to date and that data is replicated correctly.

Is there a way to accomplish this today?

Pulling a copy of the repo, I can't get tests to pass on either stable or unstable node. It looks like swarm-client and swarm-stamp have been removed from the repo (but not the Makefile or various module deps).

Would love to pitch in, but wondering if there's a stable starting point to work from.

Thanks.

I'm still trying to understand how Swarm works for collaborated editing.

Supposed one site created a new object extending a predefined model, how that new object be replicated to another site? I did observe an object with same ID that is already defined on both sites would be piped through, but a new object with new ID didn't seem to do so.

Perhaps I don't understand the flow clearly for the above? A code example will be great.

I am using following model:

ItemList is a vector, which consists of model items. When listen to the "items" changes I get back the reference to object but the object itself is empty.

This case occurs when User A adds object the other User B session listens the event but it's empty or not yet synchronized. How to fix such issue ?

var items = host.get("/ItemList#items");

items.on("init", ....)
items.on(function (spec, val, source) {
   // This gets the correct type of object but, the properties are default values
  var item = items.getObject(val);
});

Is swarm going to support mobile clients natively (via swift/java languages)? If yes, do you need help with adapting code for these platforms?

Awesome project, but it looks like the last activity was 7 months ago. Is this project still under active development?

Hello there,

Thanks for this library. I was wondering if there is a dist/ file to use in a project (standalone file)? Can I use swarm in Rails, sounds like I can but don't know any recommended approach?

Thanks in advance.

Regards,
Gaurav

What are the major differences between https://github.com/dominictarr/scuttlebutt and swarm?

Question. I'm on 0.3.25. In a certain case I have the oplogs as JSON like it is stored by filestorage (_version, _tail with oplogs). I want to hydrate my model with that data, not using Host or storage, how can I do that?

this.spec = new Swarm.Spec('/MyModel#44Gmj+some_id');
this.typeid = this.spec.filter('/#');
let Mr = new MyModel(this.typeid);
// then what?

How do I apply the tail of ops to get a current model?

Through my quick prototype it seemed the merge strategy here is "last write wins", I was wondering is it possible to implement a more refined conflict resolution, ie: let the user to manually merge two copies of data and resolve the conflict.

Thoughts?

I'm looking to give Swarm a spin in a p2p app, but the example in the readme appears to be for client-server sync.

Is p2p sync in a working state at the moment? What would we have to do differently in a p2p sync scenario compared to the client-server example?

Thank you for your work on this amazing library by the way! =)

Probably could use some update?

The current project structure is mainly flat all the files are on the same level. The code could be a bit more organized: /storages, /data-structures, /streams, /utils ?

A few questions:

1. Is swarm production ready?
2. There appears to be no actual production ready Storages (MongoDB, Mysql, Postgres?)
3. How hard is it to make custom a Storage?
4. I don't see a clear strategy for query or collection syncing (Some adds a new item, removes one, etc from a query)

Currently run-tests.sh doesn't work for me. It seems to use nodeunit but I figured out tests are written with qunit. Even after I install qunit and run test suite manually (w/o run-tests.sh) I can't get all tests to pass, because some of them require Swarm.Field which isn't present in codebase.

This is a very enticing solution to simplifying shared state management across and between users and servers... if it can be scalable, durable, and reliable. I'd like to understand what's left to be done to meet the most minimal initial marks on those three points, and if the recent lack of activity is indicating the project has stalled or not. Having some clarity here will help me determine if and hoe to contribute.

As is, everything in the library is global. Does that need to be the case? If not, then we should wrap everything in a self executing anonymous function and expose only the variables that need to be public. For example:

var Swarm = (function () {
    var stuff = {};
    /*
     ...
    */

    return {
        stuff: stuff
    };
}());

I currently have a fullblown backbone app. How would I go about integrating Swarm while keeping most of the app intact?
Could swarm for instance be used (with some modification to my app) to replace Backbone.sync?

BTW: backend is on Node (Loopback which sits on top of regular Express)

ReadMe references https://github.com/gritzko/swarm/blob/master/lib/MongoStorage.js , yet it does not exist :(

BTW an ElasticsearchStorage using official client would be great too ! I may do it by myself if I really understand either the LevelDB one or the FileSystem one

Tried to built project with cmake on ubuntu 15.04 - get next error -

cmake .

header: ev++.h, arguments: libev ==> /usr/include
library: ev, arguments: libev ==> /usr/lib/x86_64-linux-gnu/libev.so
CMake Error at cmake/locate_library.cmake:14 (MESSAGE):
libev development files are required to build.
Call Stack (most recent call first):
CMakeLists.txt:13 (locate_library)

I have instaled libev-dev package, what else I need?
Thanks.

Been playing with various CRDT solutions to implement collaborative text editing which is real-time and offline-ready and I started to wonder since CRDT tends to be memory and disk storage heavy, will there eventually be performance issue after a long editing session (meaning oplog piling up, long syncing and replay time...etc)? Thanks!

Rather than pure websockets, would it be possible to use something like the following as drop-in replacement?

https://github.com/primus/primus (this is ideal)
https://github.com/Automattic/engine.io
https://github.com/Automattic/socket.io
etc

I would like to know if there is some any way to support nested objects.

Well the overall concept is well defined and the idea seems to clear. The problem is that there are many gotchas that are not documented:

• The data structures APIs also how they work.
• What's the limit of implemented clocks ?
• Scaling and limits of the Swarm.
• etc.

https://gist.github.com/sebmarkbage/ae327f2eda03bf165261

Deprecations
React.renderComponent --> React.render
React.renderComponentToString --> React.renderToString
React.renderComponentToStaticMarkup --> React.renderToStaticMarkup

http://stackoverflow.com/questions/26657023/this-key-in-react-js-0-12

After seeing this keynote from @vjeux at React Europe, I think that It could be possible to make swarm works with relay.

For the one who have never heard of it, relay is a module that allow you to declare requirement inside the components and will handle for you the unirectional data flow in a much simpler way than flux.

See the FAQ for more info.

Basically im not sure how rellay and swarm can work together since most of the feature are shared but swarm has the advantage of beeing able to do real time and persistency (cf keynote) and i never heard that rellay aims to do it.

It may be a good idea to look at rellay once it is released in order to see if it would be possible to make something with with { relay + swarmjs }.

can you put a code example in the readme?

It makes it much easier to evaluate a module if you can see a short example of what it looks like to use, along with output. If it looks simple, then you can proceed to read the documentation and/or code.

this is a nice dataflow reactive system.

i was wondering ig this project intends to allow parallelization of a single datanode in th dag to be say 100 dag nodes of the same type. then each one receives a slice of the data on the bus and then converges.

its a logical next step i feel to use the power of crdt because you dont have to conform to map reduce i think anymore because the results can join over time .

bacon.js has a similar approach as swarm (not the same) but has formalized the map reduce without crdt.
https://github.com/baconjs/bacon.js

the two joined would be potent !!

can you have a look and see if you see what i see. i would like to converge the two and put a formal dsl that is serialisable perhaps with a web ide on top

I'm starting to work on an app that should fully work offline, allowing the user to sync it online even after several days being offline, there is nothing decided for the tech stack yet.

On one hand, grounddb is already inside meteor platform, that has built-in realtime syncing through sockets, same code on server and client, and so on.

On the other hand swarm.js seems to be more popular, have CDRT and so on, and it's not dependent of meteor

Both of them are in very early development versions too.

So, what are your thoughts? I'm not searching for a definitive answer, just arguments in favor of one or the other.

I really like your approach of solving one problem at a time (the full stack of isomorphs like Meteor is atm overblown imo). Especially since there are already great complementary libs like React for rendering.

Anyway, on topic: I looked at your demo app and saw that the traffic is sent in plain and I though, why not add protobuf to the mix? The gains are typically smaller messages (in the case of my websocket app, it was about 50% smaller) and faster encoding. If that is something you are interested in, I would work in my spare time on a pull request, which you can check out and see if you like it ;)

Thanks for the effort! 👍

I wrote wrong code:

var id = 32;
new MyModel(id);

where I used number ID instead of string. As result Swarm didn’t sync models.

But I think it is common mistake and we should throw a error on wrong type of ID.

I think Lamport clock should be a distinct npm module because it can be reusable in nodejs world. It's seems unix-way as npm wants to be. Swarm can just use Lamport clock as dependancy. What do you think about this idea?

or see youtube video
https://www.youtube.com/watch?v=fn817WgF1c8