media replication about osm-p2p-db HOT 4 CLOSED

It seems like hypercore will fit into osm-p2p-db the browser better right now.

from osm-p2p-db.

Just back from the field working with a local community monitoring team and have been putting a bit more thought into this.

I don't know if it's necessary to integrate the media storage/replication into osm-p2p-db at all. I think we just need to give each image a URI/UUID and use a standard key/value tag pair on the OSM element.

I think image ids/URIs should include a bucket ID and an image ID. The image ID can be initially derived from its hash, but should remain stable if the image is re-sized: we may want multiple sizes of the image and maintain the ID. If the initial ID is derived from the hash that provides an easy way to check whether we have the original image or a resized/modified version. A bucket ID allows us to keep collections of media in sync online and offline for specific projects. I think in a fully online environment this bucket can sit on a server, but we don't need the server address in the image ID/URI - I think this should be configured at run time when setting up an app with osm-p2p, setting the server or the local sync storage where the image bucket is stored.

I think in many applications storage will quickly become an issue, that means we can't just rely on browser storage. Ideally we will use an external drive as media storage - for size considerations, and also to avoid data loss when a laptop dies (even though the hard drive can sometimes be removed and recovered from a dead laptop, it is often hard to find someone with the skills locally to do that, and laptops die all the time in remote environments). This means that any storage needs to be able to directly access storage and external drives, via electron or a Chrome App wrapper.

We also need to figure out the concepts of selective sync and reduced size sync. For selective sync read-only clients might be all we need: a client will sync all of its media to another peer, but not sync any media from that peer. Since many devices in use will have limited storage as the media bucket grows, need need a way to only sync reduced-size media to some clients. The challenge with this is how to ensure that full-size media eventually reaches where it needs to reach e.g. clients A and B are cellphones, client C is a laptop. Client A has a full-size image, and syncs with B, but only syncs a reduced-size image. Then B syncs with C, but will only have a reduced size image to sync. I think we can work with an eventual consistency scenario where original images are preferred by certain clients, so that in this scenario client C will eventually sync with A, and replace the reduced-size image with the original.

from osm-p2p-db.

I think image ids/URIs should include a bucket ID and an image ID. The image ID can be initially derived from its hash, but should remain stable if the image is re-sized: we may want multiple sizes of the image and maintain the ID.

This is similar to how osm-p2p-db generates keys for the hyperlog. The initial bucket key is a random string and does not change.

We also need to figure out the concepts of selective sync and reduced size sync.

I think this problem is straightforward to solve in the replication layer but difficult to present to users in an intuitive way that maps to what they want to achieve.

Perhaps the replication UI could consist of a list of checkboxes for:

• replicate map
• replicate thumbnail images
• replicate full images

and buttons for adding or excluding specific thumbnail or full images, along with lists of the files that are going to be replicated and a size estimate of the data transfer over the wire for each file and for everything.

This UI would address several use cases:

• phone -> phone replication previews to avoid duplicating image collection of some features: select all thumbnails
• device -> device replication to share several full size images of specific items of interest to a local community: select the list of full images to replicate manually
• phone -> laptop data dump: check all the boxes
• phone -> phone replication of only map data: check only replicate map

In the data model, every modification (resize, crop) should point at the hash of the original source image. The metadata for images should be small, so it can be replicated fully.

from osm-p2p-db.

This issue was moved to digidem/osm-p2p-observations#6

from osm-p2p-db.