Reduce bandwidth-requirement on Bisq app startup about projects HOT 14 CLOSED

the implementation is currently being prepared to be tested in the production network. @sqrrm will upgrade his explorer-seednode to run the new code (that would be v1.3.5 + the changes of this project) so that a few devs can use it productively and see if anything bad shows up. The plan is to do so for one release cycle. If nothing bad shows up, we will proceed with the rather complex upgrade process.

from projects.

I just compiled in some more data

from projects.

Regarding the following items from the description above:

• bin up the data
• create a "special" key for addressing bins
• tie those bins to the application version
• create a new bin on every release holding only new data

Do you intend here to check these binary blobs into the main Bisq repository, or something else? I would really like to avoid adding more binary data to the repository (as we're already doing with all the stuff in p2p/src/main/resources).

from projects.

If checking the blobs in is the intended solution, @freimair, I'd like us to look into doing this properly with Git LFS instead, and at the same time migrating the p2p/src/main/resources files there, too. GitHub has a free tier we might want to start with. I ran some basic numbers and I think we could get away with it, but it depends on how many people are cloning the repository every month (because the pricing is metered on bandwidth used). We could also potentially run our own LFS server, but it would probably be best to go with the free GitHub service until we see we're running it out.

See:

• https://git-lfs.github.com/
• https://help.github.com/en/github/managing-large-files/about-storage-and-bandwidth-usage

/cc @wiz as running our own LFS server would be ops territory. Just FYI at this point.

from projects.

Also, from the introduction:

During startup, Bisq is required to send >4MB of data to seednodes in order to get its initial data.

I'm unfamiliar with this problem, and reading this doesn't help me understand what's really going on. Why would a Bisq node need to send so much data to its seednode(s)? I could understand that it might need to receive quite a bit of data, but I'm clearly missing something. I read through the rest of the description a couple times and I don't think this is ever made clear. ELI5, please :)

from projects.

Why would a Bisq node need to send so much data to its seednode(s)?

• sorry, I took that as common knowledge, because that is how Bisq always worked

• let the ELI5 commence:

  1. on startup, the Bisq app asks the seed node for a "distributed-database update"
  2. In order to not burden the seednode to send all the data (> 12MB), Bisq tells the seednode which objects it already has (ie. sends data to the seednode).
  3. The seed node then only sends the data the bisq app does not have already.

• The trouble comes with success: We now have more than 100k objects in the "distributed database" which makes bisq send 100k "keys" to the seednode (100k * 20byte hash = 2MByte = substantial, and rising).

• And because that is not enough: for redundancy purposes, the Bisq app asks two seednodes for data

• given a "bad" internet connection, Bisq simply fails to start

  • ie. if net upstream is < 35kB/s = 280kb/s (= 4MB/120 second connection timeout)
  • does not seem like a lot, but there are bug reports (labeled critical bug) out there and I encountered it myself while not at home
  • Tor is not at fault: p50 of tor speed is 28Mb/s, however, if you catch a bad circuit, it will.
  • we need more bandwidth as time goes on (because the database grows -> the number of objects grows -> the request size grows)
  • if we succeed with bisq, the required bandwidth will outgrow infrastructure development

Do you intend here to check these binary blobs into the main Bisq repository, or something else? I would really like to avoid adding more binary data to the repository (as we're already doing with all the stuff in p2p/src/main/resources).

yes, I intend to check these binary blobs into the main Bisq repository. It is exactly about the stuff in p2p/src/main/resources which is a snapshot of the "distributed-database" we ship with each release.

• Atm, there is only one blob that gets bigger and bigger. Plus it replaces the old one, so repo size grows with size(t) = size(t-1)+size(newData) per release. (actually, it is several files for different message types, but overall, it is one blob of data)
• after this project is done, a new blob will be added for every release with size(t) = size(newData), the "old" blobs are left untouched and are used as they are (historical data does not change)
• doing it that way is a very minimal change to the current release processes and we can focus on fixing the real issue quickly

I'd like us to look into doing this properly with Git LFS instead

• I totally agree that we have to move away from committing binary data to the repo, but
  • using [insert your favorite storage technology here] does not collide with this project
  • can be done later
  • should be done sooner than later
  • will look into Git LFS as a followup-project

All in all, this project aims for making small steps towards a more reliable service. Rethinking the storage synchronization and locations is a whole other can of worms.

Btw. just checked. We have 110k objects now, at the time of project creation it has been 104k -> approx. +5% in 25 days.

from projects.

The proposal looks well-formed, so I've removed the needs:triage label and added needs:approval per the process.

I am simply not well-informed enough about the details and alternatives to give a meaningful thumbs-up on approving this, but mine is just one voice. Like any other proposal, we should be looking for a broader consensus of interested and informed parties. If you are one of these people (@stejbac?), please provide feedback. The approach here looks pragmatic enough, but it would be good to see other informed opinions.

From a budgeting perspective, it appears to me this is 100% dev team budget, so @ripcurlx, I'll leave it to you to weigh in on.

from projects.

And regarding my comments about Git LFS above, see bisq-network/bisq#4114, which will be treated separately from this project.

from projects.

This is a critical issue that reproduces on slow network connections often now

from projects.

From a budgeting perspective, it appears to me this is 100% dev team budget, so @ripcurlx, I'll leave it to you to weigh in on.

For me this is a critical issue atm for some of our users, but as mentioned the group of people affected by this is growing every day. So from my side it would be a 👍 to start working on this project.

from projects.

@ripcurlx, I'll add the has:budget label, then.

It would be great to see more engagement on approval, but even though we've gotten only a few comments here, it sounds like there's consensus we should go head. I'll add the has:approval label accordingly.

from projects.

@freimair, please move this to In Progress as and when appropriate.

from projects.

the implementation is currently being prepared to be tested in the production network. @sqrrm will upgrade his explorer-seednode to run the new code (that would be v1.3.5 + the changes of this project) so that a few devs can use it productively and see if anything bad shows up. The plan is to do so for one release cycle. If nothing bad shows up, we will proceed with the rather complex upgrade process.

Is there any update on this?

from projects.

the project has been completed by bisq-network/bisq#4586

from projects.