docker-sync on mac does not sync everything about docker-sdk HOT 8 CLOSED

attached a screenshot of:

• left: docker-sync logs -f
• right: editing a file, adding file to index

what I wanted to show with the marks are the connection between "what happend when".

• first line: docker-sync syncs the .swp-file after opening the file with vim
• second line: finished editing file with vim, closing the editor, docker-sync removed .swp because it was deleted
• third line: git add edits hosts .git/index, this file is synced

what I expected to see: docker-sync also syncs whatever happens in .git/objects/... ([before|after|with] the last sync)

from docker-sdk.

Hello @tienbuide,

First of all, thank you for sharing your experience. We really appreciate this.

In general, we do not recommend to sync .git folder using docker-sync. We noticed that a huge vendor folder is quite enough to make docker-sync stuck. Adding git objects in the sync pool would bring better chances to fail.

Also, we do not fully trust docker-sync strategies, as it can mix up files from the container that have been deleted on the host. It is acceptable for source code when git controls untracked files, but unacceptable for .git folder itself.

Here https://github.com/spryker/docker-sdk/blob/1.15.0/generator/src/templates/docker-sync.yml.twig#L13 we have .git folder excluded. So I wonder why Unison syncs .git/index. Probably, that is a bug on our side or docker-sync/Unison side.

We suggest performing git related activities exclusively on the host (for Mac and Windows).

from docker-sdk.

Hello @sprymiker,

thank you for your fast reply.

Here https://github.com/spryker/docker-sdk/blob/master/generator/src/templates/docker-sync.yml.twig#L13 we have .git folder excluded. So I wonder why Unison syncs .git/index. Probably, that is a bug on our side or docker-sync/Unison side.

we are still using 1.14.3[1] and I guess the sync of .git its a bug in 1.14.3. I was wondering why you exclude /.git instead of .git ;)
even if our grumphp would work now... it will fail when we upgrade the sdk to 1.15.0 :/

Also, we do not fully trust docker-sync strategies, as it can mix up files from the container that have been deleted on the host. It is acceptable for source code when git controls untracked files, but unacceptable for .git folder itself.

I do see the point. thanks

do you have any information about SC-1857 and SC-2706[2][3]? Would SC-2706 solve this problem?

[1]https://github.com/spryker/docker-sdk/blob/1.14.3/generator/src/templates/docker-sync.yml.twig#L14
[2]#40
[3]#72

from docker-sdk.

Dear @tienbuide,

According to alternatives for docker-sync:

1. NFS support was hiddenly introduced in 1.15.0. We have not performed the usual testing and releasing cycle as the docker NFS volumes are quite slow. It is like 2x slower in comparison to devvm (Debian+ all FS caching features).

We have build mitigation tools like autoload caching and that helped to speed up page load to very good results: 300-400ms after caches are warmed up. However, console commands that work directly with files are still very slow. In comparison, transfer:generate takes 20-30 seconds vs 2-3 seconds on the local file system or 3-5 seconds on docker volume. composer install takes 10x times more than locally on host and 2x vs devvm.

So I assume it cannot fit your case, as you need static analysis that demands fast file system access.

2. Mutagen is a very good tool (see p.3). However, at the moment we tried (Oct 19) mutagen was not so mature and our solution was based on workarounds. The development has been stopped due to some critical issues and changed priorities.

Some details can be found here: https://spectrum.chat/mutagen/general/trying-to-find-solution-for-2-way-smart-sync~4e48d0e7-13eb-40ee-8698-4449a796024e

3. Docker Desktop for Mac has integrated mutagen within.

Check this video for DockerCon 2020: https://docker.events.cube365.net/docker/dockercon/content/Videos/92BAM7vob5uQ2spZf

So at the moment, we are testing the OOTB solution provided by Docker (finally). Regardless, it is still the Edge version.
Please, refer to https://gist.github.com/djs55/0e8957e1e36431a43cbfd3caaffa9165 where Dave puts updates.

I, personally, use https://desktop.docker.com/mac/edge/46268/Docker.dmg build in my every-day work and the solution works well. The downsides I see are:

• CPU usage: Docker eats 150% of CPU constantly
• Delay when files appear on the host, but automation does not know about the delay (e.g. workaround like this https://github.com/spryker/docker-sdk/blob/1.15.0/bin/command/install/bootstrap.sh#L114).
• Permissions: Docker set your host user id as an owner. But docker/sdk is used to fallback to 1000. So I have small uncommitted changes to allow it to work. The workaround could be using user 1000 on the host.

We are not going to support this solution officially until it is released. At least for now. I expect the API would be changed. The latest change is triggering mutagen sync for a volume using :delegated mode. So I assume it may be changed in next builds.

I hope the information was helpful to you. Please, follow the releases. I believe we will have something reliable in the nearest future as we at Spryker mostly use Macs and we are struggling with the same issue as you.

P.S. I would still advise using the host for static tools. I have noticed that phpstan works twice faster on the host in comparison with docker volumes.

from docker-sdk.

This PR adds support for Docker's experimental feature: https://docs.docker.com/docker-for-mac/mutagen/

#131

It works well for me with no conflicts, no consistent issues.
However, some downsides are:

• Scripting is slower: docker/for-mac#4829
• It takes ~10 seconds to see file changes in container.

from docker-sdk.

This PR uses mutagen beta feature: https://mutagen.io/documentation/orchestration/compose

#136

It works well for me with no conflicts, no consistent issues. No downsides at the moment.

from docker-sdk.

Mutagen support has been released. You can try the new approach. You can find some details in this PR: #136. Documentation will be available later.

Also, you have control over what needs to be synced OOTB. Git folders can be added as well.
Put your .dockersyncignore file in the project root. An example is here: https://github.com/spryker/docker-sdk/blob/master/.dockersyncignore.default.

from docker-sdk.

Closing this as alternative options are available. Feel free to comment here if you want to reopen the issue or create a new one.

from docker-sdk.