fpco / cache-s3 Goto Github PK
View Code? Open in Web Editor NEWLicense: BSD 3-Clause "New" or "Revised" License
cache-s3's People
Contributors
Stargazers
Watchers
cache-s3's Issues
Permission denied error during save on Windows
This problem was introduced in 0.1.8. (0.1.7 was fine) and only affects Windows. I could reproduce this on:
- Windows 2016 (AMI provided by AWS)
- Windows 10 Vagrant box.
It enough to invoke the save command to trigger this error:
> cache-s3 --prefix=cache-s3-test save --relative-path clcache
[cache-s3][Info ][Tue, 16 Apr 2019 12:58:20 UTC]: Caching: clcache
[cache-s3][Info ][Tue, 16 Apr 2019 12:58:44 UTC]: <master.cache> - No previously stored cache was found.
[cache-s3][Info ][Tue, 16 Apr 2019 12:58:44 UTC]: <master.cache> - Data change detected, caching 308.3 MiB with sha256: bNlkj0BqEBs8JBa8flvruMvhjpIgkE6LdD5q9F49quM=
cache-s3: C:\Users\vagrant\AppData\Local\Temp\cache-s3.tar7484-0.gzip: DeleteFile "\\\\?\\C:\\Users\\vagrant\\AppData\\Local\\Temp\\cache-s3.tar7484-0.gzip": permission denied (Access is denied.)
The error message displays \\?\ characters in the PATH, this suggests some Win32 path compatibility quirk.
Looks very similar to "Case 5" documented here: https://support.microsoft.com/en-us/help/320081/you-cannot-delete-a-file-or-a-folder-on-an-ntfs-file-system-volume
Caching with save stack work doesn't include ~/.stack
#!/bin/bash -eo pipefail
~/.local/bin/cache-s3 --bucket frontrow-ops --prefix ci-cache/build-backend save stack work
Stack has not been tested with GHC versions above 8.6, and using 8.8.2, this may fail
Stack has not been tested with Cabal versions above 2.4, but version 3.0.1.0 was found, this may fail
Stack has not been tested with GHC versions above 8.6, and using 8.8.2, this may fail
Stack has not been tested with Cabal versions above 2.4, but version 3.0.1.0 was found, this may fail
[cache-s3][Info ][Thu, 27 Feb 2020 15:36:31 UTC]: Caching: /root/megarepo/backend/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 15:36:31 UTC]: Caching: /root/megarepo/backend/core/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 15:36:31 UTC]: Caching: /root/megarepo/backend/data-gen/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 15:36:31 UTC]: Caching: /root/megarepo/backend/entities/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 15:36:31 UTC]: Caching: /root/megarepo/backend/fancy-api/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 15:36:31 UTC]: Caching: /root/megarepo/backend/frontrow-app/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 15:36:31 UTC]: Caching: /root/megarepo/backend/jobs/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 15:36:31 UTC]: Caching: /root/megarepo/backend/provisioning/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 15:36:31 UTC]: Caching: /root/megarepo/backend/roster-management/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 15:36:31 UTC]: Caching: /root/megarepo/backend/text-assets/.stack-work
Notice ~/.stack is not included. This makes the caching incomplete and a subsequent rebuild (with no changes) goes on to compile everything again. I think ~/.stack should always be included in a stack-aware caching.
Handle problem with continuously increasing cache
It is possible for individual files that are being cache to become irrelevant, eg. new modules added old ones deleted, would result in obsolete files that could be cleaned up in order to reduce cache size as well as the bandwidth to/from AWS.
It would be too cumbersome and complicated to keep track of individual files, but considering that we are dealing with cache, it is acceptable to loose the whole thing every so often. There are two possible solutions I have for this problem. Both are not mutually exclusive and could be implemented side by side:
- First one is lifetime restriction. It'd be possible to set a period of time on the cache file itself (which would NOT be updated during cache replacement). During that period cache file would be considered valid, but once it reached its lifetime expectancy it would be removed instead of restored.
- Second one is addition of
--max-sizeflag that would preventcache-s3to restore or save cache that hits that limit.
Both solutions would effectively clear out cache on certain conditions, thus alleviating the problem of continuously increasing cache.
Only some files are restored with restore stack
I have successfully saved a cache file using cache-s3 save stack. Now I'm restoring it, and only some files are restored:
ls -l /home/xxxxxxxxxxxxxx/.stack/
total 8
drwxrwxr-x 3 501 501 4096 Jan 29 22:30 precompiled
drwxrwxr-x 3 501 501 4096 Jan 29 22:52 programs
I have added logging of each restored file into restoreFile' like this:
restoreFile' fi = do
liftIO $ print fi
-- original functionThis is the end of the output of cache-s3 restore stack:
FileInfo {filePath = "/home/xxxxxxxxxxxxxx/.stack/precompiled", fileUserId = 501, fileUserName = "xxxxxxxxxxxxxx", fileGroupId = 501, fileGroupName = "xxxxxxxxxxxxxx", fileMode = 509, fileSize = 0, fileType = FTDirectory, fileModTime = 1517265003}
FileInfo {filePath = "/home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743", fileUserId = 501, fileUserName = "xxxxxxxxxxxxxx", fileGroupId = 501, fileGroupName = "xxxxxxxxxxxxxx", fileMode = 509, fileSize = 0, fileType = FTDirectory, fileModTime = 1517265003}
FileInfo {filePath = "/home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2", fileUserId = 501, fileUserName = "xxxxxxxxxxxxxx", fileGroupId = 501, fileGroupName = "xxxxxxxxxxxxxx", fileMode = 509, fileSize = 0, fileType = FTDirectory, fileModTime = 1517265003}
FileInfo {filePath = "/home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2/2.0.1.0", fileUserId = 501, fileUserName = "xxxxxxxxxxxxxx", fileGroupId = 501, fileGroupName = "xxxxxxxxxxxxxx", fileMode = 509, fileSize = 0, fileType = FTDirectory, fileModTime = 1517285004}
FileInfo {filePath = "/home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2/2.0.1.0/cereal-0.5.4.0@sha256:8695c775923e3a185c20e574e4b8202752f7274d94df9b291340d976ea09a742,2767", fileUserId = 501, fileUserName = "xxxxxxxxxxxxxx", fileGroupId = 501, fileGroupName = "xxxxxxxxxxxxxx", fileMode = 509, fileSize = 0, fileType = FTDirectory, fileModTime = 1517265098}
FileInfo {filePath = "/home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2/2.0.1.0/cereal-0.5.4.0@sha256:8695c775923e3a185c20e574e4b8202752f7274d94df9b291340d976ea09a742,2767/XvBt2WClc9gS1tpA1pZNS9TmIn3m5Pipp2P08CAFPcA=", fileUserId = 501, fileUserName = "xxxxxxxxxxxxxx", fileGroupId = 501, fileGroupName = "xxxxxxxxxxxxxx", fileMode = 436, fileSize = 527, fileType = FTNormal, fileModTime = 1517283950}
[cache-s3][Info ][Wed, 31 Jan 2018 05:39:33 UTC]: Progress: 60%, speed: 7.2 MiB/s
[cache-s3][Info ][Wed, 31 Jan 2018 05:39:43 UTC]: Progress: 70%, speed: 7.4 MiB/s
[cache-s3][Info ][Wed, 31 Jan 2018 05:39:53 UTC]: Progress: 80%, speed: 7.0 MiB/s
[cache-s3][Info ][Wed, 31 Jan 2018 05:40:04 UTC]: Progress: 90%, speed: 6.9 MiB/s
[cache-s3][Info ][Wed, 31 Jan 2018 05:40:17 UTC]: Progress: 100%, speed: 5.6 MiB/s
[cache-s3][Info ][Wed, 31 Jan 2018 05:40:17 UTC]: <cache-s3/xxxxxxxxxxxxxxxxxxxxxxxx/master.lts-10.3.stack.cache> - Successfully restored previous cache
However, I have downloaded the cache file manually from S3 and ran:
tar -tvf master.lts-10.3.stack.cache
(...)
drwxrwxr-x 0 xxxxxxxxxxxxxx xxxxxxxxxxxxxx 0 30 Jan 09:30 /home/xxxxxxxxxxxxxx/.stack/precompiled
drwxrwxr-x 0 xxxxxxxxxxxxxx xxxxxxxxxxxxxx 0 30 Jan 09:30 /home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743
drwxrwxr-x 0 xxxxxxxxxxxxxx xxxxxxxxxxxxxx 0 30 Jan 09:30 /home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2
drwxrwxr-x 0 xxxxxxxxxxxxxx xxxxxxxxxxxxxx 0 30 Jan 15:03 /home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2/2.0.1.0
drwxrwxr-x 0 xxxxxxxxxxxxxx xxxxxxxxxxxxxx 0 30 Jan 09:31 /home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2/2.0.1.0/cereal-0.5.4.0@sha256:8695c775923e3a185c20e574e4b8202752f7274d94df9b291340d976ea09a742,2767
-rw-rw-r-- 0 xxxxxxxxxxxxxx xxxxxxxxxxxxxx 527 30 Jan 14:45 /home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2/2.0.1.0/cereal-0.5.4.0@sha256:8695c775923e3a185c20e574e4b8202752f7274d94df9b291340d976ea09a742,2767/XvBt2WClc9gS1tpA1pZNS9TmIn3m5Pipp2P08CAFPcA=
drwxrwxr-x 0 xxxxxxxxxxxxxx xxxxxxxxxxxxxx 0 30 Jan 14:53 /home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2/2.0.1.0/free-4.12.4@sha256:51279b76403ea6e900a02fbf2ab9c90eff7ba17c427f1d4caa98633c693b26db,3243
-rw-rw-r-- 0 xxxxxxxxxxxxxx xxxxxxxxxxxxxx 515 30 Jan 14:53 /home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2/2.0.1.0/free-4.12.4@sha256:51279b76403ea6e900a02fbf2ab9c90eff7ba17c427f1d4caa98633c693b26db,3243/HpTjdNCHTBz1UBDIXua6crYg7ApZ5UDtJpbZMz2JgQQ=
drwxrwxr-x 0 xxxxxxxxxxxxxx xxxxxxxxxxxxxx 0 30 Jan 14:55 /home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2/2.0.1.0/aeson-casing-0.1.0.5@sha256:dfa52ef1ceaed3e22f5c157703359828aa6d4d295761480cf8a21fa7d3af153b,1466
(...)
There is a lot of files in /home/xxxxxxxxxxxxxx/.stack/precompiled/x86_64-linux-dkda49f7ca9b244180d3cfb1987cbc9743/ghc-8.2.2/2.0.1.0, but only one is extracted by cache-s3.
As before, I have redacted some information but kept the number of characters the same - I'm seeing paths dangerously close to 256 characters so it might be relevant.
when there is an HTTP error with AWS, display the Status Code
ATM, an HTTP error with AWS looks like:
[cache-s3][Error][... UTC]: <cache-s3/stack-cache/foobar/branch.linux.lts-12.21.stack.cache> - Critical HTTPException
While there is --verbosity debug, that is a lot of info and leaks credentials.
The HTTP Status Code can tell us a lot about the error, so it'd be nice to get the HTTP Status Code in the existing error message (without the debug/credentials or other parts of the full request).
Bug in removing duplicate paths
When we remove duplicate paths here we're not taking into account a situation where we might want to save two folders like .ghc and .ghcjs. The sort functions will sort this as [".ghc", ".ghcjs"] (so it doesn't matter how we order it) and will then conclude that .ghcjs is a subpath of .ghc and will therefore not cache it. I'm guessing there are other valid examples but this is one that I just ran into.
Add ghc version to the cache namespace
It is possible to get the desired ghc version from stack. For example
$ stack query compiler actual
ghc-8.2.2Adding ghc version to the namespace would help with isolation in presence of custom snapshots.
Retry after failing data transfer
Hi @lehins!
We're using cache-s3 in a hybrid infra: we have workers in the cloud and on-premises as well. cache-s3 is working great, but sometimes uploading the .cache file to S3 fails:
# Saving the compilation cache...
$ cache-s3 --prefix=macos save --relative-path ./ccache
[cache-s3][Info ][Mon, 2 Sep 2019 07:02:04 UTC]: Caching: ./ccache
[cache-s3][Info ][Mon, 2 Sep 2019 07:02:17 UTC]: <cache-s3/macos/master.cache> - No previously stored cache was found.
[cache-s3][Info ][Mon, 2 Sep 2019 07:02:17 UTC]: <cache-s3/macos/master.cache> - Data change detected, caching 65.6 MiB with sha256: sKL7Z16jyBiNDy+nZl+NSxyLVJtwM8aR0wvJYB+ZRKE=
[cache-s3][Error][Mon, 2 Sep 2019 07:03:29 UTC]: <cache-s3/macos/master.cache> - Critical HTTPException
The resulting cache database is only 65 MB, but it took more than an hour to finish the compilation (because we started from an empty cache). As the HTTP error fails our build, we have to start again the whole build.
To solve this issue I propose to implement a retry mechanism for the data transfer. I was looking at the retry package which could try to recover in case of an HTTPException.
I'm not sure why this exception occurs, so probably I'd start by fixing #18
What do you think? If you agree with this direction I'd be happy to submit a PR.
TarCreationError <encodeOctal>: Tar value overflow
Ran cache-s3 save stack on a local machine, got the following:
[cache-s3][Info ][Mon, 29 Jan 2018 04:31:36 UTC]: Caching: <snip>/stack
cache-s3: TarCreationError "<encodeOctal>: Tar value overflow: (\"6423500\",Just 864)"
OS: macOS 10.13.3 (17D47)
cache-s3 version: unpacked from Github release cache-s3-v0.1.1-osx-x86_64.tar.gz
There might well be a lot of cruft in .stack because this is a development machine but I'm not sure how to figure out which file(s) are tripping cache-s3.
Puzzling text in usage help
Running with no arguments or invalid arguments produces:
Missing: (-b|--bucket S3_BUCKET) (COMMAND | COMMAND | COMMAND)
Usage: cache-s3 (-b|--bucket S3_BUCKET) [-r|--region AWS_REGION] [--prefix ARG]
[--git-dir GIT_DIR] [--git-branch GIT_BRANCH] [--suffix ARG]
[-v|--verbosity ARG] [-c|--concise] [--max-size ARG] [--version]
(COMMAND | COMMAND | COMMAND) [-h|--help]
I can guess where (COMMAND | COMMAND | COMMAND) part is coming from, but it's confusing to see it in the help text.
Feature request: support unpacking a cache generated with different UID/GID
When I try to unpack a cache created by an user with different UID or GID, I get this error:
cache-s3: /home/user/.stack: setOwnerAndGroup: permission denied (Operation not permitted)
This happens because tar-conduit tries to restore the original owner of the file when unpacking.
It's reasonable to ask why someone would need this. The use case is: we download caches from the CI system onto our development machines to speed up compilation after a branch change. The environment is compatible enough that it works, but sometimes we have different user or group IDs.
What would be the best way to implement this? I see several options:
- Copy-paste
restoreFilefromtar-conduit, and remove changing permissions. Not elegant. - Modify the FileInfo passed to
restoreFileand set UID/GID to the current user. This would work, but it seems like a hack. - Extend
tar-conduit, adding an option to skip setting UID/GID.
Build failures after bump LTS / static build
I made an attempt to create a statically linked build for cache-s3, but it was a bit of a rabbit hole. I'm posting the details here so it saves someone else the time and to also get feedback on how to get through this.
I started with the fpco/alpine-haskell-stack:gmp-ghc-8.8.3 docker image, then updated stack.yaml.
First, the resolver, 15.10 is aligned with the GHC in the alpine docker image:
resolver: lts-15.10
I also commented out the extra-deps to see what Stack would say about those.
Then I built cache-s3 with: stack build --system-ghc --ghc-options="-optl-static -optl-pthread".
Stack pointed out I should add the following extra-deps, so I did:
extra-deps:
- amazonka-s3-streaming-1.1.0.0@sha256:96ee28b13de78a991aa3aa7f91107c8588d7fbe9a71a76caf2cf963cffacdcf4,2651
- git-0.3.0@sha256:dc070840ab24792c9664b9e3e69c9d55d30fc36fee41049c548bb0e8ec83c919,3448
- lz4-conduit-0.3@sha256:1e962c6038ba2568a3d6b9a53cfdcd64ed326876012c44f1e65ed34e2aa9677d,1657
There were two build errors that resulted:
I'm not sure if there's a new release of the git library, or if that lib needs to be updated as well. I'm guessing the latter. In regards to LZ4, I'm not sure if it would be easier to switch to @nh2's lz4-frame-conduit or fix the upstream.
@lehins I would be curious to know what you would recommend, and/or what you would like to see.
Cannot access AWS credentials via instance metadata
cache-s3 version 0.1.6 cannot access the AWS credentials if running on an EC2 instance with credentials provided via the metadata endpoint.
Here is a (redacted) log:
# aws sts get-caller-identity
{
"Account": "xxxx",
"UserId": "XXXX",
"Arn": "arn:aws:sts::xxxx:assumed-role/xxxx/i-xxxx"
}
# curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
iam/
instance-action
instance-id
instance-type
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
reservation-id
security-groups
services/
# curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
xxxx-role
# cache-s3 --prefix xxxx restore stack
cache-s3: MissingFileError "/var/lib/xxxx/.aws/credentials"
createSymbolicLink: already exists (File exists) - this time with docker in the path
With cache-s3 v0.1.2, this is still happening:
cache-s3: /home/xxxxxxxx/xxxxxxxxxxx/.stack-work/docker/_home/.ssh: createSymbolicLink: already exists (File exists)
The build is using Stack's Docker support.
Refactor with RIO
Current implementation uses process, monad-logger and manual passing of the environment. This setup can now be drastically simplified/ by switching to rio:
- Switch to RIO monad - avoid passing config and other arguments
- get rid of
monad-logger(andfast-logger) in favor ofLogFunc+Displayfromrio - drop
safe-exceptiondependency (ensureMonadCatchis not being used) - get rid of
processin favor ofRIO.Process
The default upload/download method is much slower than S3 cp
Hello and thank you very much for sharing this project: it has been very useful for my team to provide a generic compilation cache in our infrastructure which is AWS based.
We have noted that the uploads/downloads of the caches, done by cache-s3, are much slower than what we would get with a simple aws s3 cp, often by almost a factor of 8/10.
Consider the following log, for instance:
[cache-s3][Info ][Wed, 27 Feb 2019 15:32:50 UTC]: <cache-s3/#redacted#/master.cache> - Data change detected, caching 1.2 GiB with sha256: 3vxkxF5XYO9vJLo72s8NyLHNJRn0nOk/iHg0TjgZbZo=
[cache-s3][Info ][Wed, 27 Feb 2019 15:32:55 UTC]: Progress: 10%, speed: 25.4 MiB/s
[cache-s3][Info ][Wed, 27 Feb 2019 15:32:59 UTC]: Progress: 20%, speed: 26.3 MiB/s
[cache-s3][Info ][Wed, 27 Feb 2019 15:33:04 UTC]: Progress: 30%, speed: 26.1 MiB/s
[cache-s3][Info ][Wed, 27 Feb 2019 15:33:08 UTC]: Progress: 40%, speed: 26.3 MiB/s
[cache-s3][Info ][Wed, 27 Feb 2019 15:33:13 UTC]: Progress: 50%, speed: 26.2 MiB/s
[cache-s3][Info ][Wed, 27 Feb 2019 15:33:21 UTC]: Progress: 60%, speed: 14.0 MiB/s
[cache-s3][Info ][Wed, 27 Feb 2019 15:33:26 UTC]: Progress: 70%, speed: 25.9 MiB/s
[cache-s3][Info ][Wed, 27 Feb 2019 15:33:30 UTC]: Progress: 80%, speed: 26.2 MiB/s
[cache-s3][Info ][Wed, 27 Feb 2019 15:33:35 UTC]: Progress: 90%, speed: 25.6 MiB/s
[cache-s3][Info ][Wed, 27 Feb 2019 15:33:39 UTC]: Progress: 100%, speed: 27.2 MiB/s
[cache-s3][Info ][Wed, 27 Feb 2019 15:33:43 UTC]: <cache-s3/#redacted#/master.cache> - Finished uploading. Files are cached on S3.
As you can see from the timestamp it took 48 seconds to upload ~1200 MB -> ~25MB/s, consistent with the speed of the partial progress.
I've also compiled a list of logs from other builds and the upload and download speeds are very consistent, even across wildly different caches sizes.
However if I log into the machine and do directly a S3 cp, of the same cache file, to the same bucket it's done in ~6 seconds: ~200MB/s!
I suspect that this might be a consequence of our infrastructure fully residing in AWS and thus the method that cache-s3 natively uses to upload and download objects for S3 is not optimized when the speed can be very high.
A humble suggestion
Perhaps it could be added to cache-s3 an optional flag to simply use S3 cp when required?
I'd love to do it myself but I really know nothing of Haskell.
Thank you!
Save stack (work) says something changed when it didn't
Hi again :)
I have a CI job where I'm testing cache-s3, I run the build twice in a row and expect the second build to not save a new cache since nothing should have changed. However, it still says there were changes and uploads anyway (adding 4m33s).
Restore cache output
#!/bin/bash -eo pipefail
# Restore ~/.stack
~/.local/bin/cache-s3 \
--bucket frontrow-ops \
--prefix ci-cache/build-backend \
restore stack --base-branch master
# Restore */.stack-work
~/.local/bin/cache-s3 \
--bucket frontrow-ops \
--prefix ci-cache/build-backend \
restore stack work --base-branch master
[cache-s3][Info ][Thu, 27 Feb 2020 22:39:35 UTC]: <cache-s3/ci-cache/build-backend/pb/cache-s3../snapshot.yaml.stack.cache> - Restoring cache from Thu, 27 Feb 2020 22:05:47 UTC with total size: 954.5 MiB
[cache-s3][Info ][Thu, 27 Feb 2020 22:39:38 UTC]: Progress: 10%, speed: 30.8 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:39:46 UTC]: Progress: 20%, speed: 12.2 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:39:54 UTC]: Progress: 30%, speed: 12.9 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:02 UTC]: Progress: 40%, speed: 11.1 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:11 UTC]: Progress: 50%, speed: 11.4 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:13 UTC]: Progress: 60%, speed: 40.6 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:19 UTC]: Progress: 70%, speed: 16.0 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:26 UTC]: Progress: 80%, speed: 13.2 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:35 UTC]: Progress: 90%, speed: 11.0 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:42 UTC]: Progress: 100%, speed: 13.7 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:42 UTC]: <cache-s3/ci-cache/build-backend/pb/cache-s3../snapshot.yaml.stack.cache> - Successfully restored previous cache with hash: 8utS22ULekcFD1Zitvp1S8hXpJZ9n+E0ykvCZ56EdiY=
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:42 UTC]: <cache-s3/ci-cache/build-backend/pb/cache-s3../snapshot.yaml.stack-work.cache> - Restoring cache from Thu, 27 Feb 2020 22:07:46 UTC with total size: 494.0 MiB
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:46 UTC]: Progress: 10%, speed: 14.6 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:49 UTC]: Progress: 20%, speed: 13.8 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:53 UTC]: Progress: 30%, speed: 14.3 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:40:57 UTC]: Progress: 40%, speed: 12.5 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:41:02 UTC]: Progress: 50%, speed: 10.6 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:41:07 UTC]: Progress: 60%, speed: 9.9 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:41:11 UTC]: Progress: 70%, speed: 11.4 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:41:15 UTC]: Progress: 80%, speed: 13.8 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:41:19 UTC]: Progress: 90%, speed: 11.8 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:41:23 UTC]: Progress: 100%, speed: 11.5 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:41:23 UTC]: <cache-s3/ci-cache/build-backend/pb/cache-s3../snapshot.yaml.stack-work.cache> - Successfully restored previous cache with hash: LpHpIK1K8NOKmGaHTPPW2w3Wq3y9kMaKqz6QUYqXFpc=
CircleCI received exit code 0
Build step, showing no compilation happens -- i.e. nothing changed
#!/bin/bash -eo pipefail
# NB: We need to name all our packages explicitly to avoid packages
# which are dependencies to other packages from being built by this
# command.
packages=$(
find . -type f \
-not -wholename '*/.stack-work/*' \
-name package.yaml \
-printf '%h\n' | sed 's|.*/||'
)
stack build --no-terminal \
--only-dependencies $packages
Stack has not been tested with GHC versions above 8.6, and using 8.8.2, this may fail
Stack has not been tested with Cabal versions above 2.4, but version 3.0.1.0 was found, this may fail
CircleCI received exit code 0
#!/bin/bash -eo pipefail
mkdir -p /tmp/workspace/dist/backend-binaries
# Build non-master backend with --fast to speed things up
stack-build() {
if [ "$CIRCLE_BRANCH" = master ]; then
stack build "$@"
else
stack build --fast "$@"
fi
}
stack-build --no-terminal \
--ghc-options '-j +RTS -A64m -n2m -RTS' \
--pedantic \
--copy-bins \
--local-bin-path \
/tmp/workspace/dist/backend-binaries | ts
Stack has not been tested with GHC versions above 8.6, and using 8.8.2, this may fail
Stack has not been tested with Cabal versions above 2.4, but version 3.0.1.0 was found, this may fail
Copying from /root/megarepo/backend/.stack-work/install/x86_64-linux/6eca7dcc4c00c5efff615b45d8027edbda3e13cd0db69a175f918a0c4613d613/8.8.2/bin/build-standard-fixtures to /tmp/workspace/dist/backend-binaries/build-standard-fixtures
Copying from /root/megarepo/backend/.stack-work/install/x86_64-linux/6eca7dcc4c00c5efff615b45d8027edbda3e13cd0db69a175f918a0c4613d613/8.8.2/bin/data-gen to /tmp/workspace/dist/backend-binaries/data-gen
Copying from /root/megarepo/backend/.stack-work/install/x86_64-linux/6eca7dcc4c00c5efff615b45d8027edbda3e13cd0db69a175f918a0c4613d613/8.8.2/bin/fancy-api to /tmp/workspace/dist/backend-binaries/fancy-api
Copying from /root/megarepo/backend/.stack-work/install/x86_64-linux/6eca7dcc4c00c5efff615b45d8027edbda3e13cd0db69a175f918a0c4613d613/8.8.2/bin/fancy-api-mkroutes to /tmp/workspace/dist/backend-binaries/fancy-api-mkroutes
Copying from /root/megarepo/backend/.stack-work/install/x86_64-linux/6eca7dcc4c00c5efff615b45d8027edbda3e13cd0db69a175f918a0c4613d613/8.8.2/bin/fancy-api-routes to /tmp/workspace/dist/backend-binaries/fancy-api-routes
Copying from /root/megarepo/backend/.stack-work/install/x86_64-linux/6eca7dcc4c00c5efff615b45d8027edbda3e13cd0db69a175f918a0c4613d613/8.8.2/bin/jobs to /tmp/workspace/dist/backend-binaries/jobs
Copying from /root/megarepo/backend/.stack-work/install/x86_64-linux/6eca7dcc4c00c5efff615b45d8027edbda3e13cd0db69a175f918a0c4613d613/8.8.2/bin/provision to /tmp/workspace/dist/backend-binaries/provision
Copying from /root/megarepo/backend/.stack-work/install/x86_64-linux/6eca7dcc4c00c5efff615b45d8027edbda3e13cd0db69a175f918a0c4613d613/8.8.2/bin/sync-school-class-link to /tmp/workspace/dist/backend-binaries/sync-school-class-link
Copying from /root/megarepo/backend/.stack-work/install/x86_64-linux/6eca7dcc4c00c5efff615b45d8027edbda3e13cd0db69a175f918a0c4613d613/8.8.2/bin/text-assets to /tmp/workspace/dist/backend-binaries/text-assets
Copying from /root/megarepo/backend/.stack-work/install/x86_64-linux/6eca7dcc4c00c5efff615b45d8027edbda3e13cd0db69a175f918a0c4613d613/8.8.2/bin/verify to /tmp/workspace/dist/backend-binaries/verify
Copied executables to /tmp/workspace/dist/backend-binaries:
- build-standard-fixtures
- data-gen
- fancy-api
- fancy-api-mkroutes
- fancy-api-routes
- jobs
- provision
- sync-school-class-link
- text-assets
- verify
Warning: Installation path /tmp/workspace/dist/backend-binaries
not found on the PATH environment variable.
CircleCI received exit code 0
Save cache step, which shouldn't do any uploading, but does
#!/bin/bash -eo pipefail
# Cache ~/.stack
~/.local/bin/cache-s3 \
--bucket frontrow-ops \
--prefix ci-cache/build-backend \
save stack
# Cache ~/.stack-work
~/.local/bin/cache-s3 \
--bucket frontrow-ops \
--prefix ci-cache/build-backend \
save stack work
Stack has not been tested with GHC versions above 8.6, and using 8.8.2, this may fail
Stack has not been tested with Cabal versions above 2.4, but version 3.0.1.0 was found, this may fail
Stack has not been tested with GHC versions above 8.6, and using 8.8.2, this may fail
Stack has not been tested with Cabal versions above 2.4, but version 3.0.1.0 was found, this may fail
Stack has not been tested with GHC versions above 8.6, and using 8.8.2, this may fail
Stack has not been tested with Cabal versions above 2.4, but version 3.0.1.0 was found, this may fail
Stack has not been tested with GHC versions above 8.6, and using 8.8.2, this may fail
Stack has not been tested with Cabal versions above 2.4, but version 3.0.1.0 was found, this may fail
[cache-s3][Info ][Thu, 27 Feb 2020 22:41:29 UTC]: Caching: /root/.stack
[cache-s3][Info ][Thu, 27 Feb 2020 22:43:54 UTC]: <cache-s3/ci-cache/build-backend/pb/cache-s3../snapshot.yaml.stack.cache> - Data change detected, caching 954.3 MiB with sha256: xbj5BhKf446PcC0pj4pTDUvMkC36ZBepBv9TTMk3vxU=
[cache-s3][Info ][Thu, 27 Feb 2020 22:43:55 UTC]: Progress: 10%, speed: 161.7 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:43:57 UTC]: Progress: 20%, speed: 46.7 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:43:59 UTC]: Progress: 30%, speed: 48.8 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:01 UTC]: Progress: 40%, speed: 46.9 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:03 UTC]: Progress: 50%, speed: 44.6 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:05 UTC]: Progress: 60%, speed: 47.3 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:07 UTC]: Progress: 70%, speed: 45.4 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:09 UTC]: Progress: 80%, speed: 43.8 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:11 UTC]: Progress: 90%, speed: 46.9 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:13 UTC]: Progress: 100%, speed: 49.0 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:14 UTC]: <cache-s3/ci-cache/build-backend/pb/cache-s3../snapshot.yaml.stack.cache> - Average speed: 47.7 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:14 UTC]: <cache-s3/ci-cache/build-backend/pb/cache-s3../snapshot.yaml.stack.cache> - Finished uploading. Files are cached on S3.
Stack has not been tested with GHC versions above 8.6, and using 8.8.2, this may fail
Stack has not been tested with Cabal versions above 2.4, but version 3.0.1.0 was found, this may fail
Stack has not been tested with GHC versions above 8.6, and using 8.8.2, this may fail
Stack has not been tested with Cabal versions above 2.4, but version 3.0.1.0 was found, this may fail
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:15 UTC]: Caching: /root/megarepo/backend/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:15 UTC]: Caching: /root/megarepo/backend/core/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:15 UTC]: Caching: /root/megarepo/backend/data-gen/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:15 UTC]: Caching: /root/megarepo/backend/entities/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:15 UTC]: Caching: /root/megarepo/backend/fancy-api/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:15 UTC]: Caching: /root/megarepo/backend/frontrow-app/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:15 UTC]: Caching: /root/megarepo/backend/jobs/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:15 UTC]: Caching: /root/megarepo/backend/provisioning/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:15 UTC]: Caching: /root/megarepo/backend/roster-management/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 22:44:15 UTC]: Caching: /root/megarepo/backend/text-assets/.stack-work
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:50 UTC]: <cache-s3/ci-cache/build-backend/pb/cache-s3../snapshot.yaml.stack-work.cache> - Data change detected, caching 494.0 MiB with sha256: BRj4kPTgYz2CBSWoZ+IazcWLVQARCJkpTS+VYNKahCY=
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:50 UTC]: Progress: 10%, speed: 124.9 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:51 UTC]: Progress: 20%, speed: 188.2 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:52 UTC]: Progress: 30%, speed: 28.2 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:53 UTC]: Progress: 40%, speed: 191.5 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:55 UTC]: Progress: 50%, speed: 28.1 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:55 UTC]: Progress: 60%, speed: 190.4 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:57 UTC]: Progress: 70%, speed: 26.6 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:57 UTC]: Progress: 80%, speed: 178.9 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:59 UTC]: Progress: 90%, speed: 22.1 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:45:59 UTC]: Progress: 100%, speed: 174.1 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:46:01 UTC]: <cache-s3/ci-cache/build-backend/pb/cache-s3../snapshot.yaml.stack-work.cache> - Average speed: 44.8 MiB/s
[cache-s3][Info ][Thu, 27 Feb 2020 22:46:01 UTC]: <cache-s3/ci-cache/build-backend/pb/cache-s3../snapshot.yaml.stack-work.cache> - Finished uploading. Files are cached on S3.
Am I don't something incorrectly again?
--stack-yaml doesn't seem to be working as intended
If stack yaml is not in the current directory
--stack-yamlflag will causestacksubprocess not able to find it and thus initialize new environment.
When using --stack-yaml, we should be able to run cache-s3 from some path where stack.yaml is not. When using --stack-yaml, cache-s3 should be able to find the right stack/ghc and project correctly.
From slack. I can lookup more info in CI build logs if that would help, but I've lost the URL so would first need to find the jobs where this happened.
cache-s3 needs packages section in stack.yaml file for it to work
The packages section in stack.yaml file is optional and if it isn't present, the following is assumed by default:
packages:
- .
Reference: https://docs.haskellstack.org/en/stable/yaml_configuration/#packages-and-extra-deps
This caused a error when using cache-s3 in the Azure CI: https://dev.azure.com/commercialhaskell/stack/_build/results?buildId=329
I fixed it in this commit: commercialhaskell/stack@d322e1a
Assuming the same semantics for stack.yaml as Stack would be nice!
setFileTimes: does not exist (No such file or directory)
Running cache-s3 restore stack --base-branch=master with a previously saved cache, I get the following:
[cache-s3][Info ][Mon, 29 Jan 2018 23:45:46 UTC]: <cache-s3/xxxxxxxxxxxxx/master.lts-10.3.stack.cache> - Restoring cache with total size: 724.7 MiB
[23:45:46]cache-s3: /home/xxxxxxxxxxx/.stack/programs/x86_64-linux/ghc-8.2.2/bin/ghc: setFileTimes: does not exist (No such file or directory)
This was on a Linux build agent; I've downloaded the archive and extracted it manually on my macOS machine. Here's the contents of that directory:
ls home/svc_buildagent/.stack/programs/x86_64-linux/ghc-8.2.2/bin/ -l
total 32
lrwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 9 Jan 30 09:53 ghc -> ghc-8.2.2
-rwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 458 Jan 30 09:53 ghc-8.2.2
lrwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 13 Jan 30 09:53 ghc-pkg -> ghc-pkg-8.2.2
-rwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 490 Jan 30 09:53 ghc-pkg-8.2.2
lrwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 10 Jan 30 09:53 ghci -> ghci-8.2.2
-rwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 110 Jan 30 09:53 ghci-8.2.2
lrwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 17 Jan 30 09:53 haddock -> haddock-ghc-8.2.2
-rwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 449 Jan 30 09:53 haddock-ghc-8.2.2
-rwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 422 Jan 30 09:53 hp2ps
-rwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 420 Jan 30 09:53 hpc
-rwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 1239 Jan 30 09:53 hsc2hs
lrwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 12 Jan 30 09:53 runghc -> runghc-8.2.2
-rwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 466 Jan 30 09:53 runghc-8.2.2
lrwxr-xr-x 1 xxxxxxxxxx XXXX\Domain Users 6 Jan 30 09:53 runhaskell -> runghc
aarch64 bindist
Would it be possible to extend bindist to contain arm64 builds?
createSymbolicLink: already exists (File exists)
I'm not sure whether this is the same (or has the same cause) as #2, but here's another error on downloading the same cache on a reused build agent:
[cache-s3][Info ][Tue, 30 Jan 2018 03:11:46 UTC]: <cache-s3/xxxxxxxxxxxxxx/master.lts-10.3.stack.cache> - Restoring cache with total size: 724.7 MiB
cache-s3: /home/xxxxxxxxx/.stack/programs/x86_64-linux/ghc-8.2.2/bin/ghc: createSymbolicLink: already exists (File exists)
See the relevant archive contents in #2.
Concurrent uploading results in `WrongRequestBodyStreamSize`
Update to new version of stackage lts (thus all dependencies) resulted in concurrent uploading producing an error
[cache-s3][Info ][Sat, 14 Sep 2019 17:32:38 UTC]: <cache-s3/fpco/cache-s3/master.windows.lts-14.4.stack-work.cache> - Data change detected, caching 41.7 MiB with sha256: g1kXWt1frFHU7VM/9hvo0hjIAlJM7y4VtquJlJA6tYc=
[cache-s3][Warn ][Sat, 14 Sep 2019 17:32:40 UTC]: <cache-s3/fpco/cache-s3/master.windows.lts-14.4.stack-work.cache> - TransportError - WrongRequestBodyStreamSize 43626199 8388608
The issue has been reported upstream in axman6/amazonka-s3-streaming#17 as well as brendanhay/amazonka#548 so if anyone is willing to tackle it, feel free to do so, but until then, cache-s3 will be doing uploading sequentially from v0.1.10 until better times.
This feature was originally implemented in #20 so
CC @aledeganopix4d @wagdav, @mbj and @rvl
Storing relative paths in the .cache archive
First, thanks for making cache-s3. It's a really useful tool!
I'm using cache-s3 on a compilation cache database (created with clcache on Windows). My builds are running on a node in a temporary directory assigned randomly by the CI system. My compilation cache is a subfolder of the working directory.
If I do two consecutive builds, I'm assigned two different working directories, so that the build artifacts won't interfere. But since cache-s3 stores absolute paths I cannot restore the cache from the previous build.
In short, the first build
# working directory is $BUILD_DIR_1, distributed randomly by the CI system
[perform the build]
cache-s3 save -p $BUILD_DIR_1/cache # will save absolute paths
Now a consecutive build
# working directory is $BUILD_DIR_2, distributed randomly by the CI system
cache-s3 restore # will write into $BUILD_DIR_1/cache instead of $BUILD_DIR_2
[Something is written outside my working directory, but I don't know where?]
What would you suggest in this case? Would it be possible to save the cache content using relative paths?
Support restoring cache to a different home directory
Right now, even for files that are cached from inside a user's home directory, like ~/.stack, absolute path is stored and the files are restored to the same absolute path. This can create problems when restoring to a machine running under a different user with a different home directory (typically Permission denied).
Files cached from inside a home directory should be restored relative to the home directory, whatever it is on the new machine.
Add remove command.
It would be nice to have an ability to remove paths that would be cached/restored as a way to prepare a clean environment for restoration of previous cache. It is better to have it as a separate command cache-s3 rm that takes all the same arguments, rather than an extra step during cache-s3 restore, so it can be executed with escalated privileges if necessary. In particular this would be a good workaround for issues related to non-standard permissions as described in #6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.