Comments (14)
nigeltao
commented on July 19, 2024
4
That is weird. What does go env say for you?
CC'ing @AWSjswinney and @cuonglm for arm64 assembly things.
from snappy.
AWSjswinney
commented on July 19, 2024
1
I tested using the debian:buster container image which is using go 1.11.6. I can reproduce the problem there. It looks like that version of go requires the frame size to be 8 mod 16 to accommodate a link register. I did a little digging through the commit history and it looks like that requirement changed starting in go 1.12.
The frame sizes are currently 0 mod 16 but we can add 8 bytes to each function to make room for the link register and make this code backward compatible with 1.11 (and earlier?).
diff --git a/encode_arm64.s b/encode_arm64.s
index f8d54ad..f0c876a 100644
--- a/encode_arm64.s
+++ b/encode_arm64.s
@@ -27,7 +27,7 @@
// The unusual register allocation of local variables, such as R10 for the
// source pointer, matches the allocation used at the call site in encodeBlock,
// which makes it easier to manually inline this function.
-TEXT ·emitLiteral(SB), NOSPLIT, $32-56
+TEXT ·emitLiteral(SB), NOSPLIT, $40-56
MOVD dst_base+0(FP), R8
MOVD lit_base+24(FP), R10
MOVD lit_len+32(FP), R3
@@ -261,7 +261,7 @@ extendMatchEnd:
// "var table [maxTableSize]uint16" takes up 32768 bytes of stack space. An
// extra 64 bytes, to call other functions, and an extra 64 bytes, to spill
// local variables (registers) during calls gives 32768 + 64 + 64 = 32896.
-TEXT ·encodeBlock(SB), 0, $32896-56
+TEXT ·encodeBlock(SB), 0, $32904-56
MOVD dst_base+0(FP), R8
MOVD src_base+24(FP), R7
MOVD src_len+32(FP), R14I did just a basic go test on this, and it works on 1.11 and 1.17, but I haven't done any further testing. @nigeltao would you like to include a change like this for backward compatibility?
from snappy.
cuonglm
commented on July 19, 2024
@sigkill what's go version do you use?
from snappy.
sigkill
commented on July 19, 2024
Thank you for checking, I had been using what was supplied with the Ubuntu, Debian and Raspbian distros, but I removed and installed from the golang website instead.
The versions were:
From Debian, seems okay:
cat /etc/debian_version
11.0
go version
go version go1.15.9 linux/amd64
From Ubuntu:
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
apt info golang-go
Package: golang-go
Version: 2:1.13~1ubuntu2
Priority: optional
Section: devel
Source: golang-defaults
Origin: Ubuntu
Maintainer: Ubuntu Developers <[email protected]>
Original-Maintainer: Go Compiler Team <[email protected]>
Bugs: https://bugs.launchpad.net/ubuntu/+filebug
Installed-Size: 53.2 kB
Depends: golang-1.13-go, golang-src (>= 2:1.13~1ubuntu2)
Recommends: golang-race-detector-runtime
Suggests: git
Conflicts: gccgo-go
Breaks: gccgo-5 (<< 5.3.1-12~), gccgo-6 (<< 6-20160319-1~), golang-doc (<< 2:1.6.1+1~), golang-go-darwin-386, golang-go-darwin-amd64, golang-go-freebsd-386, golang-go-freebsd-amd64, golang-go-freebsd-arm, golang-go-linux-386, golang-go-linux-amd64, golang-go-linux-arm, golang-go-netbsd-386, golang-go-netbsd-amd64, golang-go-windows-386, golang-go-windows-amd64, golang-go.tools (<< 1:0.0~git20151026.0.0f9d71c-1~), golang-golang-x-tools (<< 1:0.0~git20151026.0.0f9d71c-1~)
Replaces: golang-doc (<< 2:1.6.1+1~), golang-go-darwin-386, golang-go-darwin-amd64, golang-go-freebsd-386, golang-go-freebsd-amd64, golang-go-freebsd-arm, golang-go-linux-386, golang-go-linux-amd64, golang-go-linux-arm, golang-go-netbsd-386, golang-go-netbsd-amd64, golang-go-windows-386, golang-go-windows-amd64, golang-go.tools (<< 1:0.0~git20151026.0.0f9d71c-1~), golang-golang-x-tools (<< 1:0.0~git20151026.0.0f9d71c-1~)
Homepage: https://golang.org
Download-Size: 22.0 kB
APT-Manual-Installed: yes
APT-Sources: http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages
Description: Go programming language compiler, linker, compiled stdlib
The Go programming language is an open source project to make programmers more
productive. Go is expressive, concise, clean, and efficient. Its concurrency
mechanisms make it easy to write programs that get the most out of multicore
and networked machines, while its novel type system enables flexible and
modular program construction. Go compiles quickly to machine code yet has the
convenience of garbage collection and the power of run-time reflection. It's a
fast, statically typed, compiled language that feels like a dynamically typed,
interpreted language.
.
This is a dependency package that depends on the default Go toolchain and
compiled standard library. It is only built on architectures that have a
native toolchain ("gc compiler"). Packages that want to build with whichever
of gc or gccgo is available should depend on golang-any.
from snappy.
sigkill
commented on July 19, 2024
The golang from the website appears to be okay, but the issue appears to have been with the package within one or two of the systems I have running Ubuntu, Raspbian, or Debian. The arm and amd64 versions from the golang site appear to work fine, and install without this issue.
from snappy.
AWSjswinney
commented on July 19, 2024
That looks like an amd64 platform which wouldn't see this problem at all, since it's for arm64 only. Presumably you are using a Raspberry Pi since you're using Raspbian. Can you check there?
from snappy.
sigkill
commented on July 19, 2024
AWSjswinney
I believe the golang-go package from Debian Buster is the same version that one of the arm based systems with the issue, prior to downloading a compiled tarball from the golang website, uninstalling the Debian package, and installing the golang website's version.
from snappy.
sigkill
commented on July 19, 2024
Aha!
Found one of the hosts that had an issue:
Golang, installed from website, no issues:
root@raspberrypi-0001:~# which go
/usr/local/go/bin/go
root@raspberrypi-0001:~# go version
go version go1.17.5 linux/arm64
The package from Raspbian appears to be the version with the issue:
root@raspberrypi-0001:~# apt info golang-go
Package: golang-go
Version: 2:1.11~1
Priority: optional
Section: devel
Source: golang-defaults
Maintainer: Go Compiler Team <[email protected]>
Installed-Size: 52.2 kB
Depends: golang-1.11-go, golang-src (>= 2:1.11~1)
Conflicts: gccgo-go
Breaks: gccgo-5 (<< 5.3.1-12~), gccgo-6 (<< 6-20160319-1~), golang-doc (<< 2:1.6.1+1~), golang-go-darwin-386, golang-go-darwin-amd64, golang-go-freebsd-386, golang-go-freebsd-amd64, golang-go-freebsd-arm, golang-go-linux-386, golang-go-linux-amd64, golang-go-linux-arm, golang-go-netbsd-386, golang-go-netbsd-amd64, golang-go-windows-386, golang-go-windows-amd64, golang-go.tools (<< 1:0.0~git20151026.0.0f9d71c-1~), golang-golang-x-tools (<< 1:0.0~git20151026.0.0f9d71c-1~)
Replaces: golang-doc (<< 2:1.6.1+1~), golang-go-darwin-386, golang-go-darwin-amd64, golang-go-freebsd-386, golang-go-freebsd-amd64, golang-go-freebsd-arm, golang-go-linux-386, golang-go-linux-amd64, golang-go-linux-arm, golang-go-netbsd-386, golang-go-netbsd-amd64, golang-go-windows-386, golang-go-windows-amd64, golang-go.tools (<< 1:0.0~git20151026.0.0f9d71c-1~), golang-golang-x-tools (<< 1:0.0~git20151026.0.0f9d71c-1~)
Homepage: https://golang.org
Tag: devel::TODO, devel::buildtools, devel::compiler, role::program
Download-Size: 23.4 kB
APT-Sources: http://deb.debian.org/debian buster/main arm64 Packages
Description: Go programming language compiler, linker, compiled stdlib
The Go programming language is an open source project to make programmers more
productive. Go is expressive, concise, clean, and efficient. Its concurrency
mechanisms make it easy to write programs that get the most out of multicore
and networked machines, while its novel type system enables flexible and
modular program construction. Go compiles quickly to machine code yet has the
convenience of garbage collection and the power of run-time reflection. It's a
fast, statically typed, compiled language that feels like a dynamically typed,
interpreted language.
.
This is a dependency package that depends on the default Go toolchain and
compiled standard library. It is only built on architectures that have a
native toolchain ("gc compiler"). Packages that want to build with whichever
of gc or gccgo is available should depend on golang-any.
from snappy.
sigkill
commented on July 19, 2024
As you stated, it appears to be based on buster:
root@raspberrypi-0001:~# cat /etc/debian_version
10.11
root@raspberrypi-0001:~# cat /etc/ap
apparmor.d/ apt/
root@raspberrypi-0001:~# cat /etc/apt/sources.list
deb http://deb.debian.org/debian buster main contrib non-free
deb http://deb.debian.org/debian-security/ buster/updates main contrib non-free
deb http://deb.debian.org/debian buster-updates main contrib non-free
# Uncomment deb-src lines below then 'apt-get update' to enable 'apt-get source'
#deb-src http://deb.debian.org/debian buster main contrib non-free
#deb-src http://deb.debian.org/debian-security/ buster/updates main contrib non-free
#deb-src http://deb.debian.org/debian buster-updates main contrib non-free
from snappy.
sigkill
commented on July 19, 2024
I did get the error on non-arm based distros as well, am trying to find the system that also had the issue, but it may have been an Alpine based host.
from snappy.
AWSjswinney
commented on July 19, 2024
The package from Raspbian appears to be the version with the issue:
Yes, that's consistent with my findings. Your options are:
- Use a go distribution from the website. Anything 1.12 or later should be fine, but 1.16 and 1.17 both have other arm64 optimizations, so I recommend the latest version.
- Manually patch snappy with the diff I posted above.
- Wait for a fix to be published by Snappy (if the project decided backward compatibility is important).
from snappy.
sigkill
commented on July 19, 2024
Thank you! I've gone with the website version for now, but very much appreciate all the effort and hard work to correct the issue and find the compatibility diff -- very cool!
from snappy.
sigkill
commented on July 19, 2024
@sigkill what's
goversion do you use?
Sorry, I should have posted these details when I opened up the issue. Thank you nigeltao and cuonglm as well.
from snappy.
nigeltao
commented on July 19, 2024
I pushed @AWSjswinney's suggestion (from a comment above). I don't have any arm64 devices handy, so if somebody could verify that it still works, I'll tag a v0.0.5.
I also assume that the $0-48 fragments below are still good (and don't need changing), even though 0 is not 8 (mod 16). Let me know if they also need updating.
$ grep ^TEXT *arm64.s
decode_arm64.s:TEXT ·decode(SB), NOSPLIT, $56-56
encode_arm64.s:TEXT ·emitLiteral(SB), NOSPLIT, $40-56
encode_arm64.s:TEXT ·emitCopy(SB), NOSPLIT, $0-48
encode_arm64.s:TEXT ·extendMatch(SB), NOSPLIT, $0-48
encode_arm64.s:TEXT ·encodeBlock(SB), 0, $32904-56
from snappy.
Related Issues (20)
- Question: Encode <> Decode size HOT 3
- I can't attach the snappy.Reader to a string.. HOT 1
- Is snappy slower than gzip in decompression HOT 3
- SemVer Releases HOT 4
- missing function HOT 1
- Why does snappy encodes the file ( increases the size ) instead of compressing it ? HOT 1
- Customize encode/decode buffer size
- not support hadoop-snappy format
- How come snappytool does not support stream format? HOT 1
- How to decompress a 2GB file on a machine with only 1GB of RAM
- Test failure on 32 bits arches HOT 1
- How should I validate encoded data? HOT 1
- Not working on apple m1 HOT 10
- [@] and[.] error
- Incorrect comment on NewBufferedWriter?
- Possible to extend output buffer to capacity when encoding/decoding? HOT 1
- How to debug snappy: corrupt input? HOT 2
- Unable to decompress Snappy JSON file using golang/snappy HOT 1
- High Memory Usage: Snappy Encoding
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snappy.