postmodern / digest-crc Goto Github PK
View Code? Open in Web Editor NEWA Cyclic Redundancy Check (CRC) library for Ruby.
License: MIT License
A Cyclic Redundancy Check (CRC) library for Ruby.
License: MIT License
The README states that:
(
digest-crc
) supports optional C extensions which increases performance by ~40x. If the C extensions cannot be compiled for whatever reason,digest-crc
will automatically fallback to the pure-Ruby implementation.
However, the gem's native extensions can't be installed (on Catalina at least) without Xcode command line tools and no fallback seems to be provided:
Would it make sense to have benchmarks for one or a few of the ruby built in digests as well, so that there is some ballpark comparison number?
I was just wondering what the performance is like:
require 'benchmark/ips'
require 'digest'
TEST_STRING = 'This is a pretty short test string.'.freeze
Benchmark.ips do |x|
x.config(time: 3, warmup: 1)
x.report('md5') do
Digest::MD5.hexdigest(TEST_STRING)
end
x.report('sha1') do
Digest::SHA1.hexdigest(TEST_STRING)
end
x.report('sha2') do
Digest::SHA2.hexdigest(TEST_STRING)
end
x.report('crc16') do
Digest::CRC16.hexdigest(TEST_STRING)
end
x.report('crc32') do
Digest::CRC32.hexdigest(TEST_STRING)
end
x.report('crc64') do
Digest::CRC64.hexdigest(TEST_STRING)
end
x.compare!
end
Results from Ruby 2.4 on Fedora 27 with Ryzen 1700:
Warming up --------------------------------------
md5 68.196k i/100ms
sha1 71.052k i/100ms
sha2 41.683k i/100ms
crc16 11.360k i/100ms
crc32 10.111k i/100ms
crc64 5.584k i/100ms
Calculating -------------------------------------
md5 879.284k (± 0.4%) i/s - 2.660M in 3.024840s
sha1 956.697k (± 0.4%) i/s - 2.913M in 3.045031s
sha2 512.701k (± 2.3%) i/s - 1.542M in 3.009928s
crc16 120.649k (± 1.2%) i/s - 363.520k in 3.013477s
crc32 106.223k (± 0.8%) i/s - 323.552k in 3.046145s
crc64 57.542k (± 0.7%) i/s - 173.104k in 3.008440s
Comparison:
sha1: 956696.7 i/s
md5: 879283.8 i/s - 1.09x slower
sha2: 512701.3 i/s - 1.87x slower
crc16: 120648.5 i/s - 7.93x slower
crc32: 106223.5 i/s - 9.01x slower
crc64: 57542.4 i/s - 16.63x slower
Performance is kinda expected, what's a bit off from my expectations is how much CRC64 is slower on 64bit native CPU. It will only get better with Ruby MJIT I guess. Thanks for this wonderful gem!
I tried to install digest-crc (for fastlane) on Windows, but It seems that trying to install digest-crc when Ruby path contains a space
character it breaks with an InvalidArgument error
Ruby installation path : D:\User Programs\Ruby30-x64
Error
current directory: D:/User Programs/Ruby30-x64/lib/ruby/gems/3.0.0/gems/digest-crc-0.6.4/ext/digest
\"D:/User\ Programs/Ruby30-x64/bin/ruby.exe\" -ID:/User\ Programs/Ruby30-x64/lib/ruby/3.0.0 -rrubygems D:/User\ Programs/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.6/exe/rake RUBYARCHDIR\=D:/User\ Programs/Ruby30-x64/lib/ruby/gems/3.0.0/extensions/x64-mingw32/3.0.0/digest-crc-0.6.4 RUBYLIBDIR\=D:/User\ Programs/Ruby30-x64/lib/ruby/gems/3.0.0/extensions/x64-mingw32/3.0.0/digest-crc-0.6.4
rake failedInvalid argument - "D:/User Programs/Ruby30-x64/bin/ruby.exe"
It could also be a Rake error
The README says the C extensions are optional, and that if it fails for whatever reason, it will fallback to a pure-Ruby implementation. However, I couldn't get it to install without the C extensions. To reproduce run:
docker run -it ruby:2.7.1-alpine gem install digest-crc
That container doesn't have any build tools installed by default, so it can't install the C extensions. It fails to install with compiler errors.
Now that there is postmodern/digest-crc.cr, this repo should be renamed to digest-crc.rb
to avoid confusion. I'll need to also hunt down any direct links and update them.
Related to issue #22, we should perform integration testing by installing a built gem in various docker images, w/o gcc or w/o make.
$ bundle install
....
Installing digest-crc 0.6.0 (was 0.5.1) with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory:
/Users/guoqi/.bundle/vendor/ruby/2.5.0/gems/digest-crc-0.6.0/ext/digest
rake
RUBYARCHDIR=/Users/guoqi/.bundle/vendor/ruby/2.5.0/extensions/x86_64-darwin-18/2.5.0/digest-crc-0.6.0
RUBYLIBDIR=/Users/guoqi/.bundle/vendor/ruby/2.5.0/extensions/x86_64-darwin-18/2.5.0/digest-crc-0.6.0
/Users/guoqi/.rvm/rubies/ruby-2.5.0/lib/ruby/site_ruby/2.5.0/rubygems.rb:289:in
`find_spec_for_exe': can't find gem rake (>= 0.a) with executable rake
(Gem::GemNotFoundException)
from
/Users/guoqi/.rvm/rubies/ruby-2.5.0/lib/ruby/site_ruby/2.5.0/rubygems.rb:308:in
`activate_bin_path'
from /Users/guoqi/.rvm/gems/ruby-2.5.0@global/bin/rake:23:in `<main>'
from /Users/guoqi/.rvm/gems/ruby-2.5.0/bin/ruby_executable_hooks:24:in `eval'
from /Users/guoqi/.rvm/gems/ruby-2.5.0/bin/ruby_executable_hooks:24:in `<main>'
There is not rake
gem in my mac and my project. I think that should add a dependency in gemspec?
CRC-16-QT defined by digest-crc has exactly the same characteristics as CRC-16-X-25.
I think it is better to redefine it as Digest::CRC16QT = Digest::CRC16X25
as an alias, or to abolish Digest::CRC16QT
itself.
If I call
gem install --no-document rake
gem install digest-crc
Output:
ERROR: Error installing digest-crc:
ERROR: Failed to build gem native extension.
current directory: /var/lib/gems/2.5.0/gems/digest-crc-0.6.3/ext/digest
/usr/bin/ruby2.5 -rrubygems /usr/share/rubygems-integration/all/gems/rake-12.3.1/exe/rake RUBYARCHDIR=/var/lib/gems/2.5.0/extensions/x86_64-linux/2.5.0/digest-crc-0.6.3 RUBYLIBDIR=/var/lib/gems/2.5.0/extensions/x86_64-linux/2.5.0/digest-crc-0.6.3
/usr/bin/ruby2.5: No such file or directory -- /usr/share/rubygems-integration/all/gems/rake-12.3.1/exe/rake (LoadError)
rake failed, exit code 1
Gem files will remain installed in /var/lib/gems/2.5.0/gems/digest-crc-0.6.3 for inspection.
Results logged to /var/lib/gems/2.5.0/extensions/x86_64-linux/2.5.0/digest-crc-0.6.3/gem_make.out
It is important that you NEVER had rake
installed with documents, because it somehow remembers them somewhere even after gem uninstall rake
.
Also I do not know, why exactly this happens. I thought I would not need rake
to build this extension, if I have gcc
and make
installed on Debian.
Can you reproduce this behavior?
Awesome project. Would be just what I need if it supported crc-32c. Could you add it? pycrc can create its table.
Now that the 16bit qt algorithm is in, is there any chance we could have an updated gem? I have another gem which wants to use qt :-)
Any one else see something similar? works in 2.5.3. Using rvm to install 2.7.1. All other gems seem to be just fine.
current directory: /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
["/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby", "-I/Users/ab/.rvm/rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems", "-rrubygems", "/Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake", "RUBYARCHDIR=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3", "RUBYLIBDIR=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3"]
/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby -S extconf.rb
checking for stdint.h... yes
checking for stddef.h... yes
creating extconf.h
creating Makefile
rake aborted!
Errno::EINTR: Interrupted system call @ dir_chdir - /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
/Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest/Rakefile:23:in `chdir'
/Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest/Rakefile:23:in `block (2 levels) in <top (required)>'
/Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake:27:in `<main>'
Tasks: TOP => default => crc16_genibus/crc16_genibus_ext.bundle => crc16_genibus/Makefile
(See full trace by running task with --trace)
ERROR: Error installing digest-crc:
ERROR: Failed to build gem native extension.
current directory: /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby -I/Users/ab/.rvm/rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems -rrubygems /Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake RUBYARCHDIR\=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3 RUBYLIBDIR\=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3
Building has failed. See above output for more information on the failure.
rake failed, exit code 1
Gem files will remain installed in /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3 for inspection.
Results logged to /Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3/gem_make.out
I got the following error trying to run bundle install
on a Fastlane project:
An error occurred while installing digest-crc (0.6.1), and Bundler cannot
continue.
Make sure that `gem install digest-crc -v '0.6.1' --source
'https://rubygems.org/'` succeeds before bundling.
In Gemfile:
xcode-install was resolved to 2.6.7, which depends on
fastlane was resolved to 2.167.0, which depends on
google-cloud-storage was resolved to 1.29.1, which depends on
digest-crc
I tried to run sudo gem install digest-crc -v '0.6.1' --source 'https://rubygems.org/'
as it says, and I got this error:
Building native extensions. This could take a while...
ERROR: Error installing digest-crc:
ERROR: Failed to build gem native extension.
current directory: /Library/Ruby/Gems/2.6.0/gems/digest-crc-0.6.1/ext/digest
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby -rrubygems /Library/Ruby/Gems/2.6.0/gems/rake-13.0.1/exe/rake RUBYARCHDIR\=/Library/Ruby/Gems/2.6.0/extensions/universal-darwin-20/2.6.0/digest-crc-0.6.1 RUBYLIBDIR\=/Library/Ruby/Gems/2.6.0/extensions/universal-darwin-20/2.6.0/digest-crc-0.6.1
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby -S extconf.rb
checking for stdint.h... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers. Check the mkmf.log file for more details. You may
need configuration options.
Provided configuration options:
--with-opt-dir
--without-opt-dir
--with-opt-include
--without-opt-include=${opt-dir}/include
--with-opt-lib
--without-opt-lib=${opt-dir}/lib
--with-make-prog
--without-make-prog
--srcdir=.
--curdir
--ruby=/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/$(RUBY_BASE_NAME)
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:467:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:585:in `block in try_compile'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:534:in `with_werror'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:585:in `try_compile'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:1109:in `block in have_header'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:959:in `block in checking_for'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:361:in `block (2 levels) in postpone'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:331:in `open'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:361:in `block in postpone'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:331:in `open'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:357:in `postpone'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:958:in `checking_for'
from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:1108:in `have_header'
from extconf.rb:3:in `<main>'
rake aborted!
Command failed with status (1): [/System/Library/Frameworks/Ruby.framework/...]
/Library/Ruby/Gems/2.6.0/gems/digest-crc-0.6.1/ext/digest/Rakefile:32:in `block (3 levels) in <top (required)>'
/Library/Ruby/Gems/2.6.0/gems/digest-crc-0.6.1/ext/digest/Rakefile:31:in `chdir'
/Library/Ruby/Gems/2.6.0/gems/digest-crc-0.6.1/ext/digest/Rakefile:31:in `block (2 levels) in <top (required)>'
/Library/Ruby/Gems/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<main>'
Tasks: TOP => default => crc16_genibus/crc16_genibus_ext.bundle => crc16_genibus/Makefile
(See full trace by running task with --trace)
rake failed, exit code 1
Gem files will remain installed in /Library/Ruby/Gems/2.6.0/gems/digest-crc-0.6.1 for inspection.
Results logged to /Library/Ruby/Gems/2.6.0/extensions/universal-darwin-20/2.6.0/digest-crc-0.6.1/gem_make.out
This was working last week, and the only change on this machine is that it got updated (against my will) to macOS 11 Big Sur.
It has been a very long time since I've worked with Ruby (other than Fastlane), so I'm not really sure where to start looking to see what the problem is.
Hello!
Is there a way to override the TABLE
constant correctly?
Hello,
I would like to ask why there is a strict version requirement in gemspec of 0.6.1
version.
It currently breaks existing systems where the original Gemfile
does not have a specific rake
dependency defined.
Default rake
that exists in ruby installations has version 12 for ruby versions 2.6 and a few earlier ones. With digest-crc
gemspec change, calling rake
directly has started to fail saying we should run with bundle exec rake
.
In short, is it possible to have a wider range of supported rake versions there?
Thanks
Attempt to port nokogiri's GitHub Actions config that uses rake-compiler-dock to cross compile. This should reduce compilation issues and allow us to drop rake
as a dependency.
I have the need for a CRC 64 "Jones" variant, which is basically the CRC64 you have now with a modified table. Additionally, I need to be able to set the initial value of the CRC state prior to the first update call.
I have code that works for this (just the table changed after all) and have added a set(value)
method which will set the crc state. However, as none of the other CRC calculators allow this set, I'm not sure if I should keep it class specific or make a PR that adds it to all CRCs.
Another option would be to change reset()
to be reset(initial_value = @initial_value)
Which would you prefer?
Hi @postmodern,
From my observation, the current implementation of CRC32C checksum is in pure Ruby. For some use cases that depends on this checksum mechanism, it looks like the performance is a huge problem. One example is that in ruby-kafka project, which is a Kafka client for ruby, the CRC32C checksum takes up to 20 - 30% of the processing time (zendesk/ruby-kafka#620). We use this algorithm heavily to ensure the integrity of messages sent to / received from Kafka.
Therefore, I would like to ask about moving this checksum algorithm to C extension. I suggest to just implement a naive porting, which still saves a huge amount of processing time, or use Google's implementation (https://github.com/google/crc32c) as a dependency.
If you agree, I will open an PR for this.
There is a C extension ext/digest/crc12_3gpp/crc12_3gpp_ext.c
file, but the Ruby implementation (lib/digest/crc12_3gpp.rb
file) is missing and consequently unavailable.
There is also no corresponding spec/crc12_3gpp_spec.rb
file.
However, there is no description in README.md
either, so there seems to be no problem even if it is not available.
Fetching digest-crc 0.6.0 (was 0.4.1)
Installing digest-crc 0.6.0 (was 0.4.1) with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
This error only happens on 0.6.0, which was released today. Didn't happen with older versions.
Any one else see something similar?
current directory: /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
["/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby", "-I/Users/ab/.rvm/rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems", "-rrubygems", "/Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake", "RUBYARCHDIR=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3", "RUBYLIBDIR=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3"]
/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby -S extconf.rb
checking for stdint.h... yes
checking for stddef.h... yes
creating extconf.h
creating Makefile
rake aborted!
Errno::EINTR: Interrupted system call @ dir_chdir - /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
/Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest/Rakefile:23:in `chdir'
/Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest/Rakefile:23:in `block (2 levels) in <top (required)>'
/Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake:27:in `<main>'
Tasks: TOP => default => crc16_genibus/crc16_genibus_ext.bundle => crc16_genibus/Makefile
(See full trace by running task with --trace)
ERROR: Error installing digest-crc:
ERROR: Failed to build gem native extension.
current directory: /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby -I/Users/ab/.rvm/rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems -rrubygems /Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake RUBYARCHDIR\=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3 RUBYLIBDIR\=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3
Building has failed. See above output for more information on the failure.
rake failed, exit code 1
Gem files will remain installed in /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3 for inspection.
Results logged to /Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3/gem_make.out
hey guys
can someone help me?
I'm trying to run fastlane with pipeline azure and then with fastlane call discord_notifier (https://github.com/nikolas-theodosis/fastlane-discord_notifier)
so
pipeline azure -> fastlane -> discord_notifier
if I only test fastlane with the discord notifier it works fine, but if the pipline triggers the fastlane (bundle exec fastlane dev) it doesn't work
erro on pipeline -> bundle exec fastlane dev:
running only fastlane and notifier:
digest-crc is instaled locally
Add a benchmark script to test the various CRC algorithms against a random block of data. This will help us establish a performance base-line. Bonus points if it can dump out the stats directly to CSV.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.