postmodern / digest-crc Goto Github PK

View Code? Open in Web Editor NEW

91.0 8.0 27.0 208 KB

A Cyclic Redundancy Check (CRC) library for Ruby.

License: MIT License

Ruby 47.04% C 52.96%

ruby digest crc crc8 crc16 crc32 crc64

digest-crc's Introduction

Digest CRC

Description

Adds support for calculating Cyclic Redundancy Check (CRC) to the Digest module.

Features

Provides support for the following CRC algorithms:
- CRC1
- CRC5
- CRC8
- CRC8 1-Wire
- CRC15
- CRC16
- CRC16 CCITT
- CRC16 DNP
- CRC16 Genibus
- CRC16 Kermit
- CRC16 Modbus
- CRC16 USB
- CRC16 X25
- CRC16 XModem
- CRC16 ZModem
- CRC16 QT
- CRC24
- CRC32
- CRC32 BZip2
- CRC32c
- CRC32 Jam
- CRC32 MPEG
- CRC32 POSIX
- CRC32 XFER
- CRC64
- CRC64 Jones
- CRC64 XZ
Pure Ruby implementation.
Provides CRC Tables for optimized calculations.
Supports optional C extensions which increases performance by ~40x.
- If the C extensions cannot be compiled for whatever reason, digest-crc will automatically fallback to the pure-Ruby implementation.

Install

gem install digest-crc

Note: to enable the C extensions ensure that you are using CRuby and have a C compiler (gcc or clang) and make installed, before installing digest-crc.

Debian / Ubuntu:
```
$ sudo apt install gcc make
```
RedHat / Fedora:
```
$ sudo dnf install gcc make
```
Alpine Linux:
```
$ apk add build-base
```
macOS: install XCode

Examples

Calculate a CRC32:

require 'digest/crc32'

Digest::CRC32.hexdigest('hello')
# => "3610a686"

Calculate a CRC32 of a file:

Digest::CRC32.file('README.md')
# => #<Digest::CRC32: 127ad531>

Incrementally calculate a CRC32:

crc = Digest::CRC32.new
crc << 'one'
crc << 'two'
crc << 'three'
crc.hexdigest
# => "09e1c092"

Directly access the checksum:

crc.checksum
# => 165789842

Defining your own CRC class:

require 'digest/crc32'

module Digest
  class CRC3000 < CRC32

    WIDTH = 32

    REFLECT_INPUT = true

    INIT_CRC = 0xffffffff

    XOR_MASK = 0xffffffff

    TABLE = [
      # ....
    ].freeze
  end
end

Benchmarks

Ruby 2.7.4 (pure Ruby)

$ bundle exec rake clean
$ bundle exec ./benchmarks.rb
Loading Digest::CRC classes ...
Generating 1000 8Kb lengthed strings ...
Benchmarking Digest::CRC classes ...
                                  user     system      total        real
Digest::CRC1#update           0.423741   0.000000   0.423741 (  0.425887)
Digest::CRC5#update           1.486578   0.000011   1.486589 (  1.493215)
Digest::CRC8#update           1.261386   0.000000   1.261386 (  1.266399)
Digest::CRC8_1Wire#update     1.250344   0.000000   1.250344 (  1.255009)
Digest::CRC15#update          1.482515   0.000000   1.482515 (  1.488131)
Digest::CRC16#update          1.216744   0.000811   1.217555 (  1.222228)
Digest::CRC16CCITT#update     1.480490   0.000000   1.480490 (  1.486745)
Digest::CRC16DNP#update       1.200067   0.000000   1.200067 (  1.204835)
Digest::CRC16Genibus#update   1.492910   0.000000   1.492910 (  1.498923)
Digest::CRC16Modbus#update    1.217449   0.000003   1.217452 (  1.222348)
Digest::CRC16QT#update        1.223311   0.000000   1.223311 (  1.229211)
Digest::CRC16USB#update       1.233744   0.000000   1.233744 (  1.238615)
Digest::CRC16X25#update       1.223077   0.000000   1.223077 (  1.227607)
Digest::CRC16XModem#update    1.487674   0.000000   1.487674 (  1.493316)
Digest::CRC16ZModem#update    1.484288   0.000000   1.484288 (  1.490096)
Digest::CRC24#update          1.490272   0.000000   1.490272 (  1.496027)
Digest::CRC32#update          1.225311   0.000000   1.225311 (  1.230572)
Digest::CRC32BZip2#update     1.503096   0.000000   1.503096 (  1.509202)
Digest::CRC32c#update         1.220390   0.000000   1.220390 (  1.225487)
Digest::CRC32Jam#update       1.216066   0.000000   1.216066 (  1.220591)
Digest::CRC32MPEG#update      1.486808   0.000000   1.486808 (  1.492611)
Digest::CRC32POSIX#update     1.494508   0.000957   1.495465 (  1.503262)
Digest::CRC32XFER#update      1.504802   0.005830   1.510632 (  1.522066)
Digest::CRC64#update          3.260784   0.015674   3.276458 (  3.310506)
Digest::CRC64Jones#update     3.195204   0.000000   3.195204 (  3.213054)
Digest::CRC64XZ#update        3.173597   0.000000   3.173597 (  3.190438)

Ruby 2.7.4 (C extensions)

$ bundle exec rake build:c_exts
...
$ bundle exec ./benchmarks.rb
Loading Digest::CRC classes ...
Generating 1000 8Kb lengthed strings ...
Benchmarking Digest::CRC classes ...
                                  user     system      total        real
Digest::CRC1#update           0.443619   0.000007   0.443626 (  0.446545)
Digest::CRC5#update           0.025134   0.000806   0.025940 (  0.026129)
Digest::CRC8#update           0.022564   0.000000   0.022564 (  0.022775)
Digest::CRC8_1Wire#update     0.021427   0.000008   0.021435 (  0.021551)
Digest::CRC15#update          0.030377   0.000833   0.031210 (  0.031406)
Digest::CRC16#update          0.024004   0.000002   0.024006 (  0.024418)
Digest::CRC16CCITT#update     0.026930   0.000001   0.026931 (  0.027238)
Digest::CRC16DNP#update       0.024279   0.000000   0.024279 (  0.024446)
Digest::CRC16Genibus#update   0.026477   0.000004   0.026481 (  0.026656)
Digest::CRC16Modbus#update    0.023568   0.000000   0.023568 (  0.023704)
Digest::CRC16QT#update        0.024161   0.000000   0.024161 (  0.024316)
Digest::CRC16USB#update       0.023891   0.000000   0.023891 (  0.024038)
Digest::CRC16X25#update       0.023849   0.000000   0.023849 (  0.023991)
Digest::CRC16XModem#update    0.026254   0.000000   0.026254 (  0.026523)
Digest::CRC16ZModem#update    0.026391   0.000000   0.026391 (  0.026529)
Digest::CRC24#update          0.028805   0.000854   0.029659 (  0.029830)
Digest::CRC32#update          0.024030   0.000000   0.024030 (  0.024200)
Digest::CRC32BZip2#update     0.026942   0.000000   0.026942 (  0.027244)
Digest::CRC32c#update         0.023989   0.000000   0.023989 (  0.024159)
Digest::CRC32Jam#update       0.023940   0.000000   0.023940 (  0.024066)
Digest::CRC32MPEG#update      0.027063   0.000000   0.027063 (  0.027213)
Digest::CRC32POSIX#update     0.027137   0.000000   0.027137 (  0.028160)
Digest::CRC32XFER#update      0.026956   0.000002   0.026958 (  0.027103)
Digest::CRC64#update          0.024222   0.000005   0.024227 (  0.024796)
Digest::CRC64Jones#update     0.025331   0.000000   0.025331 (  0.025789)
Digest::CRC64XZ#update        0.024131   0.000001   0.024132 (  0.024348)

Ruby 3.0.2 (pure Ruby)

$ bundle exec rake clean
$ bundle exec ./benchmarks.rb
Loading Digest::CRC classes ...
Generating 1000 8Kb lengthed strings ...
Benchmarking Digest::CRC classes ...
                                  user     system      total        real
Digest::CRC1#update           0.331405   0.000002   0.331407 (  0.333588)
Digest::CRC5#update           1.206847   0.000020   1.206867 (  1.224072)
Digest::CRC8#update           1.018571   0.000000   1.018571 (  1.023002)
Digest::CRC8_1Wire#update     1.018802   0.000000   1.018802 (  1.023292)
Digest::CRC15#update          1.207586   0.000000   1.207586 (  1.212691)
Digest::CRC16#update          1.032505   0.000965   1.033470 (  1.040862)
Digest::CRC16CCITT#update     1.198079   0.000000   1.198079 (  1.203134)
Digest::CRC16DNP#update       0.994582   0.000000   0.994582 (  1.006520)
Digest::CRC16Genibus#update   1.190596   0.000000   1.190596 (  1.196087)
Digest::CRC16Modbus#update    1.007826   0.000000   1.007826 (  1.012934)
Digest::CRC16QT#update        0.996298   0.000001   0.996299 (  1.000255)
Digest::CRC16USB#update       0.995806   0.000000   0.995806 (  0.999822)
Digest::CRC16X25#update       1.019589   0.000000   1.019589 (  1.031010)
Digest::CRC16XModem#update    1.146947   0.000000   1.146947 (  1.150817)
Digest::CRC16ZModem#update    1.145145   0.000000   1.145145 (  1.149483)
Digest::CRC24#update          1.149009   0.000000   1.149009 (  1.152854)
Digest::CRC32#update          0.970976   0.000000   0.970976 (  0.974227)
Digest::CRC32BZip2#update     1.148596   0.000000   1.148596 (  1.152381)
Digest::CRC32c#update         0.972566   0.000000   0.972566 (  0.975790)
Digest::CRC32Jam#update       0.975854   0.000000   0.975854 (  0.979217)
Digest::CRC32MPEG#update      1.148578   0.000000   1.148578 (  1.153088)
Digest::CRC32POSIX#update     1.146218   0.000986   1.147204 (  1.152460)
Digest::CRC32XFER#update      1.149823   0.000000   1.149823 (  1.153692)
Digest::CRC64#update          2.869948   0.000016   2.869964 (  2.884261)
Digest::CRC64Jones#update     2.867662   0.000000   2.867662 (  2.886559)
Digest::CRC64XZ#update        2.858847   0.000000   2.858847 (  2.874058)

Ruby 3.0.2 (C extensions)

$ bundle exec rake build:c_exts
...
$ bundle exec ./benchmarks.rb
Loading Digest::CRC classes ...
Generating 1000 8Kb lengthed strings ...
Benchmarking Digest::CRC classes ...
                                  user     system      total        real
Digest::CRC1#update           0.349055   0.000000   0.349055 (  0.350454)
Digest::CRC5#update           0.023144   0.000000   0.023144 (  0.023248)
Digest::CRC8#update           0.021378   0.000000   0.021378 (  0.021522)
Digest::CRC8_1Wire#update     0.021019   0.000000   0.021019 (  0.021145)
Digest::CRC15#update          0.030063   0.000003   0.030066 (  0.030245)
Digest::CRC16#update          0.024395   0.000000   0.024395 (  0.024572)
Digest::CRC16CCITT#update     0.026979   0.000000   0.026979 (  0.027138)
Digest::CRC16DNP#update       0.024665   0.000000   0.024665 (  0.024844)
Digest::CRC16Genibus#update   0.027054   0.000000   0.027054 (  0.027217)
Digest::CRC16Modbus#update    0.023963   0.000000   0.023963 (  0.024257)
Digest::CRC16QT#update        0.024218   0.000000   0.024218 (  0.024360)
Digest::CRC16USB#update       0.024393   0.000000   0.024393 (  0.024561)
Digest::CRC16X25#update       0.025127   0.000000   0.025127 (  0.025292)
Digest::CRC16XModem#update    0.028123   0.000000   0.028123 (  0.028377)
Digest::CRC16ZModem#update    0.028205   0.000000   0.028205 (  0.028571)
Digest::CRC24#update          0.031386   0.000000   0.031386 (  0.031740)
Digest::CRC32#update          0.023832   0.000000   0.023832 (  0.023948)
Digest::CRC32BZip2#update     0.027159   0.000000   0.027159 (  0.027315)
Digest::CRC32c#update         0.024172   0.000000   0.024172 (  0.024310)
Digest::CRC32Jam#update       0.024376   0.000000   0.024376 (  0.024494)
Digest::CRC32MPEG#update      0.026035   0.000784   0.026819 (  0.026940)
Digest::CRC32POSIX#update     0.026784   0.000000   0.026784 (  0.026907)
Digest::CRC32XFER#update      0.026770   0.000000   0.026770 (  0.026893)
Digest::CRC64#update          0.024400   0.000009   0.024409 (  0.024531)
Digest::CRC64Jones#update     0.023477   0.000781   0.024258 (  0.024390)
Digest::CRC64XZ#update        0.024611   0.000000   0.024611 (  0.024779)

JRuby 9.2.18.0 (pure Ruby)

$ bundle exec ./benchmarks.rb
Loading Digest::CRC classes ...
Generating 1000 8Kb lengthed strings ...
Benchmarking Digest::CRC classes ...
                                  user     system      total        real
Digest::CRC1#update           1.080000   0.050000   1.130000 (  0.676022)
Digest::CRC5#update           2.030000   0.040000   2.070000 (  1.089240)
Digest::CRC8#update           1.590000   0.000000   1.590000 (  0.999138)
Digest::CRC8_1Wire#update     0.920000   0.010000   0.930000 (  0.873813)
Digest::CRC15#update          1.470000   0.030000   1.500000 (  1.118886)
Digest::CRC16#update          1.780000   0.010000   1.790000 (  1.067874)
Digest::CRC16CCITT#update     1.500000   0.070000   1.570000 (  1.185564)
Digest::CRC16DNP#update       1.250000   0.000000   1.250000 (  0.972322)
Digest::CRC16Genibus#update   1.700000   0.010000   1.710000 (  1.092047)
Digest::CRC16Modbus#update    1.000000   0.010000   1.010000 (  0.915328)
Digest::CRC16QT#update        1.250000   0.000000   1.250000 (  0.968528)
Digest::CRC16USB#update       1.150000   0.010000   1.160000 (  0.990387)
Digest::CRC16X25#update       0.940000   0.000000   0.940000 (  0.926926)
Digest::CRC16XModem#update    1.390000   0.010000   1.400000 (  1.100584)
Digest::CRC16ZModem#update    1.760000   0.020000   1.780000 (  1.094003)
Digest::CRC24#update          1.690000   0.010000   1.700000 (  1.106875)
Digest::CRC32#update          1.410000   0.020000   1.430000 (  1.082506)
Digest::CRC32BZip2#update     1.510000   0.010000   1.520000 (  1.104225)
Digest::CRC32c#update         1.270000   0.010000   1.280000 (  1.023881)
Digest::CRC32Jam#update       1.190000   0.010000   1.200000 (  0.998146)
Digest::CRC32MPEG#update      1.580000   0.010000   1.590000 (  1.099086)
Digest::CRC32POSIX#update     1.550000   0.010000   1.560000 (  1.142051)
Digest::CRC32XFER#update      1.360000   0.000000   1.360000 (  1.071381)
Digest::CRC64#update          3.730000   0.020000   3.750000 (  2.780390)
Digest::CRC64Jones#update     2.710000   0.020000   2.730000 (  2.608007)
Digest::CRC64XZ#update        2.910000   0.020000   2.930000 (  2.629401)

TruffleRuby 21.2.0 (pure Ruby)

$ bundle exec rake clean
$ bundle exec ./benchmarks.rb
Loading Digest::CRC classes ...
Generating 1000 8Kb lengthed strings ...
Benchmarking Digest::CRC classes ...
                                  user     system      total        real
Digest::CRC1#update           0.455340   0.000000   0.455340 (  0.457710)
Digest::CRC5#update           1.406700   0.000000   1.406700 (  1.412535)
Digest::CRC8#update           1.248323   0.000000   1.248323 (  1.255452)
Digest::CRC8_1Wire#update     1.269434   0.000000   1.269434 (  1.275315)
Digest::CRC15#update          1.428752   0.000000   1.428752 (  1.434836)
Digest::CRC16#update          1.220394   0.000967   1.221361 (  1.229684)
Digest::CRC16CCITT#update     1.434932   0.001000   1.435932 (  1.452391)
Digest::CRC16DNP#update       1.191351   0.000000   1.191351 (  1.202262)
Digest::CRC16Genibus#update   1.434067   0.000000   1.434067 (  1.440300)
Digest::CRC16Modbus#update    1.200827   0.000000   1.200827 (  1.205658)
Digest::CRC16QT#update        1.195077   0.000000   1.195077 (  1.200328)
Digest::CRC16USB#update       1.196266   0.000000   1.196266 (  1.201262)
Digest::CRC16X25#update       1.206690   0.000000   1.206690 (  1.211781)
Digest::CRC16XModem#update    1.430468   0.000000   1.430468 (  1.436801)
Digest::CRC16ZModem#update    1.442524   0.000000   1.442524 (  1.448624)
Digest::CRC24#update          1.447611   0.000018   1.447629 (  1.454534)
Digest::CRC32#update          1.214314   0.000000   1.214314 (  1.219838)
Digest::CRC32BZip2#update     1.427408   0.000000   1.427408 (  1.433626)
Digest::CRC32c#update         1.204985   0.000000   1.204985 (  1.210273)
Digest::CRC32Jam#update       1.235039   0.000000   1.235039 (  1.240686)
Digest::CRC32MPEG#update      1.429731   0.000000   1.429731 (  1.435404)
Digest::CRC32POSIX#update     1.458886   0.000000   1.458886 (  1.465914)
Digest::CRC32XFER#update      1.422109   0.000000   1.422109 (  1.427635)
Digest::CRC64#update          3.283506   0.000000   3.283506 (  3.303129)
Digest::CRC64Jones#update     3.297402   0.000000   3.297402 (  3.317357)
Digest::CRC64XZ#update        3.278551   0.001875   3.280426 (  3.315165)

TruffleRuby 21.2.0 (C extensions)

$ bundle exec rake build:c_exts
...
$ bundle exec ./benchmarks.rb
Loading Digest::CRC classes ...
Generating 1000 8Kb lengthed strings ...
Benchmarking Digest::CRC classes ...
                                  user     system      total        real
Digest::CRC1#update           0.480586   0.000014   0.480600 (  0.482817)
Digest::CRC5#update           0.023795   0.000000   0.023795 (  0.023941)
Digest::CRC8#update           0.020619   0.000000   0.020619 (  0.020747)
Digest::CRC8_1Wire#update     0.020571   0.000000   0.020571 (  0.020700)
Digest::CRC15#update          0.031224   0.000000   0.031224 (  0.031412)
Digest::CRC16#update          0.024013   0.000000   0.024013 (  0.024174)
Digest::CRC16CCITT#update     0.026790   0.000000   0.026790 (  0.027079)
Digest::CRC16DNP#update       0.024253   0.000000   0.024253 (  0.024427)
Digest::CRC16Genibus#update   0.027237   0.000000   0.027237 (  0.027390)
Digest::CRC16Modbus#update    0.024376   0.000000   0.024376 (  0.024548)
Digest::CRC16QT#update        0.024361   0.000000   0.024361 (  0.024518)
Digest::CRC16USB#update       0.024142   0.000000   0.024142 (  0.024311)
Digest::CRC16X25#update       0.024098   0.000000   0.024098 (  0.024222)
Digest::CRC16XModem#update    0.026306   0.000000   0.026306 (  0.026502)
Digest::CRC16ZModem#update    0.026536   0.000000   0.026536 (  0.026688)
Digest::CRC24#update          0.029732   0.000000   0.029732 (  0.029902)
Digest::CRC32#update          0.024219   0.000000   0.024219 (  0.024391)
Digest::CRC32BZip2#update     0.026817   0.000000   0.026817 (  0.027044)
Digest::CRC32c#update         0.023681   0.000000   0.023681 (  0.023798)
Digest::CRC32Jam#update       0.024243   0.000000   0.024243 (  0.024419)
Digest::CRC32MPEG#update      0.026865   0.000000   0.026865 (  0.027020)
Digest::CRC32POSIX#update     0.026583   0.000000   0.026583 (  0.026748)
Digest::CRC32XFER#update      0.027423   0.000000   0.027423 (  0.027615)
Digest::CRC64#update          0.024150   0.000000   0.024150 (  0.024310)
Digest::CRC64Jones#update     0.024218   0.000000   0.024218 (  0.024363)
Digest::CRC64XZ#update        0.024124   0.000000   0.024124 (  0.024255)

Crystal

crystal-crc is a Crystal port of this library.

Thanks

Special thanks go out to the pycrc library which is able to generate C source-code for all of the CRC algorithms, including their CRC Tables.

License

See {file:LICENSE.txt} for license information.

digest-crc's People

Contributors

Stargazers

Watchers

digest-crc's Issues

Support for crc-32c

Awesome project. Would be just what I need if it supported crc-32c. Could you add it? pycrc can create its table.

Add better integration testing with Docker

Related to issue #22, we should perform integration testing by installing a built gem in various docker images, w/o gcc or w/o make.

Why there is a strict `rake '~>13.0'` requirement in gemspec?

Hello,

I would like to ask why there is a strict version requirement in gemspec of 0.6.1 version.

It currently breaks existing systems where the original Gemfile does not have a specific rake dependency defined.

Default rake that exists in ruby installations has version 12 for ruby versions 2.6 and a few earlier ones. With digest-crc gemspec change, calling rake directly has started to fail saying we should run with bundle exec rake.

In short, is it possible to have a wider range of supported rake versions there?

Thanks

Move CRC32C checksum to C extension

Hi @postmodern,

From my observation, the current implementation of CRC32C checksum is in pure Ruby. For some use cases that depends on this checksum mechanism, it looks like the performance is a huge problem. One example is that in ruby-kafka project, which is a Kafka client for ruby, the CRC32C checksum takes up to 20 - 30% of the processing time (zendesk/ruby-kafka#620). We use this algorithm heavily to ensure the integrity of messages sent to / received from Kafka.

Therefore, I would like to ask about moving this checksum algorithm to C extension. I suggest to just implement a naive porting, which still saves a huge amount of processing time, or use Google's implementation (https://github.com/google/crc32c) as a dependency.

If you agree, I will open an PR for this.

Could not find gem 'digest-crc (~> 0.6.1)' in locally installed gems

hey guys
can someone help me?
I'm trying to run fastlane with pipeline azure and then with fastlane call discord_notifier (https://github.com/nikolas-theodosis/fastlane-discord_notifier)
so
pipeline azure -> fastlane -> discord_notifier

if I only test fastlane with the discord notifier it works fine, but if the pipline triggers the fastlane (bundle exec fastlane dev) it doesn't work

erro on pipeline -> bundle exec fastlane dev:

yaml file pipeline:

running only fastlane and notifier:

digest-crc is instaled locally

Rename repo to digest-crc.rb

Now that there is postmodern/digest-crc.cr, this repo should be renamed to digest-crc.rb to avoid confusion. I'll need to also hunt down any direct links and update them.

Add benchmarks

Add a benchmark script to test the various CRC algorithms against a random block of data. This will help us establish a performance base-line. Bonus points if it can dump out the stats directly to CSV.

Does not install gem if installed rake is installed without documents

If I call

gem install --no-document rake
gem install digest-crc

Output:

ERROR:  Error installing digest-crc:
ERROR: Failed to build gem native extension.

current directory: /var/lib/gems/2.5.0/gems/digest-crc-0.6.3/ext/digest
/usr/bin/ruby2.5 -rrubygems /usr/share/rubygems-integration/all/gems/rake-12.3.1/exe/rake RUBYARCHDIR=/var/lib/gems/2.5.0/extensions/x86_64-linux/2.5.0/digest-crc-0.6.3 RUBYLIBDIR=/var/lib/gems/2.5.0/extensions/x86_64-linux/2.5.0/digest-crc-0.6.3
/usr/bin/ruby2.5: No such file or directory -- /usr/share/rubygems-integration/all/gems/rake-12.3.1/exe/rake (LoadError)

rake failed, exit code 1

Gem files will remain installed in /var/lib/gems/2.5.0/gems/digest-crc-0.6.3 for inspection.
Results logged to /var/lib/gems/2.5.0/extensions/x86_64-linux/2.5.0/digest-crc-0.6.3/gem_make.out

It is important that you NEVER had rake installed with documents, because it somehow remembers them somewhere even after gem uninstall rake.

Also I do not know, why exactly this happens. I thought I would not need rake to build this extension, if I have gcc and make installed on Debian.

Can you reproduce this behavior?

Build and release pre-compiled C extensions

Attempt to port nokogiri's GitHub Actions config that uses rake-compiler-dock to cross compile. This should reduce compilation issues and allow us to drop rake as a dependency.

The `Digest::CRC12_3GPP` class is not available

There is a C extension ext/digest/crc12_3gpp/crc12_3gpp_ext.c file, but the Ruby implementation (lib/digest/crc12_3gpp.rb file) is missing and consequently unavailable.
There is also no corresponding spec/crc12_3gpp_spec.rb file.

However, there is no description in README.md either, so there seems to be no problem even if it is not available.

macOS BigSur dir_chdir error when installing to ruby 2.7.1

Any one else see something similar? works in 2.5.3. Using rvm to install 2.7.1. All other gems seem to be just fine.

current directory: /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
["/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby", "-I/Users/ab/.rvm/rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems", "-rrubygems", "/Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake", "RUBYARCHDIR=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3", "RUBYLIBDIR=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3"]
/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby -S extconf.rb
checking for stdint.h... yes
checking for stddef.h... yes
creating extconf.h
creating Makefile
rake aborted!
Errno::EINTR: Interrupted system call @ dir_chdir - /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
/Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest/Rakefile:23:in `chdir'
/Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest/Rakefile:23:in `block (2 levels) in <top (required)>'
/Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake:27:in `<main>'
Tasks: TOP => default => crc16_genibus/crc16_genibus_ext.bundle => crc16_genibus/Makefile
(See full trace by running task with --trace)
ERROR:  Error installing digest-crc:
	ERROR: Failed to build gem native extension.

    current directory: /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby -I/Users/ab/.rvm/rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems -rrubygems /Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake RUBYARCHDIR\=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3 RUBYLIBDIR\=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3
Building has failed. See above output for more information on the failure.
rake failed, exit code 1

Gem files will remain installed in /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3 for inspection.
Results logged to /Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3/gem_make.out

new gem CRC16_qt?

Now that the 16bit qt algorithm is in, is there any chance we could have an updated gem? I have another gem which wants to use qt :-)

Would it make sense to have comparative benchmarks against other digests?

Would it make sense to have benchmarks for one or a few of the ruby built in digests as well, so that there is some ballpark comparison number?

Getting error when importing digest-crc as a Fastlane dependency

Fetching digest-crc 0.6.0 (was 0.4.1)
Installing digest-crc 0.6.0 (was 0.4.1) with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

This error only happens on 0.6.0, which was released today. Didn't happen with older versions.

Install fails on macOS 11 with Xcode 12.1

I got the following error trying to run bundle install on a Fastlane project:

An error occurred while installing digest-crc (0.6.1), and Bundler cannot
continue.
Make sure that `gem install digest-crc -v '0.6.1' --source
'https://rubygems.org/'` succeeds before bundling.
In Gemfile:
  xcode-install was resolved to 2.6.7, which depends on
    fastlane was resolved to 2.167.0, which depends on
      google-cloud-storage was resolved to 1.29.1, which depends on
        digest-crc

I tried to run sudo gem install digest-crc -v '0.6.1' --source 'https://rubygems.org/' as it says, and I got this error:

Building native extensions. This could take a while...
ERROR:  Error installing digest-crc:
        ERROR: Failed to build gem native extension.

    current directory: /Library/Ruby/Gems/2.6.0/gems/digest-crc-0.6.1/ext/digest
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby -rrubygems /Library/Ruby/Gems/2.6.0/gems/rake-13.0.1/exe/rake RUBYARCHDIR\=/Library/Ruby/Gems/2.6.0/extensions/universal-darwin-20/2.6.0/digest-crc-0.6.1 RUBYLIBDIR\=/Library/Ruby/Gems/2.6.0/extensions/universal-darwin-20/2.6.0/digest-crc-0.6.1
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby -S extconf.rb
checking for stdint.h... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/$(RUBY_BASE_NAME)
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:467:in `try_do': The compiler failed to generate an executable file. (RuntimeError)
You have to install development tools first.
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:585:in `block in try_compile'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:534:in `with_werror'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:585:in `try_compile'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:1109:in `block in have_header'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:959:in `block in checking_for'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:361:in `block (2 levels) in postpone'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:331:in `open'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:361:in `block in postpone'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:331:in `open'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:357:in `postpone'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:958:in `checking_for'
        from /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/mkmf.rb:1108:in `have_header'
        from extconf.rb:3:in `<main>'
rake aborted!
Command failed with status (1): [/System/Library/Frameworks/Ruby.framework/...]
/Library/Ruby/Gems/2.6.0/gems/digest-crc-0.6.1/ext/digest/Rakefile:32:in `block (3 levels) in <top (required)>'
/Library/Ruby/Gems/2.6.0/gems/digest-crc-0.6.1/ext/digest/Rakefile:31:in `chdir'
/Library/Ruby/Gems/2.6.0/gems/digest-crc-0.6.1/ext/digest/Rakefile:31:in `block (2 levels) in <top (required)>'
/Library/Ruby/Gems/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<main>'
Tasks: TOP => default => crc16_genibus/crc16_genibus_ext.bundle => crc16_genibus/Makefile
(See full trace by running task with --trace)

rake failed, exit code 1

Gem files will remain installed in /Library/Ruby/Gems/2.6.0/gems/digest-crc-0.6.1 for inspection.
Results logged to /Library/Ruby/Gems/2.6.0/extensions/universal-darwin-20/2.6.0/digest-crc-0.6.1/gem_make.out

This was working last week, and the only change on this machine is that it got updated (against my will) to macOS 11 Big Sur.

It has been a very long time since I've worked with Ruby (other than Fastlane), so I'm not really sure where to start looking to see what the problem is.

Benchmark results

I was just wondering what the performance is like:

  require 'benchmark/ips'
  require 'digest'
  
  TEST_STRING = 'This is a pretty short test string.'.freeze
  
  Benchmark.ips do |x|
    x.config(time: 3, warmup: 1)
  
    x.report('md5') do
      Digest::MD5.hexdigest(TEST_STRING)
    end
    x.report('sha1') do
      Digest::SHA1.hexdigest(TEST_STRING)
    end
    x.report('sha2') do
      Digest::SHA2.hexdigest(TEST_STRING)
    end
    x.report('crc16') do
      Digest::CRC16.hexdigest(TEST_STRING)
    end
    x.report('crc32') do
      Digest::CRC32.hexdigest(TEST_STRING)
    end
    x.report('crc64') do
      Digest::CRC64.hexdigest(TEST_STRING)
    end
  
    x.compare!
  end

Results from Ruby 2.4 on Fedora 27 with Ryzen 1700:

Warming up --------------------------------------
                 md5    68.196k i/100ms
                sha1    71.052k i/100ms
                sha2    41.683k i/100ms
               crc16    11.360k i/100ms
               crc32    10.111k i/100ms
               crc64     5.584k i/100ms
Calculating -------------------------------------
                 md5    879.284k (± 0.4%) i/s -      2.660M in   3.024840s
                sha1    956.697k (± 0.4%) i/s -      2.913M in   3.045031s
                sha2    512.701k (± 2.3%) i/s -      1.542M in   3.009928s
               crc16    120.649k (± 1.2%) i/s -    363.520k in   3.013477s
               crc32    106.223k (± 0.8%) i/s -    323.552k in   3.046145s
               crc64     57.542k (± 0.7%) i/s -    173.104k in   3.008440s

Comparison:
                sha1:   956696.7 i/s
                 md5:   879283.8 i/s - 1.09x  slower
                sha2:   512701.3 i/s - 1.87x  slower
               crc16:   120648.5 i/s - 7.93x  slower
               crc32:   106223.5 i/s - 9.01x  slower
               crc64:    57542.4 i/s - 16.63x  slower

Performance is kinda expected, what's a bit off from my expectations is how much CRC64 is slower on 64bit native CPU. It will only get better with Ruby MJIT I guess. Thanks for this wonderful gem!

CRC-16-QT is the same as CRC-16-X-25

CRC-16-QT defined by digest-crc has exactly the same characteristics as CRC-16-X-25.
I think it is better to redefine it as Digest::CRC16QT = Digest::CRC16X25 as an alias, or to abolish Digest::CRC16QT itself.

Doesn't install without C extensions.

The README says the C extensions are optional, and that if it fails for whatever reason, it will fallback to a pure-Ruby implementation. However, I couldn't get it to install without the C extensions. To reproduce run:

docker run -it ruby:2.7.1-alpine gem install digest-crc

That container doesn't have any build tools installed by default, so it can't install the C extensions. It fails to install with compiler errors.

Windows Install with space

I tried to install digest-crc (for fastlane) on Windows, but It seems that trying to install digest-crc when Ruby path contains a space character it breaks with an InvalidArgument error

Ruby installation path : D:\User Programs\Ruby30-x64

Error

current directory: D:/User Programs/Ruby30-x64/lib/ruby/gems/3.0.0/gems/digest-crc-0.6.4/ext/digest
\"D:/User\ Programs/Ruby30-x64/bin/ruby.exe\" -ID:/User\ Programs/Ruby30-x64/lib/ruby/3.0.0 -rrubygems D:/User\ Programs/Ruby30-x64/lib/ruby/gems/3.0.0/gems/rake-13.0.6/exe/rake RUBYARCHDIR\=D:/User\ Programs/Ruby30-x64/lib/ruby/gems/3.0.0/extensions/x64-mingw32/3.0.0/digest-crc-0.6.4 RUBYLIBDIR\=D:/User\ Programs/Ruby30-x64/lib/ruby/gems/3.0.0/extensions/x64-mingw32/3.0.0/digest-crc-0.6.4
rake failedInvalid argument - "D:/User Programs/Ruby30-x64/bin/ruby.exe"

It could also be a Rake error

Can't install without Xcode command line tools

The README states that:

(digest-crc) supports optional C extensions which increases performance by ~40x. If the C extensions cannot be compiled for whatever reason, digest-crc will automatically fallback to the pure-Ruby implementation.

However, the gem's native extensions can't be installed (on Catalina at least) without Xcode command line tools and no fallback seems to be provided:

TABLE override

Hello!
Is there a way to override the TABLE constant correctly?

dir_chdir error when installing on ruby 2.7.1 (works on 2.5.3)

Any one else see something similar?

current directory: /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
["/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby", "-I/Users/ab/.rvm/rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems", "-rrubygems", "/Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake", "RUBYARCHDIR=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3", "RUBYLIBDIR=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3"]
/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby -S extconf.rb
checking for stdint.h... yes
checking for stddef.h... yes
creating extconf.h
creating Makefile
rake aborted!
Errno::EINTR: Interrupted system call @ dir_chdir - /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
/Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest/Rakefile:23:in `chdir'
/Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest/Rakefile:23:in `block (2 levels) in <top (required)>'
/Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake:27:in `<main>'
Tasks: TOP => default => crc16_genibus/crc16_genibus_ext.bundle => crc16_genibus/Makefile
(See full trace by running task with --trace)
ERROR:  Error installing digest-crc:
	ERROR: Failed to build gem native extension.

    current directory: /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3/ext/digest
/Users/ab/.rvm/rubies/ruby-2.7.1/bin/ruby -I/Users/ab/.rvm/rubies/ruby-2.7.1/lib/ruby/2.7.0/rubygems -rrubygems /Users/ab/.rvm/gems/ruby-2.7.1/gems/rake-13.0.3/exe/rake RUBYARCHDIR\=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3 RUBYLIBDIR\=/Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3
Building has failed. See above output for more information on the failure.
rake failed, exit code 1

Gem files will remain installed in /Users/ab/.rvm/gems/ruby-2.7.1/gems/digest-crc-0.6.3 for inspection.
Results logged to /Users/ab/.rvm/gems/ruby-2.7.1/extensions/x86_64-darwin-20/2.7.0/digest-crc-0.6.3/gem_make.out

Require gem `rake` but no dependency

$ bundle install
....
    Installing digest-crc 0.6.0 (was 0.5.1) with native extensions
    Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
    
    current directory:
    /Users/guoqi/.bundle/vendor/ruby/2.5.0/gems/digest-crc-0.6.0/ext/digest
    rake
    RUBYARCHDIR=/Users/guoqi/.bundle/vendor/ruby/2.5.0/extensions/x86_64-darwin-18/2.5.0/digest-crc-0.6.0
    RUBYLIBDIR=/Users/guoqi/.bundle/vendor/ruby/2.5.0/extensions/x86_64-darwin-18/2.5.0/digest-crc-0.6.0
    /Users/guoqi/.rvm/rubies/ruby-2.5.0/lib/ruby/site_ruby/2.5.0/rubygems.rb:289:in
    `find_spec_for_exe': can't find gem rake (>= 0.a) with executable rake
    (Gem::GemNotFoundException)
    from
    /Users/guoqi/.rvm/rubies/ruby-2.5.0/lib/ruby/site_ruby/2.5.0/rubygems.rb:308:in
    `activate_bin_path'
    	from /Users/guoqi/.rvm/gems/ruby-2.5.0@global/bin/rake:23:in `<main>'
    	from /Users/guoqi/.rvm/gems/ruby-2.5.0/bin/ruby_executable_hooks:24:in `eval'
    	from /Users/guoqi/.rvm/gems/ruby-2.5.0/bin/ruby_executable_hooks:24:in `<main>'

There is not rake gem in my mac and my project. I think that should add a dependency in gemspec?

CRC64 "Jones" variant with initial condition

I have the need for a CRC 64 "Jones" variant, which is basically the CRC64 you have now with a modified table. Additionally, I need to be able to set the initial value of the CRC state prior to the first update call.

I have code that works for this (just the table changed after all) and have added a set(value) method which will set the crc state. However, as none of the other CRC calculators allow this set, I'm not sure if I should keep it class specific or make a PR that adds it to all CRCs.

Another option would be to change reset() to be reset(initial_value = @initial_value)

Which would you prefer?

postmodern / digest-crc Goto Github PK

digest-crc's Introduction

Digest CRC

Description

Features

Install

Examples

Benchmarks

Ruby 2.7.4 (pure Ruby)

Ruby 2.7.4 (C extensions)

Ruby 3.0.2 (pure Ruby)

Ruby 3.0.2 (C extensions)

JRuby 9.2.18.0 (pure Ruby)

TruffleRuby 21.2.0 (pure Ruby)

TruffleRuby 21.2.0 (C extensions)

Crystal

Thanks

License

digest-crc's People

Contributors

Stargazers

Watchers

Forkers

digest-crc's Issues

Recommend Projects

Recommend Topics

Recommend Org