chia-network / clvm_rs Goto Github PK

View Code? Open in Web Editor NEW

66.0 35.0 52.0 5.97 MB

Rust implementation of clvm

License: Apache License 2.0

Rust 79.29% Python 19.96% JavaScript 0.75%

chia chia-blockchain chialisp

clvm_rs's Introduction

Rust implementation of clvm.

The cargo workspace includes an rlib crate, for use with rust or other applications, and a python wheel.

The python wheel is in wheel. The npm package is in wasm.

TESTS

In order to run the unit tests, run:

cargo test

Fuzzing

The fuzzing infrastructure for clvm_rs uses cargo-fuzz.

Documentation for setting up fuzzing in rust can be found here.

To generate an initial corpus (for the run_program fuzzer), run:

cd tools
cargo run generate-fuzz-corpus

To get started, run:

cargo fuzz run fuzz_run_program --jobs=32 -- -rss_limit_mb=4096

But with whatever number of jobs works best for you.

If you find issues in clvm_rs please see the Bug Bounty program.

clvm_rs's People

Contributors

Stargazers

Watchers

clvm_rs's Issues

Maturin error

Just downloaded and tried to build something with instructions:

$ pip install maturin
$ maturin develop --release

Result:

💥 maturin failed
  Caused by: Failed to run rustc to get the host target
  Caused by: No such file or directory (os error 2)

Fails building 0.1.19 on FreeBSD 12.2

I am trying to upgrade to [email protected] to [email protected] on my FreeBSD jail, following the instructions here.
I get stuck trying to build [email protected]:

(venv) [root@chia ~/chia-blockchain/clvm_rs]# maturin develop --release
🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
🐍 Not using a specific python interpreter (With abi3, an interpreter is only required on windows)
   Compiling autocfg v1.0.1
   Compiling proc-macro2 v1.0.28
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.74
   Compiling proc-macro-hack v0.5.19
   Compiling radium v0.6.2
   Compiling pyo3-build-config v0.14.2
   Compiling tap v1.0.1
   Compiling typenum v1.13.0
   Compiling version_check v0.9.3
   Compiling once_cell v1.8.0
   Compiling funty v1.2.0
   Compiling cfg-if v1.0.0
   Compiling libc v0.2.99
   Compiling subtle v2.4.1
   Compiling rand_core v0.6.3
   Compiling scopeguard v1.1.0
   Compiling smallvec v1.6.1
   Compiling unindent v0.1.7
   Compiling byteorder v1.4.3
   Compiling cpufeatures v0.1.5
   Compiling opaque-debug v0.3.0
   Compiling lazy_static v1.4.0
   Compiling hex v0.4.3
   Compiling wyz v0.4.0
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling num-bigint v0.4.0
   Compiling generic-array v0.14.4
   Compiling instant v0.1.10
   Compiling lock_api v0.4.4
   Compiling bitvec v0.22.3
   Compiling quote v1.0.9
   Compiling parking_lot_core v0.8.3
   Compiling paste-impl v0.1.18
   Compiling pyo3 v0.14.2
   Compiling parking_lot v0.11.1
   Compiling digest v0.9.0
   Compiling block-buffer v0.9.0
   Compiling paste v0.1.18
   Compiling sha2 v0.9.5
   Compiling ff v0.10.1
   Compiling group v0.10.0
   Compiling pairing v0.20.0
   Compiling bls12_381 v0.5.0
   Compiling pyo3-macros-backend v0.14.2
   Compiling indoc-impl v0.3.6
   Compiling indoc v0.3.6
   Compiling pyo3-macros v0.14.2
   Compiling clvm v0.1.19 (/root/chia-blockchain/clvm_rs)
    Finished release [optimized] target(s) in 22.76s
💥 maturin failed
  Caused by: Cargo didn't build a cdylib. Did you miss crate-type = ["cdylib"] in the lib section of your Cargo.toml?
(venv) [root@chia ~/chia-blockchain/clvm_rs]# maturin --version
maturin 0.12.11
(venv) [root@chia ~/chia-blockchain/clvm_rs]# git branch
* (HEAD detached at 0.1.19)
  main

Fails to build on Alpine Linux with "don't know how to configure OpenSSL for x86_64-alpine-linux-musl"

I tried to install chia on Alpine Linux (3.14.2-x86_64) but it ultimately fails for me installing clvm_rs:

farmer $ git clone https://github.com/Chia-Network/chia-blockchain.git -b latest
Cloning into 'chia-blockchain'...
remote: Enumerating objects: 52845, done.
remote: Counting objects: 100% (3188/3188), done.
remote: Compressing objects: 100% (1437/1437), done.
remote: Total 52845 (delta 2251), reused 2479 (delta 1735), pack-reused 49657
Receiving objects: 100% (52845/52845), 23.69 MiB | 20.96 MiB/s, done.
Resolving deltas: 100% (40594/40594), done.

[~/chia-blockchain]
farmer $ sh install.sh
apt-get: not found
Submodule 'mozilla-ca' (https://github.com/Chia-Network/mozilla-ca.git) registered for path 'mozilla-ca'
Cloning into '/home/farmer/chia-blockchain/mozilla-ca'...
Submodule path 'mozilla-ca': checked out 'b1b808ab930004fc6b4afc4b248dee0a136f3f00'
pacman: not found
yum: not found
yum: not found
Python version is 3.9
Requirement already satisfied: pip in ./venv/lib/python3.9/site-packages (21.1.1)
Collecting pip
  Using cached pip-21.2.4-py3-none-any.whl (1.6 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.1
    Uninstalling pip-21.1.1:
      Successfully uninstalled pip-21.1.1
Successfully installed pip-21.2.4
Collecting wheel
  Using cached wheel-0.37.0-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.37.0
Looking in indexes: https://pypi.org/simple, https://pypi.chia.net/simple/
Collecting miniupnpc==2.2.2
  Using cached miniupnpc-2.2.2-cp39-cp39-linux_x86_64.whl
Installing collected packages: miniupnpc
Successfully installed miniupnpc-2.2.2
Looking in indexes: https://pypi.org/simple, https://pypi.chia.net/simple/
Obtaining file:///home/farmer/chia-blockchain
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting keyrings.cryptfile==1.3.4
  Using cached keyrings.cryptfile-1.3.4.tar.gz (17 kB)
Collecting keyring==23.0.1
  Using cached keyring-23.0.1-py3-none-any.whl (33 kB)
Collecting bitstring==3.1.9
  Using cached bitstring-3.1.9-py3-none-any.whl (38 kB)
Collecting colorlog==5.0.1
  Using cached colorlog-5.0.1-py2.py3-none-any.whl (10 kB)
Collecting sortedcontainers==2.3.0
  Using cached sortedcontainers-2.3.0-py2.py3-none-any.whl (29 kB)
Collecting clvm-tools==0.4.3
  Using cached clvm_tools-0.4.3-py3-none-any.whl (36 kB)
Collecting cryptography==3.4.7
  Using cached cryptography-3.4.7.tar.gz (546 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting aiohttp==3.7.4
  Using cached aiohttp-3.7.4.tar.gz (1.1 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting blspy==1.0.6
  Using cached blspy-1.0.6.tar.gz (284 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting click==7.1.2
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting setproctitle==1.2.2
  Using cached setproctitle-1.2.2.tar.gz (23 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting watchdog==2.1.3
  Using cached watchdog-2.1.3.tar.gz (109 kB)
Collecting chiabip158==1.0
  Using cached chiabip158-1.0.tar.gz (117 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting aiosqlite==0.17.0
  Using cached aiosqlite-0.17.0-py3-none-any.whl (15 kB)
Collecting fasteners==0.16.3
  Using cached fasteners-0.16.3-py2.py3-none-any.whl (28 kB)
Collecting PyYAML==5.4.1
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting chiapos==1.0.4
  Using cached chiapos-1.0.4.tar.gz (1.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting websockets==8.1.0
  Using cached websockets-8.1.tar.gz (58 kB)
Collecting dnspython==2.1.0
  Using cached dnspython-2.1.0-py3-none-any.whl (241 kB)
Collecting clvm==0.9.7
  Using cached clvm-0.9.7-py3-none-any.whl (19 kB)
Collecting colorama==0.4.4
  Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
ERROR: Could not find a version that satisfies the requirement clvm-rs==0.1.11 (from chia-blockchain) (from versions: 0.1.0, 0.1.1, 0.1.2, 0.1.4, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.1.10)
ERROR: No matching distribution found for clvm-rs==0.1.11

So I tried to manually install clvm_rs into the venv:

(venv) [~]
farmer $ git clone https://github.com/Chia-Network/clvm_rs.git -b 0.1.11
Cloning into 'clvm_rs'...
remote: Enumerating objects: 3087, done.
remote: Counting objects: 100% (1173/1173), done.
remote: Compressing objects: 100% (504/504), done.
remote: Total 3087 (delta 805), reused 925 (delta 652), pack-reused 1914
Receiving objects: 100% (3087/3087), 4.31 MiB | 8.30 MiB/s, done.
Resolving deltas: 100% (2101/2101), done.
Note: switching to 'e6985832959aa91ef16496090ee77232e30f39de'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

(venv) [~/clvm_rs]
farmer $ maturin develop --release

🔗 Found pyo3 bindings
🐍 Found CPython 3.9 at python
📡 Using build options bindings from pyproject.toml
   Compiling autocfg v1.0.1
   Compiling proc-macro2 v1.0.28
   Compiling unicode-xid v0.2.2
   Compiling libc v0.2.99
   Compiling proc-macro-hack v0.5.19
   Compiling pyo3-build-config v0.14.2
   Compiling syn v1.0.74
   Compiling radium v0.6.2
   Compiling cc v1.0.69
   Compiling cfg-if v1.0.0
   Compiling once_cell v1.8.0
   Compiling tap v1.0.1
   Compiling typenum v1.13.0
   Compiling version_check v0.9.3
   Compiling pkg-config v0.3.19
   Compiling funty v1.2.0
   Compiling subtle v2.4.1
   Compiling rand_core v0.6.3
   Compiling smallvec v1.6.1
   Compiling scopeguard v1.1.0
   Compiling unindent v0.1.7
   Compiling byteorder v1.4.3
   Compiling openssl v0.10.35
   Compiling foreign-types-shared v0.1.1
   Compiling opaque-debug v0.3.0
   Compiling bitflags v1.3.1
   Compiling cpufeatures v0.1.5
   Compiling hex v0.4.3
   Compiling lazy_static v1.4.0
   Compiling num-traits v0.2.14
   Compiling num-integer v0.1.44
   Compiling num-bigint v0.4.0
   Compiling instant v0.1.10
   Compiling wyz v0.4.0
   Compiling openssl-src v111.15.0+1.1.1k
   Compiling generic-array v0.14.4
   Compiling lock_api v0.4.4
   Compiling foreign-types v0.3.2
   Compiling openssl-sys v0.9.65
   Compiling quote v1.0.9
   Compiling bitvec v0.22.3
   Compiling parking_lot_core v0.8.3
   Compiling paste-impl v0.1.18
error: failed to run custom build command for `openssl-sys v0.9.65`

Caused by:
  process didn't exit successfully: `/home/farmer/clvm_rs/target/release/build/openssl-sys-19a12d6e2bc14fbe/build-script-main` (exit code: 101)
  --- stdout
  cargo:rustc-cfg=const_fn
  cargo:rerun-if-env-changed=X86_64_ALPINE_LINUX_MUSL_OPENSSL_NO_VENDOR
  X86_64_ALPINE_LINUX_MUSL_OPENSSL_NO_VENDOR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_VENDOR
  OPENSSL_NO_VENDOR unset

  --- stderr
  thread 'main' panicked at 'don't know how to configure OpenSSL for x86_64-alpine-linux-musl', /home/farmer/.cargo/registry/src/github.com-1ecc6299db9ec823/openssl-src-111.15.0+1.1.1k/src/lib.rs:275:18
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: build failed
💥 maturin failed
  Caused by: Failed to build a native library through cargo
  Caused by: Failed to build a native library through cargo
  Caused by: Cargo build finished with "exit code: 101": `cargo rustc --message-format json --manifest-path Cargo.toml --release --lib --`
(venv) 19:03:02 [~/clvm_rs]
farmer $

Any suggestions?

Cheers,
Bjoern

run_program for `op_point_add` is slow. Even slower than Python's clvm

It is really strange, but I got the same results even I tried benchmark multiple times.

I am releasing production-ready version of clvm_tools-js and I am currently benchmarking to compare performance with:

clvm_tools(js) with clvm(js)
clvm_tools(python) with clvm(python)
clvm_tools(python) with clvm(rust)

In summary, clvm in Rust implementation works great in run_program.
It is around 30-100 times faster than other implementation in most cases.
(But be aware, to_sexp_f benchmark is 5-10 times slower than other implementation. This is another story.)

When I got the benchmark result, I found that op_point_add is slower than Python implementation of clvm.
This is really strange. I haven't started investigation yet, but I'd like to share the current data with you.

The benchmark was done using benchmark files generated by costs/generate-benchmark.py in clvm_tools.
The benchmark parameters are:

value size: 4bytes(32bit)
operations count: 200

I modified generate-benchmrk.py a bit to produce benchmark files with such parameters.
I thought value_size = 4bytes and operations count = 200 seems to be typical among most of chialisp programs.

run_program

to_sexp_f

If you are interested, here is the result of to_sexp_f benchmark.

FreeBSD: No matching distribution found for clvm-rs==0.1.14

Pypi shows version 0.1.14, however there is only prebuilt wheels, no source tar.gz linked which I assume might be necessary for building on FreeBSD. 0.1.10 is the last valid clvm-rs that my pip shows:

$ pip install clvm-rs==0.1.14

ERROR: Could not find a version that satisfies the requirement clvm-rs==0.1.14 (
from versions: 0.1.0, 0.1.1, 0.1.2, 0.1.4, 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.1.10)
ERROR: No matching distribution found for clvm-rs==0.1.14

no setup.py file is included

For systems (like FreeBSD) that depend on python ports having setup.py files, it makes adding it as a user select-able port more difficult.

Please enhance this project by adding a setup.py and including typical steps like setup.py test to perform tests.

sdist too large

When maturin builds a source distribution, it includes the venv directory, which means the .tar.gz includes a bunch of junk it doesn't need. Maybe we put the venv a level above, or in a temp directory so it's not dragged in by maturin.

wasm and clvm_tools-js integration

@richardkiss
I've just confirmed that the wasm of clvm_rs can be successfully integrated into clvm_tools-js!
clvm_tools-js#clvm_rs

# Terminal
git clone https://github.com/Chia-Mine/clvm_tools-js
cd clvm_tools-js
git checkout clvm_rs
npm install
npm run build
node .dist/npm/bin/cli.js brun --experiment-backend rust --time "(+ 1 (q . 3))" "2"

However, I'm aware that current wasm API is not fully aligned with Python's clvm_rs API.
At wasm, run_clvm Rust function is exposed to wasm, which accepts only clvm program and arguments, and returns only program result without value of cost.

#[wasm_bindgen]
pub fn run_clvm(program: &[u8], args: &[u8]) -> Vec<u8> {
// ...

On the other hand, Python's API accepts additional parameters and also return cost.

#[allow(clippy::too_many_arguments)]
#[pyfunction]
pub fn deserialize_and_run_program2(
    py: Python,
    program: &[u8],
    args: &[u8],
    quote_kw: u8,
    apply_kw: u8,
    opcode_lookup_by_name: HashMap<String, Vec<u8>>,
    max_cost: Cost,
    flags: u32,
) -> PyResult<(Cost, LazyNode)> {
// ...

At a quick glance, I think arguments quote_kw, apply_kw, opcode_lookup_by_name in deserialize_and_run_program2 are not necessary for wasm's run_clvm because those arguments seem always the same value and no user interference.
Question is whether the above assuption is correct or not.

As for max_cost and flags, I'll find the good way/appropriate type to connect to JavaScript. (JavaScript built-in number cannot represent unsigned int 64bit as is. So u64 of Rust should be represented in uint8array in JavaScript or something)