opendp / smartnoise-core Goto Github PK

Documentation for initial release

see: https://docs.google.com/document/d/16LFjllHI6jAtgURweasJ733X4l-XI8Fq3Ryy0b98Y0w/edit

Either Rust directly, or more likely from the gRPC return, we need to convert the release values and analysis graph into a JSON output that validates against the schema.

depends on #14

1. gRPC Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

Repro steps:

with yarrow.Analysis() as analysis:
PUMS = yarrow.Dataset('PUMS', test_csv_path)
mean_income = yarrow.dp_mean(
PUMS[('income', float)],
epsilon=4.0,
minimum=0,
maximum=100,
num_records=1000
)

vals = []
for x in range(100):
    analysis.release()
    vals.append(analysis.release_proto.values[6].values['data'].f64.data[0])

Expected:

array of noisy answers

Result:

Sometimes works, sometimes fails with "Exception has occurred: DecodeError
Error parsing message "Error Parsing Message".

Error is triggered from line 56 of wrapper.py:
return release_pb2.Release.FromString(serialized_response)

Setting to repeat 1000 or 10000 times fails consistently. The same error is occurring in other places where release is accessed.

Background:

For the above repro length of serialized_response on all successful calls in line 56 from wrapper.py is 30. The length of failing calls is 24. Below is an example of 2 passing serialized_response strings and 1 failing string, which is 6 characters smaller.

b'\n\x1c\x08\x06\x12\x18\n\x16\n\x04data\x12\x0e\x08\x03*\n\n\x08\xe9\xe0\xe3\xaf\xce\rV@'
b'\n\x1c\x08\x06\x12\x18\n\x16\n\x04data\x12\x0e\x08\x03*\n\n\x08\xfc\xf4\x17\xfe\x19\rV@'
b'\n\x1c\x08\x06\x12\x18\n\x16\n\x04data\x12\x0e\x08\x03*\n\n\x08\xbb\xae\xc0\x14'

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

Switch to OpenAPI

1. Rewrite definitions
2. Adjust python and R bindings
3. serialization/deserialization
   1. rust: serde, openapi
   2. python: update ctypes

Document a How-To Vignette for linking from the Isthmus site.

Google Doc working draft: https://docs.google.com/document/d/1K5jOrWzi2iUBGxqdV_9ksTyArderlL9Zg3e9sld3aXA/edit#heading=h.rvqj72azac4w

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

1. Guideline API spec: https://github.com/privacytoolsproject/yarrow/wiki
2. validate_analysis: add robust sanity checks for well-formedness
   1. Every path to a sink is privatized
   2. Transforms are conformable? (IE df1.a + df2.a when len(df1) != len(df2))
   3. Compatible with definition spec
   4. Common privacy definition across privatizers
   5. Is a DAG
   6. Runtime supports all components
3. compute_privacy: response should be per-dataset and definition-specific
4. from_accuracy, to_accuracy

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

(Define types to be used within DP. Start with basic types.)

Document procedures for new contributors to add to the library.

1. gRPC Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

• Site naming, domain requisition

Related:

• Overview #56
• Call to Action #57
• FAQs #58
• https://github.com/orgs/privacytoolsproject/projects/2?card_filter_query=label%3Aminisite+milestone%3A%22march+mvp%22

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

1. Protobuff Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

• Part of the PSI Core Library

Primary Steps

• Create ProtoBuff/OpenAPI Definition
• Write Rust Runtime
  • including tests
• Write Rust Validator
  • including tests

Secondary Steps

• Python bindings
  • including tests
• R bindings
• SQL bindings

Document a How-To Vignette for linking from the Isthmus site

Document a How-To Vignette for linking from the Isthmus site.

All of the metadata for generating the components.proto is stored inside prototypes/components/. Once we have an auto-build script for components.proto, we won't need to maintain components.proto to be consistent with the json files.

It makes sense to write this right before the prost invocation in the validator's build.rs.

1. gRPC Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

Initial setup of testing library.

Google Doc working draft: https://docs.google.com/document/d/1lS0kjd-WhIZTAZwzNgUHPLvDjptnl1c75JOElxpVLlw/edit#heading=h.qhn67xnwq1k9

Document a How-To Vignette for linking from the Isthmus site.

Components.

Check off as docstring/descriptions are added to Rust

• 1. Materialize
• 2. Index
• 3. Stack
• 4. Literal

• 5. Cast
• 6. Bin

• 7. Impute
• 8. Clamp
• 9. Row-wise Min/Max
• 10. Resize

• 11. DPMean
• 12. DPCount
• 13. DPHistogram
• 14. DPCovariance

• 15. Add
• 16. Multiply
• 17. Matmul
• 18. Mod

• 19. Equal
• 20. LessThan
• 21. LogicalOr
• 21. Negate

• 23. LaplaceMech.
• 24. GaussianMech.
• 25. GeometricMech.

What's the best approach to execute the analysis against in-memory data? We'll need in-memory to pass in the SQL intermediate results from burdock, and bypassing file access will speed up the stochastic tester independent of the SQL pieces.

• API reference
• List of algorithms and citations
• Build instructions
• Test instructions
• Architecture

• Execution Environment
• Unbiased Privacy
• Other algorithms
• Report Store/Budget Store
• Private Synopsis

current draft: https://docs.google.com/document/d/1HnwrU4f1sFsPWTUgtyVk3kEPdvmKnN7cXKjp-KLSPtk/edit

Document a How-To Vignette for linking from the Isthmus site.

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

Concepts to explain

• May be short summaries with links to papers

• Re-identification attacks

• Global versus Local model

• Privacy Budget

• Using noisy data

1. Protobuff Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

For 3/1 release

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

https://docs.google.com/document/d/16TeIgjyf1WgIq0fJPNjYTPfE811SmkOj0fgZJEzwR4A/edit

• Protobuf doesn't need to be pegged to exactly 3.9.1 anymore, considering we don't have a conan dependency for C++. This can loosen us up to using precompiled protoc available on homebrew and snap, and get rid of the complicated proto install instructions.
• Rust may not need to be on the nightly channel anymore

When I follow the build steps on a new install of Ubuntu, the builds for runtime-rust and validator-rust fail in the cbindgen custom build step. I can get the build to finish and unit tests to run by commenting out the cbindgen call in both build.rs files, because the api.h in the repo is up-to-date.

The cbindgen step succeed on my previous Ubuntu laptop and my MacBook laptop, which I previously configured with all of the C++ dependencies, Conan, etc.

Here is the full backtrace for validator-rust (runtime is essentially the same):

Compiling yarrow_validator v0.1.0 (/home/joshua/yarrow/validator-rust)
error: failed to run custom build command for yarrow_validator v0.1.0 (/home/joshua/yarrow/validator-rust)

Caused by:
process didn't exit successfully: /home/joshua/yarrow/validator-rust/target/release/build/yarrow_validator-4940303593ef9cd0/build-script-build (exit code: 101)
--- stderr
thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libcore/slice/mod.rs:2791:10
stack backtrace:
0: 0x559857ed7d98 - backtrace::backtrace::libunwind::trace::h946e3de0c90abfd7
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
1: 0x559857ed7d98 - backtrace::backtrace::trace_unsynchronized::h07606d21d603c43a
at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
2: 0x559857ed7d98 - std::sys_common::backtrace::_print_fmt::h962d19ba24adc71d
at src/libstd/sys_common/backtrace.rs:77
3: 0x559857ed7d98 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h19476a0c55a309e1
at src/libstd/sys_common/backtrace.rs:59
4: 0x559857efe64c - core::fmt::write::hace03fabe0232084
at src/libcore/fmt/mod.rs:1052
5: 0x559857ed4797 - std::io::Write::write_fmt::h040dd382992db3a0
at src/libstd/io/mod.rs:1428
6: 0x559857edabe5 - std::sys_common::backtrace::_print::h31d41e7ebc455053
at src/libstd/sys_common/backtrace.rs:62
7: 0x559857edabe5 - std::sys_common::backtrace::print::hb9a7709d20b4b6bd
at src/libstd/sys_common/backtrace.rs:49
8: 0x559857edabe5 - std::panicking::default_hook::{{closure}}::h822a6c70999551c6
at src/libstd/panicking.rs:204
9: 0x559857eda926 - std::panicking::default_hook::heec975af1173ff14
at src/libstd/panicking.rs:224
10: 0x559857edb2c5 - std::panicking::rust_panic_with_hook::h5af769b06ceaa2dc
at src/libstd/panicking.rs:474
11: 0x559857edaddb - rust_begin_unwind
at src/libstd/panicking.rs:378
12: 0x559857efce81 - core::panicking::panic_fmt::hfd5708e1fe45b20b
at src/libcore/panicking.rs:85
13: 0x559857efce45 - core::panicking::panic_bounds_check::h3e8ca98fc7d37627
at src/libcore/panicking.rs:63
14: 0x559857c55d09 - <usize as core::slice::SliceIndex<[T]>>::index::had63db15143bff45
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libcore/slice/mod.rs:2791
15: 0x559857c55d09 - core::slice::<impl core::ops::index::Index for [T]>::index::h6578968298b56424
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libcore/slice/mod.rs:2656
16: 0x559857c55d09 - <alloc::vec::Vec as core::ops::index::Index>::index::h9e6176a6c9b0241c
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/liballoc/vec.rs:1885
17: 0x559857c55d09 - cbindgen::bindgen::cargo::cargo::parse_dep_string::h0bfec90bee78ff23
at /home/joshua/.cargo/registry/src/github.com-1ecc6299db9ec823/cbindgen-0.9.1/src/bindgen/cargo/cargo.rs:19
18: 0x559857cfe031 - cbindgen::bindgen::cargo::cargo::Cargo::dependencies::{{closure}}::h868f220e9053e67a
at /home/joshua/.cargo/registry/src/github.com-1ecc6299db9ec823/cbindgen-0.9.1/src/bindgen/cargo/cargo.rs:119
19: 0x559857cfe031 - core::iter::adapters::map_fold::{{closure}}::h5902d8d29d965520
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libcore/iter/adapters/mod.rs:772
20: 0x559857cfe031 - core::iter::traits::iterator::Iterator::fold::ok::{{closure}}::h0e73e72466d429ce
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libcore/iter/traits/iterator.rs:2000
21: 0x559857cfe031 - core::iter::traits::iterator::Iterator::try_fold::h7a4edaf4a4385bfe
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libcore/iter/traits/iterator.rs:1876
22: 0x559857cfe031 - core::iter::traits::iterator::Iterator::fold::h86d21a3721f2da31
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libcore/iter/traits/iterator.rs:2003
23: 0x559857cfe031 - <core::iter::adapters::Map<I,F> as core::iter::traits::iterator::Iterator>::fold::h1678eacc7d133094
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libcore/iter/adapters/mod.rs:812
24: 0x559857c7d3b4 - core::iter::traits::iterator::Iterator::for_each::h43a6585266ce6acc
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libcore/iter/traits/iterator.rs:659
25: 0x559857c7d3b4 - <alloc::vec::Vec as alloc::vec::SpecExtend<T,I>>::spec_extend::h0c2b333dfd1489c6
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/liballoc/vec.rs:2056
26: 0x559857c7d3b4 - <alloc::vec::Vec as alloc::vec::SpecExtend<T,I>>::from_iter::h99caab1f43bc3030
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/liballoc/vec.rs:2036
27: 0x559857c57396 - <alloc::vec::Vec as core::iter::traits::collect::FromIterator>::from_iter::hcc8f8f2a00b4d8bb
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/liballoc/vec.rs:1921
28: 0x559857c57396 - core::iter::traits::iterator::Iterator::collect::h68117e630341fa93
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libcore/iter/traits/iterator.rs:1658
29: 0x559857c57396 - cbindgen::bindgen::cargo::cargo::Cargo::dependencies::he23697dc9c6c5234
at /home/joshua/.cargo/registry/src/github.com-1ecc6299db9ec823/cbindgen-0.9.1/src/bindgen/cargo/cargo.rs:115
30: 0x559857d3f039 - cbindgen::bindgen::parser::Parser::parse_crate::h216589f3697a9f9f
at /home/joshua/.cargo/registry/src/github.com-1ecc6299db9ec823/cbindgen-0.9.1/src/bindgen/parser.rs:159
31: 0x559857d3e700 - cbindgen::bindgen::parser::parse_lib::h6dd33a935187c74a
at /home/joshua/.cargo/registry/src/github.com-1ecc6299db9ec823/cbindgen-0.9.1/src/bindgen/parser.rs:88
32: 0x559857c44230 - cbindgen::bindgen::builder::Builder::generate::hbc0e3b4b1c19ff35
at /home/joshua/.cargo/registry/src/github.com-1ecc6299db9ec823/cbindgen-0.9.1/src/bindgen/builder.rs:322
33: 0x559857bdaf3d - cbindgen::generate_with_config::h167e5409a3b9d478
at /home/joshua/.cargo/registry/src/github.com-1ecc6299db9ec823/cbindgen-0.9.1/src/lib.rs:37
34: 0x559857bdaf3d - build_script_build::main::hd3411c5ac314e02a
at /home/joshua/yarrow/validator-rust/build.rs:19
35: 0x559857bf7393 - std::rt::lang_start::{{closure}}::h0e88d66d3bcafed9
at /rustc/8417d68de5e063426ab6bb7f383df6117d1beeed/src/libstd/rt.rs:67
36: 0x559857edad03 - std::rt::lang_start_internal::{{closure}}::he3f8b3563423a3bd
at src/libstd/rt.rs:52
37: 0x559857edad03 - std::panicking::try::do_call::h688c5e52f98f353d
at src/libstd/panicking.rs:303
38: 0x559857ee2927 - __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:86
39: 0x559857edb769 - std::panicking::try::he72700c2f9c2c35d
at src/libstd/panicking.rs:281
40: 0x559857edb769 - std::panic::catch_unwind::h71f676a40de6e61e
at src/libstd/panic.rs:394
41: 0x559857edb769 - std::rt::lang_start_internal::hf18bc456048ccdad
at src/libstd/rt.rs:51
42: 0x559857bdbd58 - main
43: 0x7f6ed1705b97 - __libc_start_main
44: 0x559857b59eba - _start
45: 0x0 -

1. OpenAPI Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

Check off as components are added to python tests. For initial tests, they may be indirectly called in tests which validate plans/analyses

• 1. Materialize
• 2. Index
• 3. Stack
• 4. Literal

• 5. Cast
• 6. Bin

• 7. Impute
• 8. Clamp
• 9. Row-wise Min/Max
• 10. Resize

• 11. DPMean
• 12. DPCount
• 13. DPHistogram
• 14. DPCovariance

• 15. Add
• 16. Multiply
• 17. Matmul
• 18. Mod

• 19. Equal
• 20. LessThan
• 21. LogicalOr
• 21. Negate

• 23. LaplaceMech.
• 24. GaussianMech.
• 25. GeometricMech.

1. ** gRPC Definition**
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

1. Protobuff Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs

1. Protobuff Definition
   • Write definition
   • Write comments/docs
2. Rust Runtime
   • Write runtime
   • Implement tests
   • Write comments/docs
3. Rust Validator
   • Write validator
   • Implement tests
   • Write comments/docs
4. Python bindings
   • Create bindings
   • Implement tests
   • Write docs
5. R bindings
   • Create bindings
   • Implement tests
   • Write docs