ferrilab / bitvec Goto Github PK
View Code? Open in Web Editor NEWA crate for managing memory bit by bit
Home Page: https://myrrlyn.net/crates/bitvec
License: MIT License
A crate for managing memory bit by bit
Home Page: https://myrrlyn.net/crates/bitvec
License: MIT License
There is no reverse function on bitvec as there is on std::vec. By a reverse function I mean a operation that actually reverse the order in witch the bit are store in memory. I am currently building a (bad) Huffman coding project using this library and I rand into this issue.
it his possible that there is actuelle technical raison why this is not fisable
Thank in advance
Hi! I was using this crate, and I found a need to convert BitSlice
s back into integers of various lengths. This is different from the already-existing as_slice
-- that gives direct access to the underlying storage, which might include bits on either side of the BitSlice
if the BitSlice
starts or ends mid-byte.
What I would like is the ability to convert from a BitSlice
to a regular Rust integer, with bitvec
performing whatever shifts need to happen behind the scenes.
I could put together a patch that would implement TryInto<u8>
, TryInto<u16>
, etc. for BitSlice
types. Is that something that would be welcome?
Using bitvec 0.10, counting 1 bits in a bit slice within a single storage unit returns the total number of bits within the slice, not the number of 1 bits. Interestingly, it does work across when operating across byte boundaries. The following program:
fn main()
{
let bits: [u8; 2] = [0b00111100, 0b11001100];
let bv: bitvec::BitVec = (&bits as &[u8]).into();
// Here, count_ones() fails
let bs = &bv[4..8];
println!("{} {}", bs, bs.count_ones());
// Here, it works
let bs = &bv[6..10];
println!("{} {}", bs, bs.count_ones());
}
returns the following output
[1100] 4
[00, 11] 2
As mentioned in #16, the default numeric arithmetic implementations may not be suitable for all use cases of bit sequences. Furthermore, the core crate does not include suitability as a numeric type in its design goals – the minimum product is a sequence of raw bits, with no further semantic information held in the sequence itself.
This issue proposes moving all numeric behavior out of the core crate, into a separate sibling crate (bitvec_arith
? bitvec_num
?) with wrapper types over BitSlice
and BitVec
that perform numeric arithmetic using the underlying sequences as the n-ary number storage.
Our project depends on bitvec, but now our compile fails because the version is not valid
Thanks for this library!
I have noticed an interesting performance issue with the initialization of BitVec
. The following benchmarks produce an appalling result:
extern crate test;
use test::Bencher;
use bitvec::vec::BitVec;
#[bench]
fn bench_bitvec_init(b: &mut Bencher) {
b.iter(|| bitvec![0; 16 * 16 * 9]);
}
#[bench]
fn bench_vec_init(b: &mut Bencher) {
b.iter(|| vec![0u8; 16 * 16 * 9 / 8]);
}
Results:
running 2 tests
test tests::bench_vec_init ... bench: 47 ns/iter (+/- 11)
test tests::bench_bitvec_init ... bench: 63,721 ns/iter (+/- 3,576)
It would be nice to be able to use this with alloc but not std.
I am not sure if this an issue or something due to a lack of Rust knowledge, but there seems to be a lack of way for efficient bitwise comparisons. When trying to do something like xor'ing two BitVecs, ownership is moved for the XOR:
let bv2 = bv::bitvec![0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0];
let t = bv1 ^ bv2;
assert_eq!(t.count_ones(), 3);
let t2 = bv2 ^ bv1;
assert_eq!(t2.count_ones(), 3);
This fails due to a moved bv1 and bv2. I think this is due to the bitxor trait.
From what I can tell the only way around this is to clone bv1 and bv2 for every operation, which can have performance implications when dealing with a large number of large bitvectors.
Looking at creating a slice like the following doesn't work due to a lack of xor for bitslice.
let bv1 = bv::bitvec![0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
let bv2 = bv::bitvec![0, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0];
let b1: &bv::BitSlice = &bv1[..];
let b2: &bv::BitSlice = &bv2[..];
let t = b1 ^ b2;
assert_eq!(t.count_ones(), 3);
Am I missing something obvious or will a borrowed xor need to be hand rolled?
Thanks!
Consider:
use bitvec::prelude::*;
let v = bitvec![1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0];
assert_eq!(v.len(), 14);
let s = &v.as_bitslice()[(v.len() - 8)..];
assert_eq!(s.len(), 8);
for i in 0..8 {
assert_eq!(s[i], false);
}
let bitcount = s.count_ones();
assert_eq!(0, bitcount);
let s = s.change_cursor::<LittleEndian>();
assert_eq!(s.len(), 8);
let bitcount = s.count_ones();
assert_eq!(0, bitcount);
for i in 0..8 {
assert_eq!(s[i], false);
}
(EDIT: FYI, this crashes at the second bitcount assertion)
I would expect s
to still be eight zeroes, but it gets bits outside the slice. That made me wonder about the semantics of the cursor: it iterates over bytes (u8
's) in some direction, but the notion of bytes is quite lost for BitSlice
's.
In any case: I have a situation where I want to reverse every byte in a BitSlice that has size n*8...
There is a node in the docs:
/// Panics if the number of bits in the vector overflows
/// `BitPtr::<T>::MAX_ELTS`.
However I could not find MAX_ELTS
through the docs
Hi, I've been using this crate for serial protocols and it worked great! Today I ran into an issue when trying to convert a misaligned BitSlice
to a Vec<u8>
:
use bitvec::bitvec;
let bv = bitvec![
0, 0, 0, 0, 0, 0, 0, 1,
1, 0, 0, 0, 0, 0, 0, 1,
];
let slice = &bv[4..12];
println!("bv[4..12] = {:?}", slice);
let bv2 = slice.to_owned();
println!("bv[4..12].to_owned() = {:?}", bv2);
let bs: Vec<u8> = bv2.into();
println!("bv[4..12].to_owned().into() = {:?}", bs);
which gives
bv[4..12] = BitSlice<BigEndian, u8> [0001, 1000]
bv[4..12].to_owned() = BitVec<BigEndian, u8> [0001, 1000]
bv[4..12].to_owned().into() = [1, 129]
Is this an intended behavior? I originally expected bv[4..12]
to behave exactly like a regular aligned BitSlice
(i.e., bv[4..12].to_owned().into()
will just return an one element Vec<u8>
)..
We currently implement Add
and AddAssign
traits that "works just like adding numbers longhand on paper" (right-to-left). It would be nice to have a reverse_add()
method that does addition from left-to-right. We can use the same function semantics as add_assign but do not use the std::iter::Rev
trait that reverse the iteration, i.e. remove the .rev()
call on lines 2230 and 2231. I have already tested it and works fine.
Here is an example:
use bitvec::prelude::*;
let a = bitvec![1, 0, 1, 0];
let b = bitvec![1, 1, 0, 0];
let s = a.add_reverse(&b);
assert_eq!(bitvec![0, 0, 0, 1], s);
Use case
In the paper of SIMD json parser, the addition operation is done left-to-right (e.g page 7, Fig.3 C = B + ES
). Of course, I do not expect this library to be used in high performance applications, but I found quite convenient to have a such method.
First, thank you for bitvec
.
I found a strange problem (freeing of unallocated memory) when shrinking a BitVec
to a BitBox
, but I can reproduce it only on macos. I have a hunch the problem may not be in bitvec
at all, but possibly in something Rust-specific. Any guidance will be appreciated.
I created a reproducible minimal testcase in kulp/bitvec-debugging. Github Actions shows failure on macos, whereas Ubuntu succeeds (the Windows job tends to be canceled because it does not finish before macos fails). On my Mac OS 10.14.6 machine, test failure (with bitvec 0.17.3) looks like this :
$ cargo +1.42.0 test
Finished test [unoptimized + debuginfo] target(s) in 1.64s
Running target/debug/deps/debug_bitbox-776d3300cd651276
running 1 test
debug_bitbox-776d3300cd651276(79649,0x700005796000) malloc: *** error for object 0x7fb526800000: pointer being freed was not allocated
debug_bitbox-776d3300cd651276(79649,0x700005796000) malloc: *** set a breakpoint in malloc_error_break to debug
error: test failed, to rerun pass '--lib'
Caused by:
process didn't exit successfully: `/Users/kulp/Documents/Code/Rust/debug_bitbox/target/debug/deps/debug_bitbox-776d3300cd651276` (signal: 6, SIGABRT: process abort signal)
A backtrace from malloc_error_break
shows alloc::raw_vec::shrink_to_fit
occurring during core::ptr::drop_in_place
. The dynamically-nearest bitvec
code is this :
frame #9: 0x000000010000579e debug_bitbox-776d3300cd651276`bitvec::boxed::ops::_$LT$impl$u20$core..ops..drop..Drop$u20$for$u20$bitvec..boxed..BitBox$LT$O$C$T$GT$$GT$::drop::ha5df38b3a7bb05c2(self=0x000070000dd2e430) at ops.rs:129:7
126 let ptr = self.as_mut_slice().as_mut_ptr();
127 let len = self.as_slice().len();
128 // Run the `Box<[T]>` destructor.
-> 129 drop(unsafe { Vec::from_raw_parts(ptr, 0, len) }.into_boxed_slice());
130 }
131 }
132
I wrote a script that demonstrates the problem exists across a broad swathe of Rust versions and bitvec
versions. All x.0 versions of Rust from 1.36.0 through 1.42.0 and all x.0 versions of bitvec
from 0.11.0 through 0.17.0 (and 0.17.3 as well) demonstrate the issue.
For the real code in which this issue was first discovered, my only workaround is to switch to Linux, which is possible for the short term but rather worrying.
Do you think it plausible that this is a Rust compiler or library issue ? if so, can you suggest a plan of attack for chasing it down further, or reporting it elsewhere ?
The link at the top of https://github.com/myrrlyn/bitvec points to https://myrrlyn.net/bitvec which doesn't exist. Maybe should be https://myrrlyn.net/crates/bitvec.
Hello,
Is it possible to declare a BitVector of a certain dimension in a struct?
Like [bool; 5]
for a 5 bit value. I think I could just declare it as BitVector and that should work, but I also would like to enforce the length of the vector. Is this possible?
Thanks
I love BitVec, but really need something like a BitVecDeque so I can have an efficient bit queue (push back, pop front). Is this possible with BitVec as it is, or would this be an enhancement?
Is BitVec safe to send to another thread?
In the latest version of the crate, !Send is derived, however in the past, this was not the case. I was wondering whether this is deliberate?
I'm attempting to use this as an alternative to the cpu_set_t
affinity mask that the Linux kernel uses for getting and setting core affinities. The kernel operates on windows of c_ulong
widths, so I have a 2048-bit wide BitVec<LittleEndian, c_ulong>
to work with.
I have a laptop with 8 virtual cores, so it correctly writes 8 bits when used with the sched_getaffinity
syscall. However, the Bits::count_ones
method only returns 2
, and printing the bitvec returns 11
, although printing and rendering the slice correctly shows 8 ones.
pub fn main() {
let affinity = Process::current().get_affinity().unwrap();
println!("ncpus: {}", affinity.count_ones());
println!("{}", affinity);
let slice: &[u64] = affinity.as_slice();
println!("{:?}", slice);
for elt in slice {
println!("{:0w$b} ", elt, w = std::mem::size_of::<u64>() * 8);
}
}
Output:
ncpus: 2
[11]
[255]
0000000000000000000000000000000000000000000000000000000011111111
Reproduction:
use bitvec::{order::Lsb0, vec::BitVec, slice::BitSlice};
fn main() {
let mut dir = BitVec::<Lsb0, u8>::new();
let mut swdio = BitVec::<Lsb0, u8>::new();
swdio.resize(64, true);
dir.resize(64, true);
let mut seq = 0xE79E; // LSb first
for _ in 0..16 {
swdio.push(seq & 0b1 != 0);
seq >>= 1;
}
dir.resize(dir.len() + 16, true);
swdio.resize(swdio.len() + 64, true);
dir.resize(dir.len() + 64, true);
// idle cycles = host clocks interface with SWDIO low
swdio.resize(swdio.len() + 10, false);
dir.resize(dir.len() + 10, true);
}
Since 0.16 (I think?) this panics with:
thread 'main' panicked at 'Capacity overflow: 144 overflows allocation size 128', src/vec/api.rs:328:3
which is due to this assertion in set_len
:
I am working on simple exhaustive algorithm to CNF satisfiability problem. Tried my minimum code here:
pub fn solve_cnf_plain() -> bool {
let n = 3;
let one = bitvec![1];
let mut var = bitvec![0; n];
let max = bitvec![1; n];
while var <= max {
println!("{:?} {:?}", &var, &max);
var += one.clone();
}
true
}
I expect to print only 8 line of output assuring bits 000 to 111 is searched. However the actual output is like:
BitVec<Lsb0, usize> [000] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [001] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [010] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [011] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [100] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [101] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [110] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [111] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [1000] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [1001] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [1010] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [1011] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [1100] BitVec<Lsb0, usize> [111]
BitVec<Lsb0, usize> [1101] BitVec<Lsb0, usize> [111]
There are extra lines when I add 1 to 111 (into 1000), it seems that the program misjudged 1000 <= 111
(actually should be 1000 > 111
).
Is my code correct, or is there some problem in this library? Thanks!
(My persional opinion is that the problem may be found at some PartialOrd
impl inside this crate)
OS x64 Windows, toolchain stable-x86_64-pc-windows-gnu, rustc 1.40.0 (73528e339 2019-12-16).
A subtle bug appeared during tests of my no_std lib. And since this test meets no unsafe except for rust internals and bitvec internals, I suspect there's a problem with bitvec. Maybe it's Rust fault itself, but chances are low.
Comments in the test describe the problem and this is the only file where bitvec is used. There's only six bitvec operations: call to BitBox::default()
through derive
, BitVec::from(BitBox)
, BitVec.resize()
, BitVec.into_boxed_bitslice()
, BitBox.get(usize)
, and BitBox.get_mut(usize)
.
Forgot to mention: all tests pass on simple "cargo test" in debug mode, and only this one test crashes on "cargo test --release".
https://github.com/myrrlyn/bitvec/blob/83170a4c1c9537aa61cdd906d1ad92e7ecfd2f1b/src/vec.rs#L38
https://github.com/myrrlyn/bitvec/blob/83170a4c1c9537aa61cdd906d1ad92e7ecfd2f1b/src/vec.rs#L250-L261
Shouldn't the doc comment be changed?
(I'm quite new to rust, I'd make a PR but it's probably smart if I don't go that far yet just in case I missed something).
I would like to do something like this:
fn main() {
let source: u8 = 0b1010_0101;
let bv = BitVec::from(source);
}
I think I can convert a u8
to a Vec<bool>
, but I would like to avoid as many conversions as possible, since performance is critical.
I made a little iterator benchmark. On my machine the double-pointer iterator (even without folding optimization) is two times faster than original "split_first" iterator.
Though, there are some problems I've met while was playing with iterators:
Okay, the latter ain't a real problem, but nightly-only features are. Should it be optimized for "try_fold" (and "try_for_each", "fold", "for_each") at all? More #[cfg]s, now for those nightly things? Even though there's no bench for fold yet, 'cuz it's hard to bench without proper implementation (I hasn't finished it yet), it's easy to write with the new "domain" (good work there!).
And I need a little help with bit pointers: forgot everything I knew about that crate after the month full of problems. So, how could I create two bitmasks, one for the very first bit and one for the last? I think, that's something about index and ((len - index) % sizeof*8), but I just don't know how to get at least one index, lol.
Feature request by @fasterthanlime
Implementation notes:
pointer-to-pointer arithmetic is, according to C++ and thus likely LLVM,
defined only between two pointers known to be in the same allocation region.
(article 1) I do not know at this time what Miri thinks of pointer-to-pointer
arithmetic, but I strongly suspect, given Miri’s implementation of pointers in
its evaluator model (article 2) that this is suspect at best and a candidate
for rejection at worst.
Given that the rest of the crate is also UB in Miri that the compiler merely
happens to not yet reject, I am not too concerned about adding yet more
pointer operations that cause errors in Miri.
See @RalfJung’s blog article (article 3) for more about how Miri (and by
extension, eventually, rustc) treats pointer manipulation.
BitPtr
-to-BitPtr
arithmetic is required by the constraints above to only
be meaningful within the same allocation region, and thus, with the same type.
This means that the function signature is not required to generalize as
fn BitPtr<T: BitStore>::ptr_diff<U: BitStore>(self, other: BitPtr<U)>) -> _;
but may be kept specific as just
fn BitPtr<T: BitStore>::ptr_diff(self, other: Self) -> _;
We cannot enforce this in the type system, but this is an unsafe fn
with the
precondition that other
and self
be derived from the same overarching
BitSlice<_, T>
region. It is firmly undefined behavior, even by the
UB-adjacent standards of this crate, to call ptr_diff
with pointers from two
different regions.
Prior art in fn store::BitStore::offset
indicates that the return type
should be the anonymous record { elts: isize, bits: i8 }
(canonicalized as
the tuple (isize, i8)
). That function produces (isize, BitIdx)
because it
computes a jump value for ptr::offset
and an absolute bit index in the
element to which the jump value refers. This function produces a jump value,
and a bit distance between the start pointer self
and the end pointer
other
.
BitPtr<T: BitStore>
is notably missing a C: Cursor
type parameter. It is
defined behavior for two BitSlice<C: Cursor, T: BitStore>
handles drawn from
the same region to have different C
type parameters. How should this be
handled?
BitPtr::ptr_diff
can only compute BitIdx
differences. This will
necessarily produce a bit distance that describes the index difference
rather than the electrical difference, but, this is unavoidable with the
information available.
fn BitSlice<C: Cursor, T>::offset<D: Cursor>(&self, other: &BitSlice<D, T>) -> _;
could convert both self
and other
’s BitIdx
values to BitPos
using
their provided Cursor
implementations, then construct BitPtr<T>
pointers
from the BitPos
values and call ptr_diff
on them. This would compute the
electrical bit distance between the two BitSlice
handles.
Nom integration notes
bitvec
currently relies on behavior that is compiled as expected in current
rustc
but is marked as UB in Miri, and will likely eventually be rejected by
the compiler. For this reason, I am hesitant to encourage the nom
crate to
begin depending on bitvec
for its bit-stream parsing.
However, it is advantageous to have BitSlice
integrate with nom
’s traits,
so that clients who wish to use bitvec
for bitstream nom
parsing are able
to drop BitSlice
into any trait-driven nom
functions.
Once bitvec
depends on nom
, it becomes illegal for nom
to also depend on
bitvec
. I don’t know the dependency rules offhand, but I am hoping that if
bitvec
depends on nom
optionally, the default feature set of bitvec
is
dependable by nom
without creating a dependency cycle.
If nom
elects to depend on bitvec
, bitvec
’s nom integration will be
removed and placed in nom
instead. This may constitute a major-breaking
change if I release a 1.0
.
Hi @myrrlyn, thanks for this crate and blog post!
I'm a beginner bit-twiddler, so being able to access bits "normally" (instead of shifting and boolean ops) gives me a fighting chance for my project (parsing optical signals on an ARM microcontroller).
I've used the crate successfully for testing on my mac, but am unable to actually compile to to my microcontroller (an stm32f1 "black pill" / cortex m3; target = "thumbv7m-none-eabi"
).
I'm using:
bitvec = { url = "https://github.com/myrrlyn/bitvec", default-features = false, features = []}
and my lockfile shows I'm using bitvec 0.16.1
.
I believe the issue from a transitive dependency, radium:
Compiling radium v0.2.0
error[E0432]: unresolved imports `core::sync::atomic::AtomicI64`, `core::sync::atomic::AtomicU64`
--> /Users/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/radium-0.2.0/src/lib.rs:29:45
|
29 | self, AtomicBool, AtomicI16, AtomicI32, AtomicI64, AtomicI8, AtomicIsize, AtomicPtr, AtomicU16,
| ^^^^^^^^^ no `AtomicI64` in `sync::atomic`
30 | AtomicU32, AtomicU64, AtomicU8, AtomicUsize, Ordering,
| ^^^^^^^^^ no `AtomicU64` in `sync::atomic`
|
help: a similar name exists in the module
|
29 | self, AtomicBool, AtomicI16, AtomicI32, AtomicI8, AtomicI8, AtomicIsize, AtomicPtr, AtomicU16,
| ^^^^^^^^
help: a similar name exists in the module
|
30 | AtomicU32, AtomicU8, AtomicU8, AtomicUsize, Ordering,
| ^^^^^^^^
Looks like it's trying to import AtomicI64
, but there's no such type on my 32-bit microcontroller.
Any suggestions for how we should try to untangle this?
Perhaps we can:
bitvec
so that it doesn't depend on radium
?radium
to use Rust's feature flags and conditional compilation system to only import 64-bit types on the appropriate platforms?I haven't done much feature flag or no-std Rust development, but am open to learn and submit a patch if you can recommend a direction.
Thanks!
fix please a little typo, I have an error:
thread 'main' panicked at 'Index out of range: 32 >= 32',
on code *bits.at(32) = true;
where bits are from u32
Relevant Code Failing
let u16b = u16::from_ne_bytes(0x1234u16.to_le_bytes());
bytes[5 ..][.. 14].store_le(u16b);
assert_eq!(bytes[5 ..][.. 14].load_le::<u16>(), 0x1234u16);
Result
---- fields::tests::lsb0 stdout ----
thread 'main' panicked at 'assertion failed: `(left == right)`
left: `13330`,
right: `4660`', src/fields.rs:999:3
---- fields::tests::msb0 stdout ----
thread 'main' panicked at 'assertion failed: `(left == right)`
left: `13330`,
right: `4660`', src/fields.rs:1073:3
On big-endian PPC, the above code cails, but on little-endian PPC, it passes.
as_slice
of BitVec
returns all edges, even partially owned ones. However, BitSlice::as_slice
only returns full edges.
For an encoder we need to access all the internal data. Currently it requires that we convert BitSlice
to a BitVec
temporarily. It would be nice if we could prevent this conversion :)
In some cases, it is not interesting to deal with a specific endianness, and it would be more useful to use the native endianness whatever it is. One could import the right endianness depending on the target endianness, but it would be more convenient if there was a "NativeEndian" to avoid the manual gymnastic.
Rework current mutable slice split logic into non-atomic, where bitslice.split_at_mut()
would return a tuple of (&mut self, AfterSplit)
. AfterSplit logic is explained lower.
I was recently thinking about simple and fast iteration logic based on std slice iterator with some bit tweaks, but such iterator should use default access. I simply don't like carrying atomic overhead everywhere, even if I can deal with it. Also, getting rid of atomics would simplify overall logic of this crate.
On the high level AfterSplit (or whatever name it would get) is a sibling of BitSlice: it implements many traits and functions BitSlice implements (for example, it could be mutable split itself, and after.split_at_mut()
produces (self, AfterSplit)
tuple). But it is a downgrade, and so it doesn't have any public constructor and some BitSlice functions aren't there.
On a lower level AfterSplit is a three words wide structure: (owned_bits: Bitstore, slice: &mut Bitslice)
. The trick is simple: slice would always start with properly owned byte (if it's length is not zero, of course) because slices could be split only at machine word border, and the lone bitstore could only store $BITS - 1
bits at max. The latter is easy: split_mut() returns (self, ...), and "self" part there will be always greedy, so its last item will have at least one bit, and never zero (unless the whole slice has zero length, but in that case both slices have zero length). So, the BitStore part of AfterSplit always has one bit to show it's length, and that's enough (trailing_zeroes, leading_zeroes, wrapped up in an ordering trait function).
A lot of code and docs to write, three words wide instead of two, not a proper BitSlice, thus it could be limited in options.
I see three ways to deal with mutable splits: no split_mut() at all, atomics, or some kind of AfterSplit. The first way is very simple, but strongly hits functionality; the second way hits performance and now you should uphold the possible double access on most operations with slice, even if they use immutable access; the third way is a compromise: mutable splitting is allowed and doesn't interfere with other parts of library, but is somewhat unhandy for the user.
The current atomic access logic crawls everywhere due to edge cases you should think about: for example, iterator over an immutable bitslice should use atomic access, because it could be called onto the product of split_mut(), which could be mutated right now (which is a very rare case, but you are forced to uphold it). Right now atomics are everywhere. This is good to emulate std slice logic, but BitSlice is not a common slice; nobody would treat utf-8 string as ascii, unless it is checked to be ascii. And so BitSlice doesn't need to do everything in the way std slices do. For example, "string".chars() iterator produce value itself, not a reference, and I haven't heard a single complaint about it. And so could work BitSlice immutable iterator: just give 'em the bool and forget about it.
Actually, there's just one question: should it be implemented at all, and if yes, how should it be implemented? Once it's resolved, there's a strait way to write code.
After this, there would be an easy way to write fast value-producing iterator over an immutable bitslice, without thinking much about mutable access problems. Maybe, implementing some other things would become easier. I'm not very sure about it. But it's a move from copypasting common slices' logic, which could be good for performance, as bit slice is special and the fast way to do things is almost never the same as fast way with common slices.
And some things that are not related to this particular question, but related to fast iterations:
rotate_left
, rotate_right
, reverse_bits
(names are the same with numeric primitives' functions);rotate_forward
, rotate_backward
, from_ne_order
— this function takes BitStore with some bits and turns it into the required order (essentially a no-op for Lsb0 and single bit reverse for Msb0). Those functions could be easily implemented for two default orderings with functions from 1.I could have written PR with those things before that RFC, but there's no point in those functions before mutable split question is answered in some way.
I found this when investigating a flaky test case, managed to pull out an example I think. It might not be the smallest example possible, but as I started removing lines, the tests started passing... not entirely sure what's going on.
test_bitvec1
fails when it should pass, after some experimentation, it seems to be extend_from_slice
res_bits.extend_from_slice(&chunk[chunk.len() - remaining_bits..]);
If I iterate each bit and push to res_bits
instead, everything works ok.
Note: It does pass with normal cargo test...
I used the following to reproduce consistently
RUSTFLAGS="-Z sanitizer=thread" cargo +nightly test --target x86_64-unknown-linux-gnu test_bitvec
use bitvec::prelude::*;
trait Writer {
fn write(&self, output_is_le: bool, bitsize: Option<usize>) -> BitVec<Msb0, u8>;
}
impl Writer for u32 {
fn write(&self, output_is_le: bool, bit_size: Option<usize>) -> BitVec<Msb0, u8> {
let input = if output_is_le {
self.to_le_bytes()
} else {
self.to_be_bytes()
};
let input_bits: BitVec<Msb0, u8> = input.to_vec().into();
let res_bits: BitVec<Msb0, u8> = {
if let Some(bit_size) = bit_size {
if bit_size > input_bits.len() {
todo!() // TODO: return err
}
if output_is_le {
// Example read 10 bits u32 [0xAB, 0b11_000000]
// => [10101011, 00000011, 00000000, 00000000]
let mut res_bits = BitVec::<Msb0, u8>::new();
let mut remaining_bits = bit_size;
// println!("input_bits: {}", input_bits);
for chunk in input_bits.chunks(8) {
println!("chunk: {}", chunk);
if chunk.len() > remaining_bits {
res_bits.extend_from_slice(&chunk[chunk.len() - remaining_bits..]);
break;
} else {
res_bits.extend_from_slice(chunk)
}
remaining_bits -= chunk.len();
}
res_bits
} else {
// Example read 10 bits u32 [0xAB, 0b11_000000]
// => [00000000, 00000000, 00000010, 10101111]
input_bits[input_bits.len() - bit_size..].into()
}
} else {
input_bits
}
};
res_bits
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_bitvec1() {
let data = 0x03ABu32;
let data_bits = data.write(true, Some(10));
assert_eq!(bitvec![Msb0, u8; 1,0,1,0,1,0,1,1, 1,1], data_bits);
let data_vec = data_bits.into_vec();
assert_eq!(vec![0xAB, 0b11_000000], data_vec);
}
#[test]
fn test_bitvec2() {
let data = 0x03ABu32;
let data_bits = data.write(false, Some(10)).into_vec();
assert_eq!(vec![0b11, 0xAB], data_bits);
}
#[test]
fn test_bitvec3() {
let data = 0xDDCCBBAA;
let data_bits = data.write(true, None).into_vec();
assert_eq!(vec![0xAA, 0xBB, 0xCC, 0xDD], data_bits);
}
#[test]
fn test_bitvec4() {
let data = 0xDDCCBBAA;
let data_bits = data.write(false, None).into_vec();
assert_eq!(vec![0xDD, 0xCC, 0xBB, 0xAA], data_bits);
}
}
running 4 tests
==================
WARNING: ThreadSanitizer: data race (pid=24625)
Read of size 8 at 0x7b440000ff00 by main thread:
#0 memcpy /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:801 (test_bitvec-c97c6a039ce23fcb+0x26ac7)
#1 core::intrinsics::copy_nonoverlapping /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libcore/intrinsics.rs:2058 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#2 core::slice::<impl [T]>::copy_from_slice /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libcore/slice/mod.rs:2304 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#3 <alloc::vec::Vec<T> as alloc::vec::SpecExtend<&T,core::slice::Iter<T>>>::spec_extend /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/vec.rs:2178 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#4 alloc::vec::Vec<T>::extend_from_slice /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/vec.rs:1586 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#5 alloc::slice::hack::to_vec /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/slice.rs:160 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#6 alloc::slice::<impl [T]>::to_vec /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/slice.rs:395 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#7 <alloc::vec::Vec<T> as core::clone::Clone>::clone /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/vec.rs:1910 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#8 <test::event::CompletedTest as core::clone::Clone>::clone /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libtest/event.rs:13 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#9 <test::event::TestEvent as core::clone::Clone>::clone /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libtest/event.rs:33 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#10 test::console::on_test_event /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libtest/console.rs:224 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#11 test::console::run_tests_console::{{closure}} /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libtest/console.rs:280 (test_bitvec-c97c6a039ce23fcb+0xbdc8a)
#12 std::rt::lang_start::{{closure}} /home/sharks/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67 (test_bitvec-c97c6a039ce23fcb+0x95961)
#13 std::rt::lang_start_internal::{{closure}} /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:52 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#14 std::panicking::try::do_call /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:297 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#15 std::panicking::try /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:274 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#16 std::panic::catch_unwind /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panic.rs:394 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#17 std::rt::lang_start_internal /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:51 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#18 main ??:? (test_bitvec-c97c6a039ce23fcb+0x9fb0a)
Previous write of size 8 at 0x7b440000ff00 by thread T1 (mutexes: write M3231516213647360):
#0 malloc /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:650 (test_bitvec-c97c6a039ce23fcb+0x1b2f4)
#1 alloc::alloc::alloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:80 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#2 <alloc::alloc::Global as core::alloc::AllocRef>::alloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:174 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#3 alloc::raw_vec::RawVec<T,A>::allocate_in /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/raw_vec.rs:152 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#4 alloc::raw_vec::RawVec<T,A>::with_capacity_in /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/raw_vec.rs:135 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#5 alloc::raw_vec::RawVec<T>::with_capacity /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/raw_vec.rs:92 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#6 alloc::vec::Vec<T>::with_capacity /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/vec.rs:358 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#7 alloc::slice::hack::to_vec /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/slice.rs:159 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#8 alloc::slice::<impl [T]>::to_vec /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/slice.rs:395 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#9 test::run_test_in_process /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libtest/lib.rs:556 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#10 test::run_test::run_test_inner::{{closure}} /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libtest/lib.rs:450 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
Location is heap block of size 263 at 0x7b440000ff00 allocated by thread T1:
#0 malloc /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:650 (test_bitvec-c97c6a039ce23fcb+0x1b2f4)
#1 alloc::alloc::alloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:80 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#2 <alloc::alloc::Global as core::alloc::AllocRef>::alloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:174 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#3 alloc::raw_vec::RawVec<T,A>::allocate_in /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/raw_vec.rs:152 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#4 alloc::raw_vec::RawVec<T,A>::with_capacity_in /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/raw_vec.rs:135 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#5 alloc::raw_vec::RawVec<T>::with_capacity /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/raw_vec.rs:92 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#6 alloc::vec::Vec<T>::with_capacity /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/vec.rs:358 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#7 alloc::slice::hack::to_vec /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/slice.rs:159 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#8 alloc::slice::<impl [T]>::to_vec /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/slice.rs:395 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#9 test::run_test_in_process /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libtest/lib.rs:556 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
#10 test::run_test::run_test_inner::{{closure}} /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libtest/lib.rs:450 (test_bitvec-c97c6a039ce23fcb+0xd0f6a)
Mutex M3231516213647360 is already destroyed.
Thread T1 'tests::test_bit' (tid=24627, finished) created by main thread at:
#0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967 (test_bitvec-c97c6a039ce23fcb+0x1c89b)
#1 std::sys::unix::thread::Thread::new /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/sys/unix/thread.rs:66 (test_bitvec-c97c6a039ce23fcb+0xfe52c)
#2 std::rt::lang_start::{{closure}} /home/sharks/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67 (test_bitvec-c97c6a039ce23fcb+0x95961)
#3 std::rt::lang_start_internal::{{closure}} /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:52 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#4 std::panicking::try::do_call /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:297 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#5 std::panicking::try /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:274 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#6 std::panic::catch_unwind /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panic.rs:394 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#7 std::rt::lang_start_internal /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:51 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#8 main ??:? (test_bitvec-c97c6a039ce23fcb+0x9fb0a)
SUMMARY: ThreadSanitizer: data race /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libcore/intrinsics.rs:2058 in core::intrinsics::copy_nonoverlapping
==================
test tests::test_bitvec1 ... FAILED
==================
WARNING: ThreadSanitizer: data race (pid=24625)
Write of size 8 at 0x7b4000008000 by main thread:
#0 free /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:706 (test_bitvec-c97c6a039ce23fcb+0x1b948)
#1 alloc::alloc::dealloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:102 (test_bitvec-c97c6a039ce23fcb+0xa824c)
#2 <alloc::alloc::Global as core::alloc::AllocRef>::dealloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:186 (test_bitvec-c97c6a039ce23fcb+0xa824c)
#3 alloc::alloc::box_free /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:285 (test_bitvec-c97c6a039ce23fcb+0xa824c)
#4 core::ptr::drop_in_place /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libcore/ptr/mod.rs:178 (test_bitvec-c97c6a039ce23fcb+0xa824c)
#5 std::sync::mpsc::mpsc_queue::Queue<T>::pop /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sync/mpsc/mpsc_queue.rs:94 (test_bitvec-c97c6a039ce23fcb+0xa824c)
#6 std::rt::lang_start::{{closure}} /home/sharks/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67 (test_bitvec-c97c6a039ce23fcb+0x95961)
#7 std::rt::lang_start_internal::{{closure}} /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:52 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#8 std::panicking::try::do_call /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:297 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#9 std::panicking::try /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:274 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#10 std::panic::catch_unwind /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panic.rs:394 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#11 std::rt::lang_start_internal /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:51 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#12 main ??:? (test_bitvec-c97c6a039ce23fcb+0x9fb0a)
Previous write of size 8 at 0x7b4000008000 by thread T1:
#0 malloc /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:650 (test_bitvec-c97c6a039ce23fcb+0x1b2f4)
#1 alloc::alloc::alloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:80 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#2 <alloc::alloc::Global as core::alloc::AllocRef>::alloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:174 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#3 alloc::alloc::exchange_malloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:268 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#4 std::sync::mpsc::mpsc_queue::Node<T>::new /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sync/mpsc/mpsc_queue.rs:53 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#5 std::sync::mpsc::mpsc_queue::Queue<T>::push /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sync/mpsc/mpsc_queue.rs:68 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#6 std::sync::mpsc::shared::Packet<T>::send /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sync/mpsc/shared.rs:166 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#7 std::sync::mpsc::Sender<T>::send /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sync/mpsc/mod.rs:835 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
Thread T1 'tests::test_bit' (tid=24627, finished) created by main thread at:
#0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967 (test_bitvec-c97c6a039ce23fcb+0x1c89b)
#1 std::sys::unix::thread::Thread::new /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/sys/unix/thread.rs:66 (test_bitvec-c97c6a039ce23fcb+0xfe52c)
#2 std::rt::lang_start::{{closure}} /home/sharks/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67 (test_bitvec-c97c6a039ce23fcb+0x95961)
#3 std::rt::lang_start_internal::{{closure}} /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:52 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#4 std::panicking::try::do_call /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:297 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#5 std::panicking::try /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:274 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#6 std::panic::catch_unwind /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panic.rs:394 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#7 std::rt::lang_start_internal /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:51 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#8 main ??:? (test_bitvec-c97c6a039ce23fcb+0x9fb0a)
SUMMARY: ThreadSanitizer: data race /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:102 in alloc::alloc::dealloc
==================
test tests::test_bitvec2 ... ok
test tests::test_bitvec3 ... ok
test tests::test_bitvec4 ... ok
==================
WARNING: ThreadSanitizer: data race (pid=24625)
Write of size 8 at 0x7b4000010200 by main thread:
#0 free /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:706 (test_bitvec-c97c6a039ce23fcb+0x1b948)
#1 alloc::alloc::dealloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:102 (test_bitvec-c97c6a039ce23fcb+0xaf68f)
#2 <alloc::alloc::Global as core::alloc::AllocRef>::dealloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:186 (test_bitvec-c97c6a039ce23fcb+0xaf68f)
#3 alloc::alloc::box_free /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:285 (test_bitvec-c97c6a039ce23fcb+0xaf68f)
#4 core::ptr::drop_in_place /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libcore/ptr/mod.rs:178 (test_bitvec-c97c6a039ce23fcb+0xaf68f)
#5 <std::sync::mpsc::mpsc_queue::Queue<T> as core::ops::drop::Drop>::drop /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sync/mpsc/mpsc_queue.rs:109 (test_bitvec-c97c6a039ce23fcb+0xaf68f)
#6 core::ptr::drop_in_place /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libcore/ptr/mod.rs:178 (test_bitvec-c97c6a039ce23fcb+0xaf68f)
#7 core::ptr::drop_in_place /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libcore/ptr/mod.rs:178 (test_bitvec-c97c6a039ce23fcb+0xaf68f)
#8 std::rt::lang_start::{{closure}} /home/sharks/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67 (test_bitvec-c97c6a039ce23fcb+0x95961)
#9 std::rt::lang_start_internal::{{closure}} /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:52 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#10 std::panicking::try::do_call /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:297 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#11 std::panicking::try /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:274 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#12 std::panic::catch_unwind /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panic.rs:394 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#13 std::rt::lang_start_internal /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:51 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#14 main ??:? (test_bitvec-c97c6a039ce23fcb+0x9fb0a)
Previous write of size 8 at 0x7b4000010200 by thread T4:
#0 malloc /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:650 (test_bitvec-c97c6a039ce23fcb+0x1b2f4)
#1 alloc::alloc::alloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:80 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#2 <alloc::alloc::Global as core::alloc::AllocRef>::alloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:174 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#3 alloc::alloc::exchange_malloc /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:268 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#4 std::sync::mpsc::mpsc_queue::Node<T>::new /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sync/mpsc/mpsc_queue.rs:53 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#5 std::sync::mpsc::mpsc_queue::Queue<T>::push /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sync/mpsc/mpsc_queue.rs:68 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#6 std::sync::mpsc::shared::Packet<T>::send /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sync/mpsc/shared.rs:166 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
#7 std::sync::mpsc::Sender<T>::send /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/libstd/sync/mpsc/mod.rs:835 (test_bitvec-c97c6a039ce23fcb+0xa8d6f)
Thread T4 'tests::test_bit' (tid=24632, finished) created by main thread at:
#0 pthread_create /rustc/llvm/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:967 (test_bitvec-c97c6a039ce23fcb+0x1c89b)
#1 std::sys::unix::thread::Thread::new /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/sys/unix/thread.rs:66 (test_bitvec-c97c6a039ce23fcb+0xfe52c)
#2 std::rt::lang_start::{{closure}} /home/sharks/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67 (test_bitvec-c97c6a039ce23fcb+0x95961)
#3 std::rt::lang_start_internal::{{closure}} /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:52 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#4 std::panicking::try::do_call /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:297 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#5 std::panicking::try /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panicking.rs:274 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#6 std::panic::catch_unwind /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/panic.rs:394 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#7 std::rt::lang_start_internal /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8//src/libstd/rt.rs:51 (test_bitvec-c97c6a039ce23fcb+0xf85b8)
#8 main ??:? (test_bitvec-c97c6a039ce23fcb+0x9fb0a)
SUMMARY: ThreadSanitizer: data race /rustc/a74d1862d4d87a56244958416fd05976c58ca1a8/src/liballoc/alloc.rs:102 in alloc::alloc::dealloc
==================
failures:
---- tests::test_bitvec1 stdout ----
chunk: [10101011]
chunk: [00000011]
thread 'tests::test_bitvec1' panicked at 'assertion failed: `(left == right)`
left: `[171, 192]`,
right: `[171, 232]`', tests/test_bitvec.rs:66:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
tests::test_bitvec1
test result: FAILED. 3 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
ThreadSanitizer: reported 3 warnings
Hi. Recently I was needed to work with bits, and I had chosen your crate. But as soon as I imported BitVec
structure, other modules showed errors like cannot infer type for 'T'
and the trait io::Read is not implemented for &[i32]
. I was wondering and have looked a bit closer, why other modules in my crate were broken. The problem was in AsRef
and AsMut
generation for arrays.
So, consider we have the code:
use std::io::{self, Write};
// use bitvec;
pub fn breaks_as_mut() {
let bytes_to_write = "Hello, world!".as_bytes();
let mut name_buffer = [0; 5];
name_buffer.as_mut().write_all(bytes_to_write).unwrap();
}
pub fn breaks_as_ref() {
let b = [0x05];
applies_impl_read(b.as_ref()).unwrap();
}
pub fn applies_impl_read(_stream: impl io::Read) -> io::Result<()> {
Ok(())
}
If the use bitvec
line is commented, everything is fine. But when I uncomment the line, the errors occurred:
warning: unused import: `bitvec`
--> src/lib.rs:3:5
|
3 | use bitvec;
| ^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
error[E0282]: type annotations needed
--> src/lib.rs:9:17
|
9 | name_buffer.as_mut().write_all(bytes_to_write).unwrap();
| ^^^^^^ cannot infer type for `T`
|
= note: type must be known at this point
error[E0277]: the trait bound `&[i32]: std::io::Read` is not satisfied
--> src/lib.rs:14:5
|
14 | applies_impl_read(b.as_ref()).unwrap();
| ^^^^^^^^^^^^^^^^^ the trait `std::io::Read` is not implemented for `&[i32]`
...
17 | pub fn applies_impl_read(_stream: impl io::Read) -> io::Result<()> {
| ----------------- -------- required by this bound in `applies_impl_read`
|
= help: the following implementations were found:
<&[u8] as std::io::Read>
warning: unused import: `Write`
--> src/lib.rs:1:21
|
1 | use std::io::{self, Write};
| ^^^^^
error: aborting due to 2 previous errors
Some errors have detailed explanations: E0277, E0282.
For more information about an error, try `rustc --explain E0277`.
error: could not compile `checkbitvec`.
I know that I can replace name_buffer.as_mut().write_all()
with (&mut name_buffer[..]).write_all()
, and applies_impl_read(b.as_ref())
with applies_impl_read(&b[..])
, but I like to use as_ref
and as_mut
alternatives to not so beautiful &[..]
and &mut [..]
constructions. So I need to change a lot of code to reduce the errors.
The macro generations AsRef
and AsMut
can be turned in to feature that will be by default, and I will disable the feature so my code will not be broken. Or there is exists another solution...
I use rust 1.39.0 on stable-x86_64-unknown-linux-gnu.
Thanks!
/// This is different from [
.body()
] in that it always includes the head
/// and tail elements, even if they are partial.
https://github.com/myrrlyn/bitvec/blob/master/src/slice.rs#L1800
Perhaps this function has been removed?
Hi. I've been implementing a similar API to bitvec
for ternary recently.
I've come to the conclusion that in compressed encodings (such as 8 bits / 1 byte that bitvec
uses) split_at_mut
is unsound on slices that are not byte-aligned since it permits overlapping mutable views of memory.
I'd think this is quite a serious issue, and the code for split_at_mut
(https://docs.rs/bitvec/0.17.2/src/bitvec/slice/api.rs.html#1220-1223) doesn't seem to address this issue at all.
Is it the case that split_at_mut
is unsound, or have you found a way to resolve the issue? If so, how?
https://github.com/myrrlyn/bitvec/blob/7e5fdc451879976acef1028f0b92b71f21eadd34/src/vec.rs#L3090
➜ bitvec git:(master) ✗ cargo build --no-default-features --features='alloc'
Compiling bitvec v0.11.0 (/home/koushiro/Code/bitvec)
error: cannot find macro eprintln!
in this scope
--> src/vec.rs:3090:3
|
3090 | eprintln!("subtra: {}", subtrahend);
| ^^^^^^^^
error: aborting due to previous error
error: Could not compile bitvec
.
Hi,
Sorry if this question is answered somewhere in the documentation, I checked and couldn't find it.
Is there a way to convert a BitSlice to Vec of u8? And if so, how is the padding handled if the number of bits is not multiple of 8?
When I have a code like this:
use bitvec::{bitvec, BitVec};
fn foo() -> BitVec {
bitvec![0; 10]
}
I do get an error like this:
error: cannot find macro `__bitvec_impl!` in this scope
--> src/foo.rs:4:5
|
5 | bitvec![0; 10]
| ^^^^^^^^^^^^^^
|
= note: this error originates in a macro outside of the current crate (in Nightly builds, run with -Z external-macro-backtrace for more info)
error: aborting due to previous error
error: Could not compile `advent_of_code`.
To learn more, run the command again with --verbose.
I can solve this by changing the use
to use bitvec::{bitvec, BitVec, __bitvec_impl};
.
It would be nice if a way was found to not expose this implementation detail.
Are both orderings continuous on whatever endianness? Like I have LE, and both orderings look like this
Lsb0
1--> 8 9
[10101010][10101010]
9 8 <--1
Msb0
Or Msb0 ordering on LE machine is something like this?
8 <--1 9
[10101010][10101010]
It will be good to add left shift
, right shift
in-place methods for BitSlice
like existed rotate_left
, rotate_right
. I mean the Logical shift.
I'm trying to return an initialized BitVec
from a function using the bitvec!
macro in a no_std
crate. It fails for any storage type except u8
.
// lib.rs
#![no_std]
extern crate bitvec;
use bitvec::prelude::*;
fn error<T: Bits>() -> BitVec<BigEndian, T> {
bitvec![BigEndian, T; 0; 8] // It fails for any length
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_error() {
error::<u8>(); // Only this one works
error::<u16>();
error::<u32>();
error::<u64>();
}
}
# Cargo.toml
[package]
name = "error"
version = "0.1.0"
authors = ["david"]
edition = "2018"
[lib]
name = "error"
path = "src/lib.rs"
[dependencies.bitvec]
git = "https://github.com/myrrlyn/bitvec"
branch = "master"
default-features = false
features = ["alloc"]
# Cargo.lock
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
[[package]]
name = "bitvec"
version = "0.11.0"
source = "git+https://github.com/myrrlyn/bitvec#896c71a34724b9153130d374fc4dc9b170c107a8"
[[package]]
name = "error"
version = "0.1.0"
dependencies = [
"bitvec 0.11.0 (git+https://github.com/myrrlyn/bitvec)",
]
[metadata]
"checksum bitvec 0.11.0 (git+https://github.com/myrrlyn/bitvec)" = "<none>"
Backtrace:
---- error::tests::test_error stdout ----
thread 'error::tests::test_error' panicked at 'attempt to shift left with overflow', /home/david/.cargo/git/checkouts/bitvec-de08e8e455d8f2a6/896c71a/src/bits.rs:237:26
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
1: std::sys_common::backtrace::_print
at src/libstd/sys_common/backtrace.rs:71
2: std::panicking::default_hook::{{closure}}
at src/libstd/sys_common/backtrace.rs:59
at src/libstd/panicking.rs:197
3: std::panicking::default_hook
at src/libstd/panicking.rs:208
4: std::panicking::rust_panic_with_hook
at src/libstd/panicking.rs:474
5: std::panicking::continue_panic_fmt
at src/libstd/panicking.rs:381
6: rust_begin_unwind
at src/libstd/panicking.rs:308
7: core::panicking::panic_fmt
at src/libcore/panicking.rs:85
8: core::panicking::panic
at src/libcore/panicking.rs:49
9: bitvec::bits::Bits::set_at
at /home/david/.cargo/git/checkouts/bitvec-de08e8e455d8f2a6/896c71a/src/bits.rs:237
10: bitvec::bits::Bits::set
at /home/david/.cargo/git/checkouts/bitvec-de08e8e455d8f2a6/896c71a/src/bits.rs:179
11: bitvec::slice::BitSlice<C,T>::set
at /home/david/.cargo/git/checkouts/bitvec-de08e8e455d8f2a6/896c71a/src/slice.rs:474
12: bitvec::vec::BitVec<C,T>::push
at /home/david/.cargo/git/checkouts/bitvec-de08e8e455d8f2a6/896c71a/src/vec.rs:889
13: <bitvec::vec::BitVec<C,T> as core::iter::traits::collect::FromIterator<bool>>::from_iter
at /home/david/.cargo/git/checkouts/bitvec-de08e8e455d8f2a6/896c71a/src/vec.rs:1974
14: core::iter::traits::iterator::Iterator::collect
at /rustc/e938c2b9aae7e0c37c382e4e22bdc360e9a4f0b6/src/libcore/iter/traits/iterator.rs:1465
15: hamming_code::error::error
at src/error.rs:6
16: hamming_code::error::tests::test_error
at src/error.rs:17
17: hamming_code::error::tests::test_error::{{closure}}
at src/error.rs:15
18: core::ops::function::FnOnce::call_once
at /rustc/e938c2b9aae7e0c37c382e4e22bdc360e9a4f0b6/src/libcore/ops/function.rs:231
19: <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once
at /rustc/e938c2b9aae7e0c37c382e4e22bdc360e9a4f0b6/src/liballoc/boxed.rs:704
20: __rust_maybe_catch_panic
at src/libpanic_unwind/lib.rs:85
21: test::run_test::run_test_inner::{{closure}}
at /rustc/e938c2b9aae7e0c37c382e4e22bdc360e9a4f0b6/src/libstd/panicking.rs:272
at /rustc/e938c2b9aae7e0c37c382e4e22bdc360e9a4f0b6/src/libstd/panic.rs:388
at src/libtest/lib.rs:1468
Currently bitvec
has a slice, a box, and a vector, but it does not have reference-counted boxes.
I would like to export BitArc
and BitRc
types which, like BitBox
, use a BitPtr
core pointer structure and recreate the standard library's Arc
and Rc
APIs as appropriate.
This issue is not marked as "good first issue" because it will require an understanding of how the BitPtr
structure describes a memory span, as well as how the standard library's Arc
and Rc
implement their backing store.
Suggested implementation sketch:
struct BitArcBox {
strong: AtomicUsize,
weak: AtomicUsize,
region: BitSlice
}
struct BitRcBox {
// same, but non-atomic
}
struct BitArc {
ptr: BitPtr, // point to BitArcBox
}
struct BitRc {
// same, but BitRcBox
}
(add requisite type parameters; I didn't feel like including them here)
Implementation question: should the BitPtr
point to the start of the boxed structure, or to the start of the slice region inside it?
I am willing to explain the library's core BitPtr
structure enough for you to feel confident in manipulating it in the (A)Rc handles, but you do not need to be intimately familiar with its internal operation. It is probably sufficient to think of it as a very clever slice pointer. You need to be more familiar with the standard library pattern we are copying than with the internals of the crate.
Once the types are created and working, they will need the stdlib API ported over to them. A standing delivery requirement of the crate is that it should be as close to drop-in compatible with the standard types as possible. There are some impossibilities that we just cannot implement without changing the language, but everything the language permits, we should follow.
See ce41612#diff-4c2d359d551a3a9316f813f4fd14f845R966.
Seems like it should be is_valid_tail
. Already fixed in 0.13
but it would be nice to have it fixed in the 0.11.x
release as well (happy to submit a PR for it), thanks!
Currently, these two macros are implemented under macro_rules!
and their expansion results in unpleasantly large compile-time artefact and an extremely inefficient run-time constructor. I believe that a function-like procedural macro would be capable of creating the appropriate source buffer and handle which at runtime needs only to be moved into the heap.
Recent advancements in the compiler also mean that the compile-time creation of &BitSlice
or even &mut BitSlice
references is within reach, but I know less about that topic at this time.
macro_rules!
versions should be a good starting point for what you're trying to accomplishbitvec
library internals. Just as with the existing macro_rules!
macros, the proc macros are just setting up a constructor and then feeding a list of tokens into from_iter
. The only thing the proc-macro is doing differently than the current implementation is moving the buffer construction to compile time.Retain the current macro_rules!
call syntax, but run the BitVec::from_iter
constructor at compile-time
Make a bits!
macro that produces a &'static BitSlice
, and a bits_mut!
macro that produces a &'static mut BitSlice
.
I am happy to talk to you more about what this issue wants to accomplish; I just wanted to get this written down as a starting point.
I am newly trying to use the BitField
API (.load()
in particular), and I might be doing something wrong, but I found that the following code panics :
#[test]
fn fails_with_u64() {
use bitvec::prelude::*;
// The numeric value in `val` makes no difference
let val = 0_u64; // u32, u16, u8 pass; u64 fails
// Lsb0 vs. Msb0 makes no difference
let boxed = BitBox::new(val.bits::<Msb0>());
let _: u64 = boxed[0..64].load(); // 0..63 passes, 0..64 fails
}
like this :
Compiling debug_bitbox2 v0.1.0 (/Users/kulp/Documents/Code/Rust/debug_bitbox2)
Finished test [unoptimized + debuginfo] target(s) in 0.94s
Running target/debug/deps/debug_bitbox2-c997261e3972ef97
running 1 test
test fails_with_u64 ... FAILED
failures:
---- fails_with_u64 stdout ----
thread 'fails_with_u64' panicked at 'attempt to shift left with overflow', /Users/kulp/.cargo/registry/src/github.com-1ecc6299db9ec823/bitvec-0.17.4/src/fields.rs:579:25
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
fails_with_u64
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
error: test failed, to rerun pass '--lib'
whereas using a u32
(or smaller) type to initialize the BitBox
does not panic.
It smells like a fencepost error, given that boxed[0..63]
passes while boxed[0..64]
fails.
This occurs for me both on macOS 10.14.6 and on Arch Linux, with rustc 1.42.0 (b8cedc004 2020-03-09)
.
It isn't an issue, more like a feature request, but I had several situations in which I need to do the following:
if vec.len() <= bit {
vec.resize(bit + 1, false);
}
vec.set(bit, true);
Am I missing a specific method that does exactly that? I would like to send a pull request if you think that can be useful.
On various architectures, where serde::Serialize
is not implemented for AtomicU8
, the serde_json tests fail.
error[E0277]: the trait bound `std::sync::atomic::AtomicU8: serde::Serialize` is not satisfied
--> tests/serdes.rs:18:13
|
18 | let json = serde_json::to_string(&bv).expect("cannot fail to serialize");
| ^^^^^^^^^^^^^^^^^^^^^ the trait `serde::Serialize` is not implemented for `std::sync::atomic::AtomicU8`
|
= note: required because of the requirements on the impl of `serde::Serialize` for `bitvec::prelude::BitVec<bitvec::order::Msb0, u8>`
= note: required by `serde_json::to_string`
Looking at serde, it looks like serde::Serialize is only implemented for architectures where it knows std::sync::atomic is know to exist, which is the following architectures:
Using target_arch
to conditionally compile the tests should likely fix the issue.
I am writing a program where i need to get a rolling window of 3 booleans. I saw the windows
method in the docs so my plan is to create a BitVec
from a u64 generated by the rand crate and iterate through windows of bits.
I did not see a example of creating a BitVec
from a regular unsigned number so i ask what is the best way to do that. Thanks
I've found myself wanting to dump some BitVecs into JSON; it'd be nice if BitVec implemented Serde's Serialize/Deserialize (likely behind a crate feature flag).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.