m4b / scroll Goto Github PK
View Code? Open in Web Editor NEWScroll - making scrolling through buffers fun since 2016
License: MIT License
Scroll - making scrolling through buffers fun since 2016
License: MIT License
u128
/i128
are missing from the list of primitives that get implementations of all the nice traits in scroll.
This fails:
> cargo test --all-features
failures:
---- src/lib.rs - readme_tests (line 391) stdout ----
error[E0277]: the trait bound `usize: FromCtx<_>` is not satisfied
--> src/lib.rs:401:30
|
12 | let foo = bytes.ioread::<usize>()?;
| ^^^^^ the trait `FromCtx<_>` is not implemented for `usize`
|
...
assert_eq
borrows the packed structs which is now Against The Rules
I'm working on parsing media streams for thirtythreeforty/neolink. Currently I'm using Nom for parsing but I'm shopping around because Nom parsers are difficult to read and write in my opinion.
The main kicker about the protocol I'm parsing is that basically every layer is stateful. This sucks for a variety of reasons but it typically means that I have to pass around a mutable context object as the struct is parsed.
Can scroll
support this use case? The current Context object feels more like a "configuration" object, in that it's passed by value, and doesn't really seem designed to be modified during parsing. In fact, now that I think about it, it wouldn't necessarily be possible to read different parts in parallel, again because of the need for a single mutable state.
The following rust code:
#[derive(Pread, Pwrite, IOread, IOwrite)]
struct Data {
id: u32,
x: u32,
y: u32,
z1: u32,
}
#[inline(never)]
fn make(input: &[u8]) -> Result<Data, scroll::Error> {
let data = input.pread_with::<Data>(0, LE);
data
}
fn main (){
let bytes = Vec::new();
make(&bytes).unwrap();
}
compiles to:
pread_example::make:
push rbp
mov rbp, rsp
mov r8d, 1
test rdx, rdx
je LBB19_4
cmp rdx, 4
jae LBB19_5
mov esi, 4
xor eax, eax
LBB19_3:
xor ecx, ecx
jmp LBB19_9
LBB19_4:
xor esi, esi
mov eax, 1
xor ecx, ecx
jmp LBB19_8
LBB19_5:
cmp rdx, 4
jne LBB19_11
xor ecx, ecx
mov esi, 4
LBB19_7:
mov eax, 1
LBB19_8:
LBB19_9:
LBB19_10:
or rax, rcx
mov dword, ptr, [rdi], r8d
mov dword, ptr, [rdi, +, 4], r9d
mov qword, ptr, [rdi, +, 8], rax
mov qword, ptr, [rdi, +, 16], rsi
mov qword, ptr, [rdi, +, 24], rdx
pop rbp
ret
LBB19_11:
lea r9, [rdx, -, 4]
cmp r9, 4
jae LBB19_13
LBB19_12:
mov esi, 4
xor eax, eax
xor ecx, ecx
mov rdx, r9
jmp LBB19_9
LBB19_13:
cmp rdx, 9
jb LBB19_18
lea r9, [rdx, -, 8]
cmp r9, 4
jb LBB19_12
cmp rdx, 13
jb LBB19_19
add rdx, -12
xor eax, eax
cmp rdx, 4
jae LBB19_20
mov esi, 4
jmp LBB19_3
LBB19_18:
xor ecx, ecx
mov esi, 8
jmp LBB19_7
LBB19_19:
xor ecx, ecx
mov esi, 12
jmp LBB19_7
LBB19_20:
mov r9d, dword, ptr, [rsi]
mov eax, dword, ptr, [rsi, +, 4]
mov ecx, dword, ptr, [rsi, +, 8]
shl rcx, 32
mov esi, dword, ptr, [rsi, +, 12]
xor r8d, r8d
jmp LBB19_10
It would be nice if make() compiled to a single length check and a single memcpy for all of the fields. We did some work on improving bincode's deserialization code for WebRender and I have a rough idea for how to fix this.
Hey! It's me again.
I'm continuing implementing CLR runtime in rust, and I wonder if there is some ideomatic way to express paddings between fields. Let's take an example:
I'm writing following code:
#[repr(C)]
#[derive(Debug)]
struct MetadataRoot<'a> {
pub signature: u32,
pub major_version: u16,
pub minor_version: u16,
_reserved: u32,
pub length: u32,
pub version: &'a str,
pub flags: u16,
pub streams: u16,
pub stream_headers: Vec<StreamHeader<'a>>
}
#[repr(C)]
#[derive(Debug)]
struct StreamHeader<'a> {
pub offset: u32,
pub size: u32,
pub name: &'a str
}
impl<'a> TryFromCtx<'a, Endian> for MetadataRoot<'a> {
type Error = scroll::Error;
type Size = usize;
// and the lifetime annotation on `&'a [u8]` here
fn try_from_ctx(src: &'a [u8], endian: Endian) -> Result<(Self, Self::Size), Self::Error> {
let offset = &mut 0;
let signature = src.gread_with(offset, endian)?;
let major_version = src.gread_with(offset, endian)?;
let minor_version = src.gread_with(offset, endian)?;
let reserved = src.gread_with(offset, endian)?;
let length = src.gread_with(offset, endian)?;
let version = src.gread(offset)?;
let padding = 4 - *offset % 4;
if padding < 4 {
*offset += padding;
}
let flags = src.gread_with(offset, endian)?;
let streams: u16 = src.gread_with(offset, endian)?;
let mut stream_headers = Vec::with_capacity(streams as usize);
for _ in 0..streams {
stream_headers.push( src.gread(offset)?);
let padding = 4 - *offset % 4;
if padding < 4 {
*offset += padding;
}
}
Ok((
Self {
signature,
major_version,
minor_version,
_reserved: reserved,
length,
version,
flags,
streams,
stream_headers
},
*offset,
))
}
}
impl<'a> TryFromCtx<'a, Endian> for StreamHeader<'a> {
type Error = scroll::Error;
type Size = usize;
// and the lifetime annotation on `&'a [u8]` here
fn try_from_ctx(src: &'a [u8], endian: Endian) -> Result<(Self, Self::Size), Self::Error> {
let offset = &mut 0;
let offset_field = src.gread_with(offset, endian)?;
let size = src.gread_with(offset, endian)?;
let name = src.gread(offset)?;
Ok((
Self {
offset: offset_field,
size,
name
},
*offset,
))
}
}
I'd like to ask if there is more ideomatic way to express such a structure, and if there isn't, maybe it may give you some thoughts about implementing it? It may be really useful.
P.S. Great lib, thank you for your work :)
from #7, we bounds check twice in Pread
, once in the TryFromCtx
, and again in FromCtx
(technically, it's everytime we access the byte array).
Consequently, there's room for some optimization here.
To be clear:
TryFromCtx
(break backwards compat, since we're below 1.0), remove offset logic from the trait, and move into Pread
itself, and make fully generic with Index trait bounds, and reduce it to the bounds check + assert, which we might be ok to live with, again, needs measuring.Anyway, as @goandylok suggests, some possible avenues:
Honestly, 1. is a "nuclear option" for me; I'm really hesitant to mark it unsafe, and would really require evidence (specifically benchmarks) to do so. Once it's unsafe, it's hard to put that genie back in the bottle; history has shown people are very unlikely to check bounds, even when its in their best interest, so this is definitely going to require some serious consideration. I might even prefer a UnsafeFromCtx, and have this isolated to just the Cread
trait, but that might be too much.
I also don't like the divergence from the std trait; why is FromCtx
unsafe? It just shouldn't fail, similar to From
trait in std, which isn't unsafe.
Anyway, those are some thoughts.
In anticipation of #47, the final remaining piece for working with arbitrary data safely is to figure out how to pread large objects without blowing the stack.
Most system languages accomplish this by boxing the object and manipulating it through the heap pointer.
I'm wondering if we can accomplish this in a semi-automated way, without compromising scroll's interesting type based reading/writing.
Doing this properly would allow us to round trip a large stack object within scroll.
Specifically, could we do something like:
#[derive(Pread, Pwrite, Copy, Clone)]
struct Big {
arr: [u8; 100000000],
}
let mut bytes = vec![42; 100000001];
let big = box bytes.pread_with::<Box<Big>>(0, LE)?; // this is boxed
// by the scroll Box impl, using unsafe unitialized memory
// which is only returned if the pread succeeds, so it remains safe?
let _ = bytes.pwrite_with(big, 0, LE)?; // Note: there is no `&*`,
// ideally scroll can deal with this case, though I don't know if its possible?
In worst case, we should be able to expose a new pread_with_box
method, that does the above, but pushes the box type information into the method name. This is fine, but i think its a more beautiful api design to allow the user to choose which type to read out via turbofish.
cc @philipc @lzutao @willglynn @luser
What you all think?
Ostensibly, it would be similar to scroll::Buffer
, but which wraps std::io::File
, and also wraps the pread
, pwrite
libc calls on linux/osx, (and fakes it on windows).
This way, similar to a Cursor/File, we can pass a Buffer/Pfile equally amongst generic functions if we like.
Fun fact: the docs for the gread
method do not even contain the letter "g" :)
Reads a value from self at offset with a default Ctx. For the primitive numeric values, this will read at the machine's endianness. Updates the offset
I was able to figure out the meaning of cread
and pread
, but gread
escapes me. What does the "g" stand for?
The example in the README on the scroll_derive repo doesn't work out of the box, and there aren't any docs on using derive in the scroll repo. It would be nice if there was an example to crib from somewhere!
In the interest of exploring pure safe alternatives to unsafe functions, this issue tracks all uses and potential replacements. The unsafe portions usually are quite similar, so only one entry will be made per function.
transmute
#88copy_nonoverlapping
CStr::from_bytes_with_nul_unchecked
First of all, I just wanted to say that I really like scroll
! The context system and type-based parsing and everything works perfectly for my use case.
However, there is one thing that I can't seem to figure out how to do - writing types into a container like a Vec
where the full size of the type is unknown.
Consider this (admittedly contrived) example:
pub struct MyType;
impl TryIntoCtx<'_> for MyType {
type Error = scroll::Error;
fn try_into_ctx(self, into: &mut [u8], _: ()) -> Result<usize, Self::Error> {
let offset = &mut 0;
if /* something that isn't statically determined */
{
into.gwrite_with(1u16, scroll::LE)?;
} else {
into.gwrite_with(1u32, scroll::LE)?;
}
Ok(*offset)
}
}
If I pre-allocated a let mut buf = [0u8; 4]
to cover the u32 case, then I'd be wasting space on the u16 case.
What I would love to do is something like this:
let mut buf = vec![];
buf.gwrite(MyType, 0)?;
where the Vec<u8>
only allocates as-needed.
Is this something that's possible, potentially with a new container trait?
Naturally, because the tests depends on std
. But ideally it should be working, or disabled when std
isn't enabled.
0.8.0 (scroll) + 0.8.1 (scroll_derive) results into failing build of goblin:
error[E0433]: failed to resolve. Could not find `export` in `scroll`
--> src/elf/program_header.rs:340:40
|
340 | #[cfg_attr(feature = "std", derive(Pread, Pwrite, SizeWith))]
| ^^^^^ Could not find `export` in `scroll`
[…]
I have a struct which has [u8; 8]
field and with Rust 1.47.0 it does not build anymore.
I am using scroll 0.10.1
.
56 | #[derive(Debug, Default, Pread, Pwrite)]
| ^^^^^ the trait `scroll::ctx::TryFromCtx<'_, _>` is not implemented for `[u8; 8]`
Hello everyone!
I am working on a project in which we have enums that whey (de)serialized add the Enum::Case(CaseStruct)
as a u8
or u16
before (de)serializing the CaseStruct
.
Currently we've implemented custom TryFromCtx
& TryIntoCtx
, but it would be nice to be able to derive it and add something like serde
enum tag
attribute and specify the type per case(maybe?!).
#[derive(Pread, Pwrite)]
#[scroll(tag = "u16")]
enum MyEnum {
#[scroll(tag_value = "0")]
Case(CaseStruct),
#[scroll(tag_value = "1")]
CaseTwo(CaseTwoStruct),
}
impl<'a> TryFromCtx<'a, ()> for SomeType { /* does one thing */ }
impl<'a> TryFromCtx<'a, scroll::Endian> for SomeType { /* does something completely different */ }
is a situation that is totally possible to come up, and I bet that in this sort of case the methods pread
and gread
would not function as it wouldn't be able to figure out which trait implementation to use.
pread_with
and gread_with
should still function because you can disambiguate the context by providing an argument.
However, really, this begs the question: did you intend for this to be possible in the first place? Do you still want it to be possible?
The code that handles array fields initializes its temporary array to [0; #size as usize]
, which doesn't work for an array of structs, like:
#[derive(Pread)]
struct A {
a: u32,
}
#[derive(Pread)]
struct B {
b: [A; 4],
}
You get:
error[E0308]: mismatched types
--> src/main.rs:9:10
|
9 | #[derive(Pread)]
| ^^^^^
| |
| expected struct `A`, found integral variable
| help: try using a variant of the expected type: `A(0)`
|
= note: expected type `A`
found type `{integer}`
error[E0277]: the trait bound `A: std::marker::Copy` is not satisfied
--> src/main.rs:9:10
|
9 | #[derive(Pread)]
| ^^^^^ the trait `std::marker::Copy` is not implemented for `A`
|
= note: the `Copy` trait is required because the repeated element will be copied
error: aborting due to 2 previous errors
The additional error about Copy
is because the [x; N]
array initializer syntax will copy the x
value.
I suppose the most straightforward way to fix both of these would be to change that code to just generate the array inline, like:
[ src.gread_with::<#whatever>(offset, ctx)?, src.gread_with::<#whatever>(offset, ctx)? ... ]
It is time for scroll to become 1.0
I believe the work started by @willglynn and seconded by others, namely, having pwrite take a reference is a good idea, is highly invasive breakage, and probably top priority item.
I may also swap the default endianness from ()
back to ctx::Endian
, unless someone protests highly?
I would also like to survey unsafe, remove commented code, and some other details.
So:
&str
pwrite?semi-random people who might be interested (feel free to ignore if busy, etc. :):
I haven't thought about whether its possible, but:
Adding support for
#[derive(Pread)]
pub struct Foo(u64);
would be super cool.
I.e., this error message sucks:
type is too big (68719476740) for 7828
I just realized tho, that we can "recover" the error; if we simplify our internal types to return TooBig, we can then have pread
"catch" TooBig
errors, and then recompute the offset, requested size, the offset requested at, and size of the bytes for us, all in a nice package
When you're trying to parse a file format filled with strings and other data with a specific endian, it may be convenient if you were able to derive the implementation for a struct
#[derive(Pread)]
struct Example<'a> {
// reads a bigendian u16
important: EndianWrapper<u16, byteorder::BE>,
// reads a u8 length field and then reads out a string from the following bytes
some_string: LengthData<&'a str, u8>
}
instead of having to manually implement TryFromCtx/TryIntoCtx like so:
struct Example<'a> {
important: u16,
some_string: &'a str
}
impl <'a> TryFromCtx<'a> for Example {
type Error = scroll::Error;
type Size = usize;
fn try_from_ctx(src: &'a [u8], _ctx: ()) -> Result<(Self, Self::Size), Self::Error> {
let important: u16 = src.pread_with(0, scroll::Endian::Big)?;
let length: usize = src.pread::<u8>(2)?.into();
let some_string: &'a str = src.pread_with(3, scroll::ctx::StrCtx::Length(length))?;
Ok((Example { important, some_string }, 3 + some_string.len()))
}
}
(Apologies if this doesn't actually work as-is, but you should get the idea.)
I bring this up because earlier today I created a couple of types intended for this purpose, including the LengthData
type I mentioned above, if you're interested: https://gist.github.com/kitlith/feae830bfd48558e487a87becbd39ed1
EndianWrapper
could probably be done similarly to the UTF16
wrapper included in the gist, just operating on a single piece of data instead of a slice.
this was a terrible mistake and should never have been done (by me).
Currently it doesn't seem possible to pread
out data which has a lifetime, because we can't impl TryFromCtx
correctly, e.g. the following data type with a lifetime bound:
struct Data<'a> {
name: &'a str,
id: u32,
}
should be allowed this impl for TryFromCtx
:
use scroll::ctx;
use scroll::{Error, Endian};
impl<'a> ctx::TryFromCtx for Data<'a> {
type Error = Error;
// iiuc the lifetime of Data is tied to the lifetime of src
// the lifetime of the str slice is tied to the lifetime of src
// so why doesn't this compile?
fn try_from_ctx<'b> (src: &'b [u8], (offset, le): (usize, Endian)) -> Result<Data<'b>, Self::Error> {
use scroll::Pread;
let name: &'b str = src.pread_slice(offset, 8)?;
let id = src.pread(offset+(name.len()), le)?;
Ok(Data { name: name, id: id })
}
}
But it returns the following error (notice also the suggested annotation is the one we've already used):
error[E0495]: cannot infer an appropriate lifetime for lifetime parameter 'b in generic type due to conflicting requirements
--> src/lib.rs:438:9
|
438 | fn try_from_ctx<'b> (src: &'b [u8], (offset, le): (usize, Endian)) -> Result<Data<'b>, Self::Error> {
| _________^ starting here...
439 | | use super::Pread;
440 | | let name: &'b str = src.pread_slice(offset, 8)?;
441 | | let id = src.pread(offset+(name.len()), le)?;
442 | | Ok(Data { name: name, id: id })
443 | | }
| |_________^ ...ending here
|
help: consider using an explicit lifetime parameter as shown: fn try_from_ctx<'b>(src: &'b [u8], (offset, le): (usize, Endian))
-> Result<Data<'b>, <Self>::Error>
--> src/lib.rs:438:9
|
438 | fn try_from_ctx<'b> (src: &'b [u8], (offset, le): (usize, Endian)) -> Result<Data<'b>, Self::Error> {
| _________^ starting here...
439 | | use super::Pread;
440 | | let name: &'b str = src.pread_slice(offset, 8)?;
441 | | let id = src.pread(offset+(name.len()), le)?;
442 | | Ok(Data { name: name, id: id })
443 | | }
| |_________^ ...ending here
error: aborting due to previous error
Instead it seems to require explicit lifetimes, or something like the following in order to work:
pub trait TryFromCtxL<'a, Ctx: Copy = (usize, super::Endian), This: ?Sized = [u8]> where Self: 'a + Sized {
type Error;
#[inline]
// note 'a is tied to trait 'a lifetime
fn try_from_ctx(from: &'a This, ctx: Ctx) -> Result<Self, Self::Error>;
}
impl<'a> ctx::TryFromCtxL<'a> for Data<'a> {
type Error = scroll::Error;
fn try_from_ctx (src: &'a [u8], ctx: (usize, scroll::Endian)) -> Result<Self, <Self>::Error> {
let offset = ctx.0;
let le = ctx.1;
let name: &str = src.pread_slice(offset, 8)?;
let id = src.pread(offset+(name.len()), le)?;
Ok(Data { name: name, id: id })
}
}
struct Foo(u32);
impl<'a> ctx::TryFromCtxL<'a> for Foo {
type Error = scroll::Error;
fn try_from_ctx (src: &[u8], ctx: (usize, scroll::Endian)) -> Result<Foo, <Self>::Error> {
let offset = ctx.0;
let le = ctx.1;
let id: u32 = src.pread(offset, le)?;
Ok(Foo(id))
}
}
let bytes = scroll::Buffer::new(b"UserName\x01\x02\x03\x04");
let data: Data = ctx::TryFromCtxL::try_from_ctx(bytes.as_slice(), (0, BE)).unwrap();
//let data = bytes.pread::<Data>(0, BE).unwrap();
assert_eq!(data.id, 0x01020304);
let foo: Foo = ctx::TryFromCtxL::try_from_ctx(bytes.as_slice(), (0, BE)).unwrap();
assert_eq!(foo.0, 1433625970);
I believe this can be solved by adding a lifetime to Pread
, and implementing the various Ctx
's in the above manner, but not sure if its the best course of action, primarily from a usability perspective.
Alternatively, we could consider a PreadExt
which does provide a lifetime bound and a single parsing entity for doing so, but this would require another Ctx
trait, etc.
There are some problems of the API design:
ctx
defaults to Endian
? I can't see how generic the type is, and I think it's meaningless to select any of its variants to be default when ctx is not necessary. The unit ()
may be a better choice.Sleb128
and Leb128
? I think we could split them into a new crate, with dependency on scroll
.FromCtx
and TryFromCtx
has a lot of code duplication in implementation.TryRefFrom
requires ctx implements Copy when RefFrom
don't.MeasureWith
semantic.pread
was actually a wrapper of trait TryFromCtx
after reading source code. Also, the name of trait IntoCtx
and FromCtx
is vague and misleading.So I suggest the following modifications:
DefaultCtx
and replace by ()
TryFromCtx
, TryIntoCtx
to TryRead
, TryWrite
, which seem more fit their purposes. (the main idea is to reduce priority of the concept Context
because this library is mainly to scroll
across data then Read and Write things, but not working around some context)FromCtx
IntoCtx
, and then use TryRead
and TryWrite
in cread
by unwrap()
or expect()
. This will make it easier to implement these traits.RefFrom
and TryRefFrom
TryRefInto
into TryRead
and TryWrite
. This is a very early though and maybe I will open an pull request latter on this. In brief, impl TryRefFrom for [usize]
is equal to impl TryFrom for &[usize]
.SizeWith
into two new traits WriteSize
and ReadSize
. WriteSize
takes a reference to self(&self) while ReadSize
takes a reference to slice.Endian
into scroll::ctx
, I think that's the right place for it.TryRead
, TryWrite
, WriteSize
, ReadSize
to root module. I've two reasons to do that. First, they are not actually contexts, when the Endian
is. Second, TryRead
is related to Pread
, Cread
family, and they should have the same priority. This is one of the reasons I get confused in the beginning.TryOffsetWith
=> TryOffset
. There is no ambiguity like that between pread
and pread_with
. Also, the reason for removing the word with
is same a (2), is to reduce the attention on context
.LE
, BE
when Endian::LE
is more idiomatic.LEB128
because it's just const NATIVE
NATIVE
and NETWORK
into scroll::ctx
Leb128
, which is ambiguous and just an alias to NATIVE
.Looking forward to your opinion and I think I can help with these refactoring.
---- tests::pread stdout ----
thread 'tests::pread' panicked at 'assertion failed: `(left == right)`
left: `61310`,
right: `32495`', src/lib.rs:225:8
Current working draft is on a branch, core_api, and basically lets you do this:
#[repr(packed)]
struct Bar {
foo: i32,
bar: u32,
}
impl scroll::ctx::FromCtx for Bar {
fn from_ctx(bytes: &[u8], ctx: scroll::Endian) -> Self {
use scroll::Cread;
Bar { foo: bytes.cread_with(0, ctx), bar: bytes.cread_with(4, ctx) }
}
}
#[test]
fn cread_api() {
use scroll::Cread;
let bytes = [0x01,0x00,0x00,0x00,0x00,0x00,0x00,0x00, 0xef,0xbe,0x00,0x00,];
let foo = bytes.cread::<usize>(0);
let bar = bytes.cread::<u32>(8);
assert_eq!(foo, 1);
assert_eq!(bar, 0xbeef);
}
#[test]
fn cread_api_customtype() {
use scroll::Cread;
let bytes = [0xff, 0xff, 0xff, 0xff, 0xef,0xbe,0xad,0xde,];
let bar = bytes.cread::<Bar>(0);
assert_eq!(bar.foo, -1);
assert_eq!(bar.bar, 0xdeadbeef);
}
All the basic numeric types are supported, and anything that implements FromCtx
(which is much easier to understand and implement than it's evil cousin, TryFromCtx
), will automatically be deseriablizable without result oriented api via:
let your_type = bytes.cread::<YourType>(offset)
as show above in the example.
Yay!
I'm partial to 2., since it doesn't require adding another trait as an import; but then again, perhaps who cares?
Of the maybe 1 or 2 people paying attention, any opinions welcome. 😎
The latest versions on crates.io are scroll 0.9.2 and scroll_derive 0.9.5 while on GitHub they are on 0.10.0 and 0.10.1.
Writing an implementation for TryFromCtx<'_, T>
where T != scroll::Endian
(including T == ()
) doesn't work if you want to use it in a struct where you're using #[derive(Pread)]
. This can be worked around for cases where you didn't care about the ctx anyway, but it doesn't work in cases where you actually wanted to use another ctx type -- but I'm not sure how you'd be able to tell.
Running target/debug/deps/readme-1392794e1e0e15a9
running 1 test
running 4 tests
test README.md - _::Scroll___cast_some_magic::Overview (line 37) ... FAILED
test README.md - _API (line 150) ... FAILED
test README.md - _API (line 128) ... FAILED
test README.md - Advanced_Uses (line 178) ... FAILED
failures:
---- README.md - _::Scroll___cast_some_magic::Overview (line 37) stdout ----
error[E0433]: failed to resolve: maybe a missing crate `scroll`?
--> README.md:68:9
|
32 | use scroll::ctx::*;
| ^^^^^^ maybe a missing crate `scroll`?
error[E0432]: unresolved import `scroll`
--> README.md:38:5
|
2 | use scroll::{ctx, Pread, LE};
| ^^^^^^ maybe a missing crate `scroll`?
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729597.27629/scroll-0.10.1/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:40:26
|
4 | fn parse() -> Result<(), scroll::Error> {
| ^^^^^^ can't find crate
error: aborting due to 4 previous errors
Some errors have detailed explanations: E0432, E0433, E0463.
For more information about an error, try `rustc --explain E0432`.
Couldn't compile the test.
---- README.md - _API (line 150) stdout ----
error[E0432]: unresolved import `scroll`
--> README.md:151:5
|
2 | use scroll::{IOwrite, LE, BE};
| ^^^^^^ maybe a missing crate `scroll`?
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729597.27629/scroll-0.10.1/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:154:29
|
5 | fn write_io() -> Result<(), scroll::Error> {
| ^^^^^^ can't find crate
error: aborting due to 3 previous errors
Some errors have detailed explanations: E0432, E0463.
For more information about an error, try `rustc --explain E0432`.
Couldn't compile the test.
---- README.md - _API (line 128) stdout ----
error[E0432]: unresolved import `scroll`
--> README.md:130:5
|
3 | use scroll::IOread;
| ^^^^^^ maybe a missing crate `scroll`?
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729597.27629/scroll-0.10.1/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:132:29
|
5 | fn parse_io() -> Result<(), scroll::Error> {
| ^^^^^^ can't find crate
error: aborting due to 3 previous errors
Some errors have detailed explanations: E0432, E0463.
For more information about an error, try `rustc --explain E0432`.
Couldn't compile the test.
---- README.md - Advanced_Uses (line 178) stdout ----
error[E0432]: unresolved import `scroll`
--> README.md:179:5
|
2 | use scroll::{ctx, Pread, BE, Endian};
| ^^^^^^ maybe a missing crate `scroll`?
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729597.27629/scroll-0.10.1/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:188:16
|
11 | type Error = scroll::Error;
| ^^^^^^ can't find crate
error: aborting due to 3 previous errors
Some errors have detailed explanations: E0432, E0463.
For more information about an error, try `rustc --explain E0432`.
Couldn't compile the test.
failures:
README.md - Advanced_Uses (line 178)
README.md - _::Scroll___cast_some_magic::Overview (line 37)
README.md - _API (line 128)
README.md - _API (line 150)
test result: FAILED. 0 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out
test readme_test ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Running target/debug/deps/readme-0aa9dd19900c77c6
running 1 test
running 5 tests
test README.md - _::Scroll___cast_some_magic::Deriving_ (line 86) ... FAILED
test README.md - _API (line 130) ... FAILED
test README.md - Advanced_Uses (line 184) ... FAILED
test README.md - _::Scroll___cast_some_magic::Overview (line 37) ... FAILED
test README.md - _API (line 154) ... FAILED
failures:
---- README.md - _::Scroll___cast_some_magic::Deriving_ (line 86) stdout ----
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:88:1
|
3 | extern crate scroll;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0463`.
Couldn't compile the test.
---- README.md - _API (line 130) stdout ----
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:131:1
|
2 | extern crate scroll;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0463`.
Couldn't compile the test.
---- README.md - Advanced_Uses (line 184) stdout ----
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:185:1
|
2 | extern crate scroll;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0463`.
Couldn't compile the test.
---- README.md - _::Scroll___cast_some_magic::Overview (line 37) stdout ----
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:38:1
|
2 | extern crate scroll;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0463`.
Couldn't compile the test.
---- README.md - _API (line 154) stdout ----
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:155:1
|
2 | extern crate scroll;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0463`.
Couldn't compile the test.
failures:
README.md - Advanced_Uses (line 184)
README.md - _::Scroll___cast_some_magic::Deriving_ (line 86)
README.md - _::Scroll___cast_some_magic::Overview (line 37)
README.md - _API (line 130)
README.md - _API (line 154)
test result: FAILED. 0 passed; 5 failed; 0 ignored; 0 measured; 0 filtered out
test readme_test ... FAILED
failures:
---- readme_test stdout ----
Running `"rustdoc" "--verbose" "--test" "-L" "/home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug" "-L" "/home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/deps" "--extern" "scroll=/home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib" "README.md"`
thread 'readme_test' panicked at 'Failed to run rustdoc tests on README.md!', tests/readme.rs:27:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
failures:
readme_test
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
error: test failed, to rerun pass '--test readme'
I wish I had thought of this earlier...
fn gread<'a, N: SizeWith<Ctx, Units = I> + TryFromCtx<'a, Ctx, <Self as Index<RangeFrom<I>>>::Output, Error = E, Size = I>>
This would be an overall gain across the board, for ergonomics and probably performance, since the client implement TryFromCtx doesn't have to return size, and the gread method would increment the offset for you...
Similarly for pwrite, it would write the data, then update with SizeWith; since SizeWith is implemented for all the basic types, and there is a deriving SizeWith, and it would enforce contraints on pread, this would have been the best solution all around I think...
It's a breaking change but I am very strongly inclined to remove them in next release.
Reasons:
usize
to pwrite
and it writes an 8 byte number (if you're on 64-bit machine), i.e., something coming from len
()`, when you actually meant to write a u32 (but forgot to cast). This has happened several times to me, and tracking down why the 4 extra bytes got written is evil.usize
, choose a u32
or u64
and serialize, so all machines can write the same on-disk/wire datatype.Unless someone provides compelling reasons to not remove, I will do so; please let me know if you really need this feature!
Lots of types implement TryIntoCtx but not IntoCtx in goblin (like elf::ProgramHeader). It'd be nice if those types would be usable with IOWrite. Furthermore, this would grant the user more control over error handling.
Maybe something like:
fn iowrite_with_try<N>(&mut self, n: N, ctx: Ctx) -> Result<(), N::Error>
where
N: scroll::ctx::SizeWith<Ctx, Units = usize> + scroll::ctx::TryIntoCtx<Ctx>>
N::Error: From<std::io::Error>
{
let mut buf = [0u8; 256];
let size = N::size_with(&ctx);
let buf = &mut buf[0..size];
n.try_into_ctx(buf, ctx)?;
self.write_all(buf)?;
Ok(())
}
Consider this structure:
#[derive(Pwrite)]
pub struct SymbolTable<'a> {
symbols: &'a [u8],
}
(extracted from Goblin codebase)
This induces proc-macro derive produced unparseable tokens
, on rustc 1.68.1.
I don't know what happens here.
I've stumbled over this before while writing functions that were generic over TryIntoCtx
and SizeWith
--the constraints need to explicitly call out TryIntoCtx<Size=usize>
or SizeWith<Units=usize>
in order to be able to actually use the sizes for anything meaningful.
Both MeasureWith
and SizeWith
have an associated Units
type:
https://docs.rs/scroll/0.9.0/scroll/ctx/trait.MeasureWith.html#associatedtype.Units
https://docs.rs/scroll/0.9.0/scroll/ctx/trait.SizeWith.html#associatedtype.Units
And both TryFromCtx
and TryIntoCtx
have an associated Size
type:
https://docs.rs/scroll/0.9.0/scroll/ctx/trait.TryFromCtx.html#associatedtype.Size
https://docs.rs/scroll/0.9.0/scroll/ctx/trait.TryIntoCtx.html#associatedtype.Size
I'm sure you had an intended use case for this being generic when you wrote these, but is it actually something that has proven useful in practice? grepping through the scroll
, scroll_derive
and goblin
repositories shows that there aren't any implementations of these traits using anything but usize
, and most usage in goblin
is via the derives, which all provide implementations where the types are usize
.
I know it would be a breaking change, but removing these associated types and just using usize
everywhere seems like it would simplify things a bit.
The only counter-argument I can come up with on my own is that perhaps you might want to have types that have sizes that don't fit in 32 bits but you want to use them with these traits in 32-bit code, but this feels a bit contrived and also would suggest that just using u64
instead of usize
everywhere would be the answer.
The trait signatures are:
/// Writes `Self` into `This` using the context `Ctx`
pub trait IntoCtx<Ctx: Copy = (), This: ?Sized = [u8]>: Sized {
fn into_ctx(self, &mut This, ctx: Ctx);
}
/// Tries to write `Self` into `This` using the context `Ctx`
pub trait TryIntoCtx<Ctx: Copy = (), This: ?Sized = [u8]>: Sized {
type Error;
fn try_into_ctx(self, &mut This, ctx: Ctx) -> Result<usize, Self::Error>;
}
These traits mirror Into
which consumes self
. However, these might be the wrong function signatures for how scroll
is used in practice; both traits are primarily used on borrowed values.
scroll
implements IntoCtx
and TryIntoCtx
on borrowed primitive types, suggesting that [Try]IntoCtx
needn't consume its value after all.
scroll_derive
generates code implementing IntoCtx
and TryIntoCtx
on Self
by borrowing and delegating to the implementation on &'a Self
.
Should these traits take &self
instead of self
?
@luser when i do cargo test I see readme failures with:
failures:
---- README.md - std__io (line 130) stdout ----
error: extern location for scroll does not exist: /home/m4b/projects/scroll/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:131:1
|
2 | extern crate scroll;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate
thread 'README.md - std__io (line 130)' panicked at 'couldn't compile the test', librustdoc/test.rs:333:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.
---- README.md - _::Scroll___cast_some_magic::Deriving_ (line 86) stdout ----
error: extern location for scroll does not exist: /home/m4b/projects/scroll/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:88:1
|
3 | extern crate scroll;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate
thread 'README.md - _::Scroll___cast_some_magic::Deriving_ (line 86)' panicked at 'couldn't compile the test', librustdoc/test.rs:333:13
---- README.md - _::Scroll___cast_some_magic::Overview (line 37) stdout ----
error: extern location for scroll does not exist: /home/m4b/projects/scroll/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:38:1
|
2 | extern crate scroll;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate
thread 'README.md - _::Scroll___cast_some_magic::Overview (line 37)' panicked at 'couldn't compile the test', librustdoc/test.rs:333:13
---- README.md - Advanced_Uses (line 184) stdout ----
error: extern location for scroll does not exist: /home/m4b/projects/scroll/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:185:1
|
2 | extern crate scroll;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate
thread 'README.md - Advanced_Uses (line 184)' panicked at 'couldn't compile the test', librustdoc/test.rs:333:13
---- README.md - std__io (line 154) stdout ----
error: extern location for scroll does not exist: /home/m4b/projects/scroll/target/debug/libscroll.rlib
error[E0463]: can't find crate for `scroll`
--> README.md:155:1
|
2 | extern crate scroll;
| ^^^^^^^^^^^^^^^^^^^^ can't find crate
thread 'README.md - std__io (line 154)' panicked at 'couldn't compile the test', librustdoc/test.rs:333:13
failures:
README.md - Advanced_Uses (line 184)
README.md - _::Scroll___cast_some_magic::Deriving_ (line 86)
README.md - _::Scroll___cast_some_magic::Overview (line 37)
README.md - std__io (line 130)
README.md - std__io (line 154)
test result: FAILED. 0 passed; 5 failed; 0 ignored; 0 measured; 0 filtered out
test readme_test ... FAILED
failures:
---- readme_test stdout ----
Running `"rustdoc" "--verbose" "--test" "-L" "/home/m4b/projects/scroll/target/debug" "-L" "/home/m4b/projects/scroll/target/debug/deps" "--extern" "scroll=/home/m4b/projects/scroll/target/debug/libscroll.rlib" "README.md"`
thread 'readme_test' panicked at 'Failed to run rustdoc tests on README.md!', tests/readme.rs:27:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failures:
readme_test
I don't really understand what this means? But the CI seems to be passing; if I update the README, do I have do something special to get the tests to pass or?
It goes to https://github.com/m4b/scroll_derive
e.g., Lread, and what in particular this api will look like. After converting all of goblin to use scroll, I no longer have any need whatsoever for a mutable Reader, and in fact it's presence makes any easy parallelism unnecessarily difficult, so not sure it's worthwhile?
I think that custom Ctx
will be more common than custom Error
, and hence first param should be Ctx
and not Error
to make the API more ergonomic.
Cranelift currently supports compiling with Rust 1.25.0, and it uses Faerie which uses Scroll. The Scroll 0.9.1 update doesn't compile under Rust 1.25.0, since it uses i128, added in #32, which isn't supported in that version of Rust.
error[E0658]: 128-bit type is unstable (see issue #35118)
--> /home/me/.cargo/registry/src/github.com-1ecc6299db9ec823/scroll-0.9.1/src/ctx.rs:223:38
|
223 | impl<'a> FromCtx<Endian> for $typ {
| ^^^^
They're currently omitted due to lack of time
https://github.com/m4b/scroll/blob/master/src/lib.rs#L55
test src/lib.rs - (line 55) ... FAILED
(...)
failures:
---- src/lib.rs - (line 55) stdout ----
Test executable failed (exit code 101).
stderr:
thread 'main' panicked at 'assertion failed: `(left == right)`
left: `173`, # AD
right: `190`', src/lib.rs:22:1 # BE
All other tests and doctests pass. So it looks like the example data that's defined for the example is wrong / typo'd?
It's hard for me to tell with assertion is actually triggering the panic, because the line numbers inside the doctest are messed up (probably due to the #cfgs), but I think it's this one: assert_eq!(byte, 0xbe);
(matching the 190 / BE
"right" side in the assertion)
I think I'm the only person who uses it and only in goblin and I can't even remember why anymore.
E.g., this is already going to be a problem: m4b/goblin#105
Need to figure out path forward for packed structs (and if there is any work that scroll needs to do to ameliorate the situation)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.