Giter VIP home page Giter VIP logo

scroll's Introduction

Actions crates.io version

Scroll - cast some magic

         _______________
    ()==(              (@==()
         '______________'|
           |             |
           |   ἀρετή     |
         __)_____________|
    ()==(               (@==()
         '--------------'

Documentation

https://docs.rs/scroll

Usage

Add to your Cargo.toml

[dependencies]
scroll = "0.11"

Overview

Scroll implements several traits for read/writing generic containers (byte buffers are currently implemented by default). Most familiar will likely be the Pread trait, which at its basic takes an immutable reference to self, an immutable offset to read at, (and a parsing context, more on that later), and then returns the deserialized value.

Because self is immutable, all reads can be performed in parallel and hence are trivially parallelizable.

A simple example demonstrates its flexibility:

use scroll::{ctx, Pread, LE};

fn main() -> Result<(), scroll::Error> {
    let bytes: [u8; 4] = [0xde, 0xad, 0xbe, 0xef];

    // reads a u32 out of `b` with the endianness of the host machine, at offset 0, turbofish-style
    let number: u32 = bytes.pread::<u32>(0)?;
    // ...or a byte, with type ascription on the binding.
    let byte: u8 = bytes.pread(0)?;

    //If the type is known another way by the compiler, say reading into a struct field, we can omit the turbofish, and type ascription altogether!

    // If we want, we can explicitly add a endianness to read with by calling `pread_with`.
    // The following reads a u32 out of `b` with Big Endian byte order, at offset 0
    let be_number: u32 = bytes.pread_with(0, scroll::BE)?;
    // or a u16 - specify the type either on the variable or with the beloved turbofish
    let be_number2 = bytes.pread_with::<u16>(2, scroll::BE)?;

    // Scroll has core friendly errors (no allocation). This will have the type `scroll::Error::BadOffset` because it tried to read beyond the bound
    let byte: scroll::Result<i64> = bytes.pread(0);

    // Scroll is extensible: as long as the type implements `TryWithCtx`, then you can read your type out of the byte array!

    // We can parse out custom datatypes, or types with lifetimes
    // if they implement the conversion trait `TryFromCtx`; here we parse a C-style \0 delimited &str (safely)
    let hello: &[u8] = b"hello_world\0more words";
    let hello_world: &str = hello.pread(0)?;
    assert_eq!("hello_world", hello_world);

    // ... and this parses the string if its space separated!
    use scroll::ctx::*;
    let spaces: &[u8] = b"hello world some junk";
    let world: &str = spaces.pread_with(6, StrCtx::Delimiter(SPACE))?;
    assert_eq!("world", world);
    Ok(())
}

Deriving Pread and Pwrite

Scroll implements a custom derive that can provide Pread and Pwrite implementations for your structs.

use scroll::{Pread, Pwrite, BE};

#[derive(Pread, Pwrite)]
struct Data {
    one: u32,
    two: u16,
    three: u8,
}

fn main() -> Result<(), scroll::Error> {
    let bytes: [u8; 7] = [0xde, 0xad, 0xbe, 0xef, 0xfa, 0xce, 0xff];
    // Read a single `Data` at offset zero in big-endian byte order.
    let data: Data = bytes.pread_with(0, BE)?;
    assert_eq!(data.one, 0xdeadbeef);
    assert_eq!(data.two, 0xface);
    assert_eq!(data.three, 0xff);

    // Write it back to a buffer
    let mut out: [u8; 7] = [0; 7];
    out.pwrite_with(data, 0, BE)?;
    assert_eq!(bytes, out);
    Ok(())
}

This feature is not enabled by default, you must enable the derive feature in Cargo.toml to use it:

[dependencies]
scroll = { version = "0.10", features = ["derive"] }

std::io API

Scroll can also read/write simple types from a std::io::Read or std::io::Write implementor. The built-in numeric types are taken care of for you. If you want to read a custom type, you need to implement the FromCtx (how to parse) and SizeWith (how big the parsed thing will be) traits. You must compile with default features. For example:

use std::io::Cursor;
use scroll::IOread;

fn main() -> Result<(), scroll::Error> {
    let bytes_ = [0x01,0x00,0x00,0x00,0x00,0x00,0x00,0x00, 0xef,0xbe,0x00,0x00,];
    let mut bytes = Cursor::new(bytes_);

    // this will bump the cursor's Seek
    let foo = bytes.ioread::<usize>()?;
    // ..ditto
    let bar = bytes.ioread::<u32>()?;
    Ok(())
}

Similarly, we can write to anything that implements std::io::Write quite naturally:

use scroll::{IOwrite, LE, BE};
use std::io::{Write, Cursor};

fn main() -> Result<(), scroll::Error> {
    let mut bytes = [0x0u8; 10];
    let mut cursor = Cursor::new(&mut bytes[..]);
    cursor.write_all(b"hello")?;
    cursor.iowrite_with(0xdeadbeef as u32, BE)?;
    assert_eq!(cursor.into_inner(), [0x68, 0x65, 0x6c, 0x6c, 0x6f, 0xde, 0xad, 0xbe, 0xef, 0x0]);
    Ok(())
}

Advanced Uses

Scroll is designed to be highly configurable - it allows you to implement various context (Ctx) sensitive traits, which then grants the implementor automatic uses of the Pread and/or Pwrite traits.

For example, suppose we have a datatype and we want to specify how to parse or serialize this datatype out of some arbitrary byte buffer. In order to do this, we need to provide a TryFromCtx impl for our datatype.

In particular, if we do this for the [u8] target, using the convention (usize, YourCtx), you will automatically get access to calling pread_with::<YourDatatype> on arrays of bytes.

use scroll::{ctx, Pread, BE, Endian};

struct Data<'a> {
  name: &'a str,
  id: u32,
}

// note the lifetime specified here
impl<'a> ctx::TryFromCtx<'a, Endian> for Data<'a> {
  type Error = scroll::Error;
  // and the lifetime annotation on `&'a [u8]` here
  fn try_from_ctx (src: &'a [u8], endian: Endian)
    -> Result<(Self, usize), Self::Error> {
    let offset = &mut 0;
    let name = src.gread::<&str>(offset)?;
    let id = src.gread_with(offset, endian)?;
    Ok((Data { name: name, id: id }, *offset))
  }
}

fn main() -> Result<(), scroll::Error> {
    let bytes = b"UserName\x00\x01\x02\x03\x04";
    let data = bytes.pread_with::<Data>(0, BE)?;
    assert_eq!(data.id, 0x01020304);
    assert_eq!(data.name.to_string(), "UserName".to_string());
    Ok(())
}

Please see the official documentation, or a simple example for more.

Contributing

Any ideas, thoughts, or contributions are welcome!

scroll's People

Contributors

andylokandy avatar atouchet avatar cuviper avatar dequbed avatar emilio avatar frostie314159 avatar hdhoang avatar jan-auer avatar le-jzr avatar llogiq avatar luser avatar m4b avatar messense avatar michael-f-bryan avatar nickbclifford avatar nyurik avatar philipc avatar raitobezarius avatar sunfishcode avatar systemcluster avatar tesuji avatar whitequark avatar willglynn avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

scroll's Issues

Packed structs and Scroll

E.g., this is already going to be a problem: m4b/goblin#105

Need to figure out path forward for packed structs (and if there is any work that scroll needs to do to ameliorate the situation)

derive(Pread) doesn't work with fields containing an array of structs

The code that handles array fields initializes its temporary array to [0; #size as usize], which doesn't work for an array of structs, like:

#[derive(Pread)]
struct A {
    a: u32,
}

#[derive(Pread)]
struct B {
    b: [A; 4],
}

You get:

error[E0308]: mismatched types
 --> src/main.rs:9:10
  |
9 | #[derive(Pread)]
  |          ^^^^^
  |          |
  |          expected struct `A`, found integral variable
  |          help: try using a variant of the expected type: `A(0)`
  |
  = note: expected type `A`
             found type `{integer}`

error[E0277]: the trait bound `A: std::marker::Copy` is not satisfied
 --> src/main.rs:9:10
  |
9 | #[derive(Pread)]
  |          ^^^^^ the trait `std::marker::Copy` is not implemented for `A`
  |
  = note: the `Copy` trait is required because the repeated element will be copied

error: aborting due to 2 previous errors

The additional error about Copy is because the [x; N] array initializer syntax will copy the x value.

I suppose the most straightforward way to fix both of these would be to change that code to just generate the array inline, like:
[ src.gread_with::<#whatever>(offset, ctx)?, src.gread_with::<#whatever>(offset, ctx)? ... ]

derive(Pread) in structs with elements that don't use scroll::Endian as ctx

Writing an implementation for TryFromCtx<'_, T> where T != scroll::Endian (including T == ()) doesn't work if you want to use it in a struct where you're using #[derive(Pread)]. This can be worked around for cases where you didn't care about the ctx anyway, but it doesn't work in cases where you actually wanted to use another ctx type -- but I'm not sure how you'd be able to tell.

Pread extra bounds checks

from #7, we bounds check twice in Pread, once in the TryFromCtx, and again in FromCtx (technically, it's everytime we access the byte array).

Consequently, there's room for some optimization here.

To be clear:

  1. I'm definitely for optimization at this point, as the crate is becoming mature and I'd like to hit 1.0 soon, and well optimized numeric readers would be great
  2. I'd like to see some basic benchmarks as to whether these bounds checks are actually worth optimizing.
  3. It might be better to clean up TryFromCtx (break backwards compat, since we're below 1.0), remove offset logic from the trait, and move into Pread itself, and make fully generic with Index trait bounds, and reduce it to the bounds check + assert, which we might be ok to live with, again, needs measuring.

Anyway, as @goandylok suggests, some possible avenues:

  1. Mark FromCtx and cread‘s members unsafe to give away the responsibility of bound check to call site.
  2. Make FromCtx to be a wrapper of TryFromCtx by calling expect() on TryFromCtx's result.

Honestly, 1. is a "nuclear option" for me; I'm really hesitant to mark it unsafe, and would really require evidence (specifically benchmarks) to do so. Once it's unsafe, it's hard to put that genie back in the bottle; history has shown people are very unlikely to check bounds, even when its in their best interest, so this is definitely going to require some serious consideration. I might even prefer a UnsafeFromCtx, and have this isolated to just the Cread trait, but that might be too much.

I also don't like the divergence from the std trait; why is FromCtx unsafe? It just shouldn't fail, similar to From trait in std, which isn't unsafe.

Anyway, those are some thoughts.

Writing into a dynamic container

First of all, I just wanted to say that I really like scroll! The context system and type-based parsing and everything works perfectly for my use case.

However, there is one thing that I can't seem to figure out how to do - writing types into a container like a Vec where the full size of the type is unknown.

Consider this (admittedly contrived) example:

pub struct MyType;
impl TryIntoCtx<'_> for MyType {
    type Error = scroll::Error;

    fn try_into_ctx(self, into: &mut [u8], _: ()) -> Result<usize, Self::Error> {
        let offset = &mut 0;

        if /* something that isn't statically determined */
        {
            into.gwrite_with(1u16, scroll::LE)?;
        } else {
            into.gwrite_with(1u32, scroll::LE)?;
        }
        
        Ok(*offset)
    }
}

If I pre-allocated a let mut buf = [0u8; 4] to cover the u32 case, then I'd be wasting space on the u16 case.

What I would love to do is something like this:

let mut buf = vec![];
buf.gwrite(MyType, 0)?;

where the Vec<u8> only allocates as-needed.

Is this something that's possible, potentially with a new container trait?

Add lifetimes to `Pread`?

Currently it doesn't seem possible to pread out data which has a lifetime, because we can't impl TryFromCtx correctly, e.g. the following data type with a lifetime bound:

 struct Data<'a> {
   name: &'a str,
   id: u32,
 }

should be allowed this impl for TryFromCtx:

use scroll::ctx;
use scroll::{Error, Endian};

impl<'a> ctx::TryFromCtx for Data<'a> {
    type Error = Error;
    // iiuc the lifetime of Data is tied to the lifetime of src
    // the lifetime of the str slice is tied to the lifetime of src
    // so why doesn't this compile?
    fn try_from_ctx<'b> (src: &'b [u8], (offset, le): (usize, Endian)) -> Result<Data<'b>, Self::Error> {
        use scroll::Pread;
        let name: &'b str = src.pread_slice(offset, 8)?;
        let id = src.pread(offset+(name.len()), le)?;
        Ok(Data { name: name, id: id })
    }
}

But it returns the following error (notice also the suggested annotation is the one we've already used):

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter 'b in generic type due to conflicting requirements
   --> src/lib.rs:438:9
    |
438 |           fn try_from_ctx<'b> (src: &'b [u8], (offset, le): (usize, Endian)) -> Result<Data<'b>, Self::Error> {
    |  _________^ starting here...
439 | |             use super::Pread;
440 | |             let name: &'b str = src.pread_slice(offset, 8)?;
441 | |             let id = src.pread(offset+(name.len()), le)?;
442 | |             Ok(Data { name: name, id: id })
443 | |         }
    | |_________^ ...ending here
    |
help: consider using an explicit lifetime parameter as shown: fn try_from_ctx<'b>(src: &'b [u8], (offset, le): (usize, Endian))
 -> Result<Data<'b>, <Self>::Error>
   --> src/lib.rs:438:9
    |
438 |           fn try_from_ctx<'b> (src: &'b [u8], (offset, le): (usize, Endian)) -> Result<Data<'b>, Self::Error> {
    |  _________^ starting here...
439 | |             use super::Pread;
440 | |             let name: &'b str = src.pread_slice(offset, 8)?;
441 | |             let id = src.pread(offset+(name.len()), le)?;
442 | |             Ok(Data { name: name, id: id })
443 | |         }
    | |_________^ ...ending here

error: aborting due to previous error

Instead it seems to require explicit lifetimes, or something like the following in order to work:

pub trait TryFromCtxL<'a, Ctx: Copy = (usize, super::Endian), This: ?Sized = [u8]> where Self: 'a + Sized {
    type Error;
    #[inline]
    // note 'a is tied to trait 'a lifetime
    fn try_from_ctx(from: &'a This, ctx: Ctx) -> Result<Self, Self::Error>;
}

 impl<'a> ctx::TryFromCtxL<'a> for Data<'a> {
   type Error = scroll::Error;
   fn try_from_ctx (src: &'a [u8], ctx: (usize, scroll::Endian)) -> Result<Self, <Self>::Error> {
     let offset = ctx.0;
     let le = ctx.1;
     let name: &str = src.pread_slice(offset, 8)?;
     let id = src.pread(offset+(name.len()), le)?;
     Ok(Data { name: name, id: id })
   }
 }

 struct Foo(u32);
 impl<'a> ctx::TryFromCtxL<'a> for Foo {
   type Error = scroll::Error;
   fn try_from_ctx (src: &[u8], ctx: (usize, scroll::Endian)) -> Result<Foo, <Self>::Error> {
     let offset = ctx.0;
     let le = ctx.1;
     let id: u32 = src.pread(offset, le)?;
     Ok(Foo(id))
   }
 }

 let bytes = scroll::Buffer::new(b"UserName\x01\x02\x03\x04");
 let data: Data = ctx::TryFromCtxL::try_from_ctx(bytes.as_slice(), (0, BE)).unwrap();
 //let data = bytes.pread::<Data>(0, BE).unwrap();
 assert_eq!(data.id, 0x01020304);
 let foo: Foo = ctx::TryFromCtxL::try_from_ctx(bytes.as_slice(), (0, BE)).unwrap();
 assert_eq!(foo.0, 1433625970);

I believe this can be solved by adding a lifetime to Pread, and implementing the various Ctx's in the above manner, but not sure if its the best course of action, primarily from a usability perspective.

Alternatively, we could consider a PreadExt which does provide a lifetime bound and a single parsing entity for doing so, but this would require another Ctx trait, etc.

Refactor Pread/Gread to use Index generics

  1. This will have the effect of making the traits completely generic without need for impls
  2. I believe this will allow TryFromCtx to remove offsetting logic
  3. I believe this will allow complete library side implementations. E.g., as long as the bounds are satisfied, one could define custom structures to Pread out of !

What does the "g" in "gread" stand for?

Fun fact: the docs for the gread method do not even contain the letter "g" :)

Reads a value from self at offset with a default Ctx. For the primitive numeric values, this will read at the machine's endianness. Updates the offset

I was able to figure out the meaning of cread and pread, but gread escapes me. What does the "g" stand for?

Tracking issue for replacing unsafe code.

In the interest of exploring pure safe alternatives to unsafe functions, this issue tracks all uses and potential replacements. The unsafe portions usually are quite similar, so only one entry will be made per function.

  • transmute #88
  • copy_nonoverlapping
  • CStr::from_bytes_with_nul_unchecked

derive newtypes?

I haven't thought about whether its possible, but:

Adding support for

#[derive(Pread)]
pub struct Foo(u64);

would be super cool.

Scroll 0.9.1 doesn't compile under Rust 1.25.0 due to i128

Cranelift currently supports compiling with Rust 1.25.0, and it uses Faerie which uses Scroll. The Scroll 0.9.1 update doesn't compile under Rust 1.25.0, since it uses i128, added in #32, which isn't supported in that version of Rust.

Sample compilation error
error[E0658]: 128-bit type is unstable (see issue #35118)
   --> /home/me/.cargo/registry/src/github.com-1ecc6299db9ec823/scroll-0.9.1/src/ctx.rs:223:38
    |
223 |         impl<'a> FromCtx<Endian> for $typ {
    |                                      ^^^^


I can work around this for now by just fixing the dependency to scroll 0.9.0, and I'll look into other options on Cranelift side, but I'm interested if you have any ideas here too.

Docs mention lost Gread and Gwrite traits

It appears that the trait Gread was demoted to oblivion in 6054557; however, various parts of the API documentation still mention it. Here are two examples:

scroll/src/lib.rs

Lines 17 to 21 in e94a84d

//! There are 3 traits for reading that you can import:
//!
//! 1. [Pread](trait.Pread.html), for reading (immutable) data at an offset;
//! 2. [Gread](trait.Gread.html), for reading data at an offset which automatically gets incremented by the size;
//! 3. [IOread](trait.IOread.html), for reading _simple_ data out of a `std::io::Read` based interface, e.g., a stream. (**Note**: only available when compiled with `std`)

/// Gets the size of `Self` with a `Ctx`, and in `Self::Units`. Implementors can then call `Gread` related functions

IOWrite using TryIntoCtx

Lots of types implement TryIntoCtx but not IntoCtx in goblin (like elf::ProgramHeader). It'd be nice if those types would be usable with IOWrite. Furthermore, this would grant the user more control over error handling.

Maybe something like:

fn iowrite_with_try<N>(&mut self, n: N, ctx: Ctx) -> Result<(), N::Error>
where
    N: scroll::ctx::SizeWith<Ctx, Units = usize> + scroll::ctx::TryIntoCtx<Ctx>>
    N::Error: From<std::io::Error>
{
    let mut buf = [0u8; 256];
    let size = N::size_with(&ctx);
    let buf = &mut buf[0..size];
    n.try_into_ctx(buf, ctx)?;
    self.write_all(buf)?;
    Ok(())
}

Tutorial: how to add paddings

Hey! It's me again.

I'm continuing implementing CLR runtime in rust, and I wonder if there is some ideomatic way to express paddings between fields. Let's take an example:

image

I'm writing following code:

#[repr(C)]
#[derive(Debug)]
struct MetadataRoot<'a> {
    pub signature: u32,
    pub major_version: u16,
    pub minor_version: u16,
    _reserved: u32,
    pub length: u32,
    pub version: &'a str,
    pub flags: u16,
    pub streams: u16,
    pub stream_headers: Vec<StreamHeader<'a>>
}

#[repr(C)]
#[derive(Debug)]
struct StreamHeader<'a> {
    pub offset: u32,
    pub size: u32,
    pub name: &'a str
}

impl<'a> TryFromCtx<'a, Endian> for MetadataRoot<'a> {
    type Error = scroll::Error;
    type Size = usize;
    // and the lifetime annotation on `&'a [u8]` here
    fn try_from_ctx(src: &'a [u8], endian: Endian) -> Result<(Self, Self::Size), Self::Error> {
        let offset = &mut 0;
        let signature = src.gread_with(offset, endian)?;
        let major_version = src.gread_with(offset, endian)?;
        let minor_version = src.gread_with(offset, endian)?;
        let reserved = src.gread_with(offset, endian)?;
        let length = src.gread_with(offset, endian)?;
        let version = src.gread(offset)?;
        let padding = 4 - *offset % 4;
        if padding < 4 {
            *offset += padding;
        }
        let flags = src.gread_with(offset, endian)?;
        let streams: u16 = src.gread_with(offset, endian)?;
        let mut stream_headers = Vec::with_capacity(streams as usize);
        for _ in 0..streams {
            stream_headers.push( src.gread(offset)?);
            let padding = 4 - *offset % 4;
            if padding < 4 {
                *offset += padding;
            }
        }
        Ok((
            Self {
                signature,
                major_version,
                minor_version,
                _reserved: reserved,
                length,
                version,
                flags,
                streams,
                stream_headers
            },
            *offset,
        ))
    }
}

impl<'a> TryFromCtx<'a, Endian> for StreamHeader<'a> {
    type Error = scroll::Error;
    type Size = usize;
    // and the lifetime annotation on `&'a [u8]` here
    fn try_from_ctx(src: &'a [u8], endian: Endian) -> Result<(Self, Self::Size), Self::Error> {
        let offset = &mut 0;
        let offset_field = src.gread_with(offset, endian)?;
        let size = src.gread_with(offset, endian)?;
        let name = src.gread(offset)?;
        Ok((
            Self {
                offset: offset_field,
                size,
                name
            },
            *offset,
        ))
    }
}

I'd like to ask if there is more ideomatic way to express such a structure, and if there isn't, maybe it may give you some thoughts about implementing it? It may be really useful.

P.S. Great lib, thank you for your work :)

scroll probably needs to depend on exact scroll_derive version

0.8.0 (scroll) + 0.8.1 (scroll_derive) results into failing build of goblin:

error[E0433]: failed to resolve. Could not find `export` in `scroll`
   --> src/elf/program_header.rs:340:40
    |
340 |     #[cfg_attr(feature = "std", derive(Pread, Pwrite, SizeWith))]
    |                                        ^^^^^ Could not find `export` in `scroll`
[]

cargo test readme fails locally

@luser when i do cargo test I see readme failures with:

failures:

---- README.md - std__io (line 130) stdout ----
error: extern location for scroll does not exist: /home/m4b/projects/scroll/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:131:1
  |
2 | extern crate scroll;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

thread 'README.md - std__io (line 130)' panicked at 'couldn't compile the test', librustdoc/test.rs:333:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- README.md - _::Scroll___cast_some_magic::Deriving_ (line 86) stdout ----
error: extern location for scroll does not exist: /home/m4b/projects/scroll/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:88:1
  |
3 | extern crate scroll;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

thread 'README.md - _::Scroll___cast_some_magic::Deriving_ (line 86)' panicked at 'couldn't compile the test', librustdoc/test.rs:333:13

---- README.md - _::Scroll___cast_some_magic::Overview (line 37) stdout ----
error: extern location for scroll does not exist: /home/m4b/projects/scroll/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:38:1
  |
2 | extern crate scroll;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

thread 'README.md - _::Scroll___cast_some_magic::Overview (line 37)' panicked at 'couldn't compile the test', librustdoc/test.rs:333:13

---- README.md - Advanced_Uses (line 184) stdout ----
error: extern location for scroll does not exist: /home/m4b/projects/scroll/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:185:1
  |
2 | extern crate scroll;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

thread 'README.md - Advanced_Uses (line 184)' panicked at 'couldn't compile the test', librustdoc/test.rs:333:13

---- README.md - std__io (line 154) stdout ----
error: extern location for scroll does not exist: /home/m4b/projects/scroll/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:155:1
  |
2 | extern crate scroll;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

thread 'README.md - std__io (line 154)' panicked at 'couldn't compile the test', librustdoc/test.rs:333:13


failures:
    README.md - Advanced_Uses (line 184)
    README.md - _::Scroll___cast_some_magic::Deriving_ (line 86)
    README.md - _::Scroll___cast_some_magic::Overview (line 37)
    README.md - std__io (line 130)
    README.md - std__io (line 154)

test result: FAILED. 0 passed; 5 failed; 0 ignored; 0 measured; 0 filtered out

test readme_test ... FAILED

failures:

---- readme_test stdout ----
Running `"rustdoc" "--verbose" "--test" "-L" "/home/m4b/projects/scroll/target/debug" "-L" "/home/m4b/projects/scroll/target/debug/deps" "--extern" "scroll=/home/m4b/projects/scroll/target/debug/libscroll.rlib" "README.md"`
thread 'readme_test' panicked at 'Failed to run rustdoc tests on README.md!', tests/readme.rs:27:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.


failures:
    readme_test

I don't really understand what this means? But the CI seems to be passing; if I update the README, do I have do something special to get the tests to pass or?

Is it possible to do stateful parsing?

I'm working on parsing media streams for thirtythreeforty/neolink. Currently I'm using Nom for parsing but I'm shopping around because Nom parsers are difficult to read and write in my opinion.

The main kicker about the protocol I'm parsing is that basically every layer is stateful. This sucks for a variety of reasons but it typically means that I have to pass around a mutable context object as the struct is parsed.

Can scroll support this use case? The current Context object feels more like a "configuration" object, in that it's passed by value, and doesn't really seem designed to be modified during parsing. In fact, now that I think about it, it wouldn't necessarily be possible to read different parts in parallel, again because of the need for a single mutable state.

src/lib.rs doctest fails on s390x / IBM System Z (big endian)

https://github.com/m4b/scroll/blob/master/src/lib.rs#L55

test src/lib.rs - (line 55) ... FAILED
(...)
failures:
---- src/lib.rs - (line 55) stdout ----
Test executable failed (exit code 101).
stderr:
thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `173`,                                     # AD
 right: `190`', src/lib.rs:22:1                    # BE

All other tests and doctests pass. So it looks like the example data that's defined for the example is wrong / typo'd?

It's hard for me to tell with assertion is actually triggering the panic, because the line numbers inside the doctest are messed up (probably due to the #cfgs), but I think it's this one: assert_eq!(byte, 0xbe); (matching the 190 / BE "right" side in the assertion)

[Feature Request] Wrapper types for providing a kind of Type-level (de)serialization context

When you're trying to parse a file format filled with strings and other data with a specific endian, it may be convenient if you were able to derive the implementation for a struct

#[derive(Pread)]
struct Example<'a> {
    // reads a bigendian u16
    important: EndianWrapper<u16, byteorder::BE>,
    // reads a u8 length field and then reads out a string from the following bytes
    some_string: LengthData<&'a str, u8>
}

instead of having to manually implement TryFromCtx/TryIntoCtx like so:

struct Example<'a> {
    important: u16,
    some_string: &'a str
}

impl <'a> TryFromCtx<'a> for Example {
    type Error = scroll::Error;
    type Size = usize;
    fn try_from_ctx(src: &'a [u8], _ctx: ()) -> Result<(Self, Self::Size), Self::Error> {
        let important: u16 = src.pread_with(0, scroll::Endian::Big)?;
        let length: usize = src.pread::<u8>(2)?.into();
        let some_string: &'a str = src.pread_with(3, scroll::ctx::StrCtx::Length(length))?;
        Ok((Example { important, some_string }, 3 + some_string.len()))
    }
}

(Apologies if this doesn't actually work as-is, but you should get the idea.)

I bring this up because earlier today I created a couple of types intended for this purpose, including the LengthData type I mentioned above, if you're interested: https://gist.github.com/kitlith/feae830bfd48558e487a87becbd39ed1

EndianWrapper could probably be done similarly to the UTF16 wrapper included in the gist, just operating on a single piece of data instead of a slice.

IntoCtx/TryIntoCtx on borrowed values

The trait signatures are:

/// Writes `Self` into `This` using the context `Ctx`
pub trait IntoCtx<Ctx: Copy = (), This: ?Sized = [u8]>: Sized {
    fn into_ctx(self, &mut This, ctx: Ctx);
}

/// Tries to write `Self` into `This` using the context `Ctx`
pub trait TryIntoCtx<Ctx: Copy = (), This: ?Sized = [u8]>: Sized {
    type Error;
    fn try_into_ctx(self, &mut This, ctx: Ctx) -> Result<usize, Self::Error>;
}

These traits mirror Into which consumes self. However, these might be the wrong function signatures for how scroll is used in practice; both traits are primarily used on borrowed values.

scroll implements IntoCtx and TryIntoCtx on borrowed primitive types, suggesting that [Try]IntoCtx needn't consume its value after all.

scroll_derive generates code implementing IntoCtx and TryIntoCtx on Self by borrowing and delegating to the implementation on &'a Self.

Should these traits take &self instead of self?

Add core friendly, no_std, no result, Read/Write trait and/or api

Example

Current working draft is on a branch, core_api, and basically lets you do this:

#[repr(packed)]
struct Bar {
    foo: i32,
    bar: u32,
}

impl scroll::ctx::FromCtx for Bar {
    fn from_ctx(bytes: &[u8], ctx: scroll::Endian) -> Self {
        use scroll::Cread;
        Bar { foo: bytes.cread_with(0, ctx), bar: bytes.cread_with(4, ctx) }
    }
}

#[test]
fn cread_api() {
    use scroll::Cread;
    let bytes = [0x01,0x00,0x00,0x00,0x00,0x00,0x00,0x00, 0xef,0xbe,0x00,0x00,];
    let foo = bytes.cread::<usize>(0);
    let bar = bytes.cread::<u32>(8);
    assert_eq!(foo, 1);
    assert_eq!(bar, 0xbeef);
}

#[test]
fn cread_api_customtype() {
    use scroll::Cread;
    let bytes = [0xff, 0xff, 0xff, 0xff, 0xef,0xbe,0xad,0xde,];
    let bar = bytes.cread::<Bar>(0);
    assert_eq!(bar.foo, -1);
    assert_eq!(bar.bar, 0xdeadbeef);
}

All the basic numeric types are supported, and anything that implements FromCtx (which is much easier to understand and implement than it's evil cousin, TryFromCtx), will automatically be deseriablizable without result oriented api via:

let your_type = bytes.cread::<YourType>(offset)

as show above in the example.

Yay!

Some considerations

  1. Need to decide on name
  2. Need to decide whether this should be incorporated in another trait like pread, and add extra methods there (and what those methods should be called)

I'm partial to 2., since it doesn't require adding another trait as an import; but then again, perhaps who cares?

Of the maybe 1 or 2 people paying attention, any opinions welcome. 😎

usize/isize impls must die

It's a breaking change but I am very strongly inclined to remove them in next release.

Reasons:

  1. one can accidentally pass usize to pwrite and it writes an 8 byte number (if you're on 64-bit machine), i.e., something coming from len()`, when you actually meant to write a u32 (but forgot to cast). This has happened several times to me, and tracking down why the 4 extra bytes got written is evil.
  2. there are little to no reasons why you need to serialize what is effectively a variable sized type to disk; if you want it to be a usize, cast the known length thing after you deserialize; if you want to serialize a usize, choose a u32 or u64 and serialize, so all machines can write the same on-disk/wire datatype.

Unless someone provides compelling reasons to not remove, I will do so; please let me know if you really need this feature!

scroll 1.0

It is time for scroll to become 1.0

I believe the work started by @willglynn and seconded by others, namely, having pwrite take a reference is a good idea, is highly invasive breakage, and probably top priority item.

I may also swap the default endianness from () back to ctx::Endian, unless someone protests highly?

I would also like to survey unsafe, remove commented code, and some other details.

So:

  1. Pwrite references
  2. Default endianness ?
  3. Remove usize/isize impls
  4. Add a &str pwrite?

semi-random people who might be interested (feel free to ignore if busy, etc. :):

@luser @philipc @lzutao @willglynn

[Question] Is it intentional that you can impl Try{From,Into}Ctx multiple times for a type using different ctx types?

impl<'a> TryFromCtx<'a, ()> for SomeType { /* does one thing */ }

impl<'a> TryFromCtx<'a, scroll::Endian> for SomeType { /* does something completely different */ }

is a situation that is totally possible to come up, and I bet that in this sort of case the methods pread and gread would not function as it wouldn't be able to figure out which trait implementation to use.

pread_with and gread_with should still function because you can disambiguate the context by providing an argument.

However, really, this begs the question: did you intend for this to be possible in the first place? Do you still want it to be possible?

Generated Pread code is large/slow

The following rust code:

#[derive(Pread, Pwrite, IOread, IOwrite)]
struct Data {
    id: u32,
    x: u32,
    y: u32,
    z1: u32,
}

#[inline(never)]
fn make(input: &[u8]) -> Result<Data, scroll::Error> {
    let data = input.pread_with::<Data>(0, LE);
    data
}

fn main (){
    let bytes = Vec::new();
    make(&bytes).unwrap();
}

compiles to:

pread_example::make:
 push    rbp
 mov     rbp, rsp
 mov     r8d, 1
 test    rdx, rdx
 je      LBB19_4
 cmp     rdx, 4
 jae     LBB19_5
 mov     esi, 4
 xor     eax, eax
LBB19_3:
 xor     ecx, ecx
 jmp     LBB19_9
LBB19_4:
 xor     esi, esi
 mov     eax, 1
 xor     ecx, ecx
 jmp     LBB19_8
LBB19_5:
 cmp     rdx, 4
 jne     LBB19_11
 xor     ecx, ecx
 mov     esi, 4
LBB19_7:
 mov     eax, 1
LBB19_8:
LBB19_9:
LBB19_10:
 or      rax, rcx
 mov     dword, ptr, [rdi], r8d
 mov     dword, ptr, [rdi, +, 4], r9d
 mov     qword, ptr, [rdi, +, 8], rax
 mov     qword, ptr, [rdi, +, 16], rsi
 mov     qword, ptr, [rdi, +, 24], rdx
 pop     rbp
 ret
LBB19_11:
 lea     r9, [rdx, -, 4]
 cmp     r9, 4
 jae     LBB19_13
LBB19_12:
 mov     esi, 4
 xor     eax, eax
 xor     ecx, ecx
 mov     rdx, r9
 jmp     LBB19_9
LBB19_13:
 cmp     rdx, 9
 jb      LBB19_18
 lea     r9, [rdx, -, 8]
 cmp     r9, 4
 jb      LBB19_12
 cmp     rdx, 13
 jb      LBB19_19
 add     rdx, -12
 xor     eax, eax
 cmp     rdx, 4
 jae     LBB19_20
 mov     esi, 4
 jmp     LBB19_3
LBB19_18:
 xor     ecx, ecx
 mov     esi, 8
 jmp     LBB19_7
LBB19_19:
 xor     ecx, ecx
 mov     esi, 12
 jmp     LBB19_7
LBB19_20:
 mov     r9d, dword, ptr, [rsi]
 mov     eax, dword, ptr, [rsi, +, 4]
 mov     ecx, dword, ptr, [rsi, +, 8]
 shl     rcx, 32
 mov     esi, dword, ptr, [rsi, +, 12]
 xor     r8d, r8d
 jmp     LBB19_10

It would be nice if make() compiled to a single length check and a single memcpy for all of the fields. We did some work on improving bincode's deserialization code for WebRender and I have a rough idea for how to fix this.

cc: @mystor, @gankro, @mstange

Add support for u128/i128

u128/i128 are missing from the list of primitives that get implementations of all the nice traits in scroll.

test::pread fails on s390x

---- tests::pread stdout ----
	thread 'tests::pread' panicked at 'assertion failed: `(left == right)`
  left: `61310`,
 right: `32495`', src/lib.rs:225:8

tests fail in README.md on published crates

cargo test for scroll 0.10.1
     Running target/debug/deps/readme-1392794e1e0e15a9

running 1 test

running 4 tests
test README.md - _::Scroll___cast_some_magic::Overview (line 37) ... FAILED
test README.md - _API (line 150) ... FAILED
test README.md - _API (line 128) ... FAILED
test README.md - Advanced_Uses (line 178) ... FAILED

failures:

---- README.md - _::Scroll___cast_some_magic::Overview (line 37) stdout ----
error[E0433]: failed to resolve: maybe a missing crate `scroll`?
  --> README.md:68:9
   |
32 |     use scroll::ctx::*;
   |         ^^^^^^ maybe a missing crate `scroll`?

error[E0432]: unresolved import `scroll`
 --> README.md:38:5
  |
2 | use scroll::{ctx, Pread, LE};
  |     ^^^^^^ maybe a missing crate `scroll`?

error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729597.27629/scroll-0.10.1/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:40:26
  |
4 | fn parse() -> Result<(), scroll::Error> {
  |                          ^^^^^^ can't find crate

error: aborting due to 4 previous errors

Some errors have detailed explanations: E0432, E0433, E0463.
For more information about an error, try `rustc --explain E0432`.
Couldn't compile the test.
---- README.md - _API (line 150) stdout ----
error[E0432]: unresolved import `scroll`
 --> README.md:151:5
  |
2 | use scroll::{IOwrite, LE, BE};
  |     ^^^^^^ maybe a missing crate `scroll`?

error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729597.27629/scroll-0.10.1/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:154:29
  |
5 | fn write_io() -> Result<(), scroll::Error> {
  |                             ^^^^^^ can't find crate

error: aborting due to 3 previous errors

Some errors have detailed explanations: E0432, E0463.
For more information about an error, try `rustc --explain E0432`.
Couldn't compile the test.
---- README.md - _API (line 128) stdout ----
error[E0432]: unresolved import `scroll`
 --> README.md:130:5
  |
3 | use scroll::IOread;
  |     ^^^^^^ maybe a missing crate `scroll`?

error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729597.27629/scroll-0.10.1/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:132:29
  |
5 | fn parse_io() -> Result<(), scroll::Error> {
  |                             ^^^^^^ can't find crate

error: aborting due to 3 previous errors

Some errors have detailed explanations: E0432, E0463.
For more information about an error, try `rustc --explain E0432`.
Couldn't compile the test.
---- README.md - Advanced_Uses (line 178) stdout ----
error[E0432]: unresolved import `scroll`
 --> README.md:179:5
  |
2 | use scroll::{ctx, Pread, BE, Endian};
  |     ^^^^^^ maybe a missing crate `scroll`?

error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729597.27629/scroll-0.10.1/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
  --> README.md:188:16
   |
11 |   type Error = scroll::Error;
   |                ^^^^^^ can't find crate

error: aborting due to 3 previous errors

Some errors have detailed explanations: E0432, E0463.
For more information about an error, try `rustc --explain E0432`.
Couldn't compile the test.

failures:
    README.md - Advanced_Uses (line 178)
    README.md - _::Scroll___cast_some_magic::Overview (line 37)
    README.md - _API (line 128)
    README.md - _API (line 150)

test result: FAILED. 0 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out

test readme_test ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
cargo test for scroll 0.9.2
     Running target/debug/deps/readme-0aa9dd19900c77c6

running 1 test

running 5 tests
test README.md - _::Scroll___cast_some_magic::Deriving_ (line 86) ... FAILED
test README.md - _API (line 130) ... FAILED
test README.md - Advanced_Uses (line 184) ... FAILED
test README.md - _::Scroll___cast_some_magic::Overview (line 37) ... FAILED
test README.md - _API (line 154) ... FAILED

failures:

---- README.md - _::Scroll___cast_some_magic::Deriving_ (line 86) stdout ----
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:88:1
  |
3 | extern crate scroll;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0463`.
Couldn't compile the test.
---- README.md - _API (line 130) stdout ----
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:131:1
  |
2 | extern crate scroll;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0463`.
Couldn't compile the test.
---- README.md - Advanced_Uses (line 184) stdout ----
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:185:1
  |
2 | extern crate scroll;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0463`.
Couldn't compile the test.
---- README.md - _::Scroll___cast_some_magic::Overview (line 37) stdout ----
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:38:1
  |
2 | extern crate scroll;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0463`.
Couldn't compile the test.
---- README.md - _API (line 154) stdout ----
error: extern location for scroll does not exist: /home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib

error[E0463]: can't find crate for `scroll`
 --> README.md:155:1
  |
2 | extern crate scroll;
  | ^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0463`.
Couldn't compile the test.

failures:
    README.md - Advanced_Uses (line 184)
    README.md - _::Scroll___cast_some_magic::Deriving_ (line 86)
    README.md - _::Scroll___cast_some_magic::Overview (line 37)
    README.md - _API (line 130)
    README.md - _API (line 154)

test result: FAILED. 0 passed; 5 failed; 0 ignored; 0 measured; 0 filtered out

test readme_test ... FAILED

failures:

---- readme_test stdout ----
Running `"rustdoc" "--verbose" "--test" "-L" "/home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug" "-L" "/home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/deps" "--extern" "scroll=/home/kent/.cpanm/work/1573729474.26594/scroll-0.9.2/target/debug/libscroll.rlib" "README.md"`
thread 'readme_test' panicked at 'Failed to run rustdoc tests on README.md!', tests/readme.rs:27:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.


failures:
    readme_test

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--test readme'

Add a Pfile type

Ostensibly, it would be similar to scroll::Buffer, but which wraps std::io::File, and also wraps the pread, pwrite libc calls on linux/osx, (and fakes it on windows).

This way, similar to a Cursor/File, we can pass a Buffer/Pfile equally amongst generic functions if we like.

API enhencement

There are some problems of the API design:

  • Why ctx defaults to Endian? I can't see how generic the type is, and I think it's meaningless to select any of its variants to be default when ctx is not necessary. The unit () may be a better choice.
  • Why this crate delivers Sleb128 and Leb128? I think we could split them into a new crate, with dependency on scroll.
  • Trait FromCtx and TryFromCtx has a lot of code duplication in implementation.
  • Trait TryRefFrom requires ctx implements Copy when RefFrom don't.
  • Lack of MeasureWith semantic.
  • The name. To be honest, I was confused at the beginning when reading the document and tring to figure out the relationship between a lot of traits in this crate. Finally, I came to realized that pread was actually a wrapper of trait TryFromCtx after reading source code. Also, the name of trait IntoCtx and FromCtx is vague and misleading.

So I suggest the following modifications:

  1. Remove DefaultCtx and replace by ()
  2. Rename TryFromCtx, TryIntoCtx to TryRead, TryWrite, which seem more fit their purposes. (the main idea is to reduce priority of the concept Context because this library is mainly to scroll across data then Read and Write things, but not working around some context)
  3. Remove trait FromCtx IntoCtx, and then use TryRead and TryWrite in cread by unwrap() or expect(). This will make it easier to implement these traits.
  4. Unify traits RefFrom and TryRefFrom TryRefInto into TryRead and TryWrite. This is a very early though and maybe I will open an pull request latter on this. In brief, impl TryRefFrom for [usize] is equal to impl TryFrom for &[usize].
  5. Split SizeWith into two new traits WriteSize and ReadSize. WriteSize takes a reference to self(&self) while ReadSize takes a reference to slice.
  6. Move Endian into scroll::ctx, I think that's the right place for it.
  7. Move traits TryRead, TryWrite, WriteSize, ReadSize to root module. I've two reasons to do that. First, they are not actually contexts, when the Endian is. Second, TryRead is related to Pread, Cread family, and they should have the same priority. This is one of the reasons I get confused in the beginning.
  8. Rename TryOffsetWith => TryOffset. There is no ambiguity like that between pread and pread_with. Also, the reason for removing the word with is same a (2), is to reduce the attention on context.
  9. Remove const LE, BE when Endian::LE is more idiomatic.
  10. Remove const LEB128 because it's just const NATIVE
  11. Move const NATIVE and NETWORK into scroll::ctx
  12. Remove type definition Leb128, which is ambiguous and just an alias to NATIVE.

Looking forward to your opinion and I think I can help with these refactoring.

gread and pwrite, gwrite can require SizeWith

I wish I had thought of this earlier...

fn gread<'a, N: SizeWith<Ctx, Units = I> + TryFromCtx<'a, Ctx, <Self as Index<RangeFrom<I>>>::Output, Error = E, Size = I>>

This would be an overall gain across the board, for ergonomics and probably performance, since the client implement TryFromCtx doesn't have to return size, and the gread method would increment the offset for you...

Similarly for pwrite, it would write the data, then update with SizeWith; since SizeWith is implemented for all the basic types, and there is a deriving SizeWith, and it would enforce contraints on pread, this would have been the best solution all around I think...

EASY: better error messages

I.e., this error message sucks:

type is too big (68719476740) for 7828

I just realized tho, that we can "recover" the error; if we simplify our internal types to return TooBig, we can then have pread "catch" TooBig errors, and then recompute the offset, requested size, the offset requested at, and size of the bytes for us, all in a nice package

Reasoning behind associated `Size` / `Units` types on various traits?

I've stumbled over this before while writing functions that were generic over TryIntoCtx and SizeWith--the constraints need to explicitly call out TryIntoCtx<Size=usize> or SizeWith<Units=usize> in order to be able to actually use the sizes for anything meaningful.

Both MeasureWith and SizeWith have an associated Units type:
https://docs.rs/scroll/0.9.0/scroll/ctx/trait.MeasureWith.html#associatedtype.Units
https://docs.rs/scroll/0.9.0/scroll/ctx/trait.SizeWith.html#associatedtype.Units

And both TryFromCtx and TryIntoCtx have an associated Size type:
https://docs.rs/scroll/0.9.0/scroll/ctx/trait.TryFromCtx.html#associatedtype.Size
https://docs.rs/scroll/0.9.0/scroll/ctx/trait.TryIntoCtx.html#associatedtype.Size

I'm sure you had an intended use case for this being generic when you wrote these, but is it actually something that has proven useful in practice? grepping through the scroll, scroll_derive and goblin repositories shows that there aren't any implementations of these traits using anything but usize, and most usage in goblin is via the derives, which all provide implementations where the types are usize.

I know it would be a breaking change, but removing these associated types and just using usize everywhere seems like it would simplify things a bit.

The only counter-argument I can come up with on my own is that perhaps you might want to have types that have sizes that don't fit in 32 bits but you want to use them with these traits in 32-bit code, but this feels a bit contrived and also would suggest that just using u64 instead of usize everywhere would be the answer.

implement the Lesser Read and Write Traits

e.g., Lread, and what in particular this api will look like. After converting all of goblin to use scroll, I no longer have any need whatsoever for a mutable Reader, and in fact it's presence makes any easy parallelism unnecessarily difficult, so not sure it's worthwhile?

determine generic param order

I think that custom Ctx will be more common than custom Error, and hence first param should be Ctx and not Error to make the API more ergonomic.

Design: boxed preads

In anticipation of #47, the final remaining piece for working with arbitrary data safely is to figure out how to pread large objects without blowing the stack.
Most system languages accomplish this by boxing the object and manipulating it through the heap pointer.
I'm wondering if we can accomplish this in a semi-automated way, without compromising scroll's interesting type based reading/writing.

Doing this properly would allow us to round trip a large stack object within scroll.

Specifically, could we do something like:

#[derive(Pread, Pwrite, Copy, Clone)]
struct Big {
    arr: [u8; 100000000],
}
let mut bytes = vec![42; 100000001];
let big = box bytes.pread_with::<Box<Big>>(0, LE)?; // this is boxed
// by the scroll Box impl, using unsafe unitialized memory
// which is only returned if the pread succeeds, so it remains safe?
let _ = bytes.pwrite_with(big, 0, LE)?; // Note: there is no `&*`,
// ideally scroll can deal with this case, though I don't know if its possible?

In worst case, we should be able to expose a new pread_with_box method, that does the above, but pushes the box type information into the method name. This is fine, but i think its a more beautiful api design to allow the user to choose which type to read out via turbofish.

cc @philipc @lzutao @willglynn @luser
What you all think?

Enums derive

Hello everyone!

I am working on a project in which we have enums that whey (de)serialized add the Enum::Case(CaseStruct) as a u8 or u16 before (de)serializing the CaseStruct.
Currently we've implemented custom TryFromCtx & TryIntoCtx, but it would be nice to be able to derive it and add something like serde enum tag attribute and specify the type per case(maybe?!).

#[derive(Pread, Pwrite)]
#[scroll(tag = "u16")]
enum MyEnum {
   #[scroll(tag_value = "0")]
   Case(CaseStruct),
   #[scroll(tag_value = "1")]
   CaseTwo(CaseTwoStruct),
}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.