[Feature Request] Wrapper types for providing a kind of Type-level (de)serialization context about scroll HOT 12 CLOSED

So reading back through this issue, not sure what the resolution was here, or what would be actionable; it sounds like maybe some scroll extensions with commonly used types/targets might be useful? Not sure; e.g., i've written code in the past where I had "wrapper types" which were totally generic but took care of e.g., decoding something like a length encoded list of types, or a length encoded list of key value pairs, (like an encoded hashmap so to speak). But all this could technically be done as a third party crate; going to close, feel free to reopen if have other ideas

from scroll.

@kitlith Does https://docs.rs/scroll_derive (better site: https://github.com/m4b/scroll_derive) not do what you want?

You should be able to do:

#[macro_use]
extern crate scroll;

#[derive(Pread)]
struct Foo {
  a: u64,
  b: u16,
}

And it should usually do what you want.

NOTE: to use #[macro_use] like above, must turn on the "derive" feature via your toml, something like:

[dependencies]
scroll = { features = ["derive"], version = "0.9" }

(i forget the exact syntax, but that's the gist)

from scroll.

So, the idea is that this can be used with scroll_derive. How do you specify the context to that (say) Big Endian u64 and that Little Endian u16? How do you specify the context (which is a value passed to a function) at the "I'm just defining a struct and deriving" level?

Edit -- I did get the derive macro name wrong, but that's because I keep not having a chance to use it because I keep needing to write implementations for TryFromCtx because it's not a simple file type.

from scroll.

Oh I think I understand. You want different endianness contexts for each field?

If yes, currently this must be done manually. It’s possible that the derive macro implementation could also receive annotation hints, similar to how serde or structopt annotates individual fields.

I’m not sure though; not a huge macro guru; would require some investigation. You are of course welcome to suggest ideas here or sketch out a possible PR :)

As long as it’s backwards compatible I’d be ok with adding something like:

#[ctx = Little]

On fields in a derived struct. I think this would only work with the endian enum? Not sure; it’s also probably some good amount of work for a potentially rare usecase ?

from scroll.

I want to be able to provide context at compile-time where possible -- this doesn't require modifying the derive macros, it just requires using different types. If I had actually implemented an EndianWrapper type, I believe it would "just work" with the derive macro.

It also wouldn't be limited to just the Endian ctx, for example the "LengthData" type uses a template parameter to provide the data type that represents the length field -- so I can work with files that have a u16 length and files that have a u32 length... or both in the same file >>. (yes, I have a file that uses UTF16 LE and BE and different sized length fields. ..)

I guess I'm just providing an idea that I think would be really useful if it were expanded upon some more, because the only opportunity I've had to use the derive macro was on a file header that was used inside a TryFromCtx implementation to create a user-facing struct.

from scroll.

I want to be able to provide context at compile-time where possible -- this doesn't require modifying the derive macros, it just requires using different types.

Right, I think what you want are field annotations similar to structopt/serde; so you'd write:

#[derive(Pread)]
struct Foo {
  #[ctx = Endian::Little]
  hdr: u16,
  #[ctx = SomeOtherContext]
  n: FooWithOtherContext,
}

Which feeds those contexts into the derived implementation. I believe this is doable.

However, hooking up reading out a byte, which then serves as the length to the next string field would be hard; I'm not sure how to do that via a derive macro annotation. By its very nature, it just seems programmatic to me; hence one needs to implement the trait.

But I wouldn't mind seeing a cool argument as to why one wouldn't have to :D

from scroll.

That's why... I implemented the trait on a wrapper type that information tagged onto it using the templace system? Seriously, having a struct which contains a LengthData<UTF16<byteorder::BE>, u16> should already work with the code I've written in my gist that I linked in the OP. I'll try and write up an actual working test in a day or so.

from scroll.

Here's a full example: https://github.com/kitlith/scroll-stuff

I'm replicating the (relatively small) final usage in that crate without all of the type and trait impls here:

#[derive(Pread)]
struct Example {
    big: EndianWrapper<u16, BE>,
    little: EndianWrapper<u16, LE>,
    var: LengthData<UTF16<LE>, EndianWrapper<u16, BE>>
}

fn main() {
    let src = [0u8, 42, 42, 0, 0, 10, 0x48, 0, 0x65, 0, 0x6c, 0, 0x6c, 0, 0x6f, 0];
    let example: Example = src.pread(0).unwrap();
    let big: u16 = example.big.into();
    let little: u16 = example.little.into();
    let var: String = example.var.into();

    println!("Example {{ big: {}, little: {}, var: {} }}", big, little, var);

    assert_eq!(big, 42u16);
    assert_eq!(little, 42u16);
    assert_eq!(var, "Hello".to_owned());
}

Something that would make end usage nicer for the user would be a struct that removes all the wrapper types for easy usage, but my point stands that this is completely possible.

On my system, it prints Example { big: 42, little: 42, var: Hello } without triggering any asserts.

from scroll.

@Kilith Um, wow; I’ve never seen scroll used like that before :P

So, if I understand correctly, you’d like to remove the two wrapper structs (at the very least) and their impls ?

This would remove a lot of the boilerplate.

If yes, again, this looks like you want annotations on your derive, explicitly specifying the ctx for some (or all) fields, which should be possible.

However, I don’t see how it’s possible to remove the last wrapper, since that uses programmatic information to extract the String, iiuc? In which case we can’t do anything at derive/compile time.

Please correct me if I’m wrong and feel free to add anything else you have in mind. And thanks you so much for the example !

from scroll.

Nah, I don't want to remove the wrapper structs (that seems tricky anyway wait, we're using that phrase in different ways). I want to include some types like this in scroll, because stuff like this seems useful for decreasing the number of manual implementations that need to be done of Try{From,To}Ctx.

Basically, after seeing this, how useful do you think it is? What type of this stuff would you like (or not like) to see in scroll?

from scroll.

Unfortunately it's still a little unclear to me what you want precisely.

At this stage, I think it would be useful for you to describe precisely what you do and do not want to see, as I seem to keep misunderstanding you. Your example was useful, and apparently compiles?

When you said you wanted to reduce impl boilerplate I assumed this meant adding new derive functionality as I described above, but apparently this is not what you meant, so I'm going to need some help understanding :)

Perhaps you could start with:

1. what you mean by "including types like this in scroll"
2. what this would look like
3. why it is useful/what burden it would reduce, perhaps with a motivating example?

from scroll.

1. "types like this" -> EndianWrapper, maybe LengthData. I don't have a full list of stuff that might be useful, because I've just started looking at this stuff.
2. To the end user, it'd look roughly like what I copied out of my example. scroll would hold the implementations of EndianWrapper and such.
3. The user no longer has to implement TryFromCtx/TryIntoCtx in situations that these wrapper types would cover. I can provide a manual implementation for the example above, to compare and contrast, if that would help?

Perhaps this is better suited for a Pull Request, except that I wanted to discuss if the idea of wrapper types that work like EndianWrapper being provided in scroll is appealing before starting a PR.

from scroll.