Addressing ergonomics around Rust and embedded/no_std development about wg HOT 13 CLOSED

I have a question about #![no_std] attribute. Why do we need to have it in the source code? Why not just a compiler flag? I think there are many crates like ref_slice that doesn't need anything from std, and would have happily compiled for no_std target. But instead you need to ask each crate's author to modify their source code. This feels counterproductive. Also having std/core renaming makes matters even worse, not only do you need to add no_std attribute, but you also have to rename the core back to std. So why it was renamed in the first place?

Small example:
Three weeks ago I needed to parse a MessagePack format on a Cortex-M4. There is a great library on crates.io called rmp but, as you might have guessed, it doesn't have no_std feature. This is what I had to do to make it work on my MCU:

1. Add modified num-traits crate which doesn't have floating points (rmp doesn't need them anyway)
2. Provide my own traits like std::io::Read and std::error::Error
3. Add std::io::Cursor struct
4. Add part of byteorder crate that depends on Cursor

As you can see, I didn't have to do any real changes to rmp crate itself, the only thing that it needs is a Cursor and a bunch of io traits. On the other hand, rewriting rmp so it doesn't use Cursor and byteorder, would require a lot of effort and probably make the code much more complex and less maintainable.

If I had some easy way to provide a custom std based on core + Cursor, I could have just added rmp as a regular dependency in my Cargo.toml. And If you include alloc and collections in such hypothetical modular std, then I'm sure it will be possible to compile hundreds more crates from crates.io without any modifications.

from wg.

@Susurrus Thanks for the write up!

Some comments

2. This is rust-lang/rfcs material. Sounds good to me, but it seems non
   backwards compatible: For example a no_std crate with a module named core
   in its root when compiled with the "std" feature enabled.

3. Same comment as above.

4. I wish cargo add accepted a --no-default-features flag. A magic solution
   here would to disable the "std" feature of all crates when the top crate
   contains #![no_std]. Again rust-lang/rfcs material and possibly non
   backward compatible.

5. Some float operations, like abs, are available on core via the unstable
   Float trait. I think the only reason that the trait is unstable is because
   those methods should be inherent to float rather require a trait import.
   However, the current implementation doesn't support splitting inherent
   implementations on primitive types across crates (it would be easy to violate
   coherence if that was allowed). For the other float operations, like sqrt,
   is because they come from libc, at least when you use std, and core should
   not depend on libc. This can be solved by having a Rust implementation of
   libm in core (core can't have dependencies). Though that's a lot of work
   and nontrivial (you have to make sure your implementations have acceptable
   rounding errors). For OsStr, PathBuf, etc. I don't think they should be in
   core as they can live in other crates on top of core. IIRC, the reason that's
   not the case is because of coherence. TL;DR I agree no_std floats should have
   operations like sqrt and abs as inherent implementations; however, that's not
   easy to implement.

6. 👍. Sounds hard though.

7. I don't see how this can happen. If you try to publish a crate A that depends
   on crate B with default-features = true and that has a "std" feature then
   publication would fail because compilation would fail. Now if crate A depends
   on crate B with default-features = false and you are building a crate C that
   depends on both A and B but you forgot to set default-features = false for
   your dependency B then compilation of A would fail because Cargo features
   are additive and A's dependency B would also get compiled with the feature
   "std". However, this is not the fault of the author of crate A.

8. FWIW, there's a category for this.

10. If I'm reading this correctly cargo test should not only add implicitly
    inject a dependency on test but also dependency on std. I'm 👎 on this
    as this would impede testing via no_std testing frameworks like
    utest, which are useful if you want to run tests on no_std targets
    like microcontrollers

from wg.

These are on point. I've run up against many of the same places of friction.

from wg.

@pftbest instead of writing custom IO traits, please consider using genio crate. It has no_std support and addresses some other problems with std::io. Let me know if it's missing something you need.

from wg.

@Kixunil
My goal was to make modifications as small as possible, so it would be easy for me to update the crate in the future if I need to.

If I understand correctly, genio uses a different approach to error handling than std::io, so it would require some heavy modifications to both rmp and byteorder. Which I don't have the time to do at the moment, sorry.

I did consider using core_io crate, it even has its own byteorder fork, but I had a problem running tests, because they use Vec and Cursor from full std. I needed to implement my custom io::Write for std Vec. I do see that this will not be a problem in genio, which is nice.

from wg.

Thank you for feedback!

I'm considering to implement byteorder-like trait for genio. Regarding rmp, maybe we could convince the author to implement it on top of genio as genio wants to be compatible with std::io. (Anything that impls std::io should be able to impl genio.)

from wg.

@Kixunil

Maybe it would be better to add genio support in byteorder crate itself. If such popular crate would gain support for genio then it would be easier to convince rmp and other libraries to switch.

from wg.

@pftbest Since genio still needs some work, I doubt 1.0 crate will want to depend on 0.1 crate.

I looked at genio and realized it already has native implementation of byteorder-like methods, if you turn on byteorder feature.

from wg.

@pftbest are you interested in posting your fork of rmp on crates.io? I need pretty much the exact same thing. I've been unable to find an existing rust library on crates.io that supports a "robust" serialization format (meaning contains the description of the data schema as well as the bytes) e.g. messagepack, CBOR, avro, and supports serde, and supports no_std.

There is corepack but it seems to be broken in our tests i.e. when we swap rmp for corepack in part of our system we start getting serialization failures, and it has some other open issues

from wg.

@cbeck88 Sorry, I've missed your message. I don't think I will post post my fork on crates.io, mainly because it is low quality hack. For example I've defined my own io::Read and io::Write traits, and implemented them only for arrayvec and arraystring, so if you want to use them with some other containers you'll have to modify the source code which is not possible with crates.io. Also there is a hard limit on string length of 512 bytes, because I don't use allocations, and have to create strings on the stack. You might want to change this number for your project.
Currently the code is based on old version of rmp, so it might contain bugs. I need to update it to latest version of rmp and maybe also replace io traits and parts of byteorder with genio but unfortunately I don't have time to do that at the moment.

If you still want to see the code you can download it here:
msgp_1.tar.gz

from wg.

@pftbest Hey, I did some work to update your patch. I agree it's not the prettiest but it's really the best option right now at least for us. You can see our updated version of your patch here: 3Hren/msgpack-rust#187

I didn't fully understand what is needed for byteorder -- byteorder got an update not so long ago so it now has default-feature = false allowing it to be used in no_std, I believe it has std feature much like serde, and otherwise is core only. So it's possible that only the genio work is left?

We're interested in investing time and energy in this to make this as maintainable as possible. Thanks for making your patch available.

from wg.

@cbeck88, byteorder is needed because of how rmp works internally. It is using functions like read_u32 to get data from io::Read stream. byteorder provides this functions for std::io::Read when std feature is enabled. But in this case we have our own io::Read trait so I had to copy paste this code from byteorder. I saw that @Kixunil was working on a similar feature for genio but I don't know if it's complete.

P.S. I completely forgot that I had this code on github, I thought I only had a local copy. Thanks for finding it.

from wg.

Sorry for replying this late, yes, it does work, you just need to enable byteorder feature.

from wg.