Rust roadmap about meltano HOT 2 OPEN

Hey @reidlai!

Nothing as concrete as a roadmap but some folks in the community and myself have done interesting PoCs for Rust-based connectors.

See for example:

• https://github.com/edgarrmondragon/singer-rust
• https://github.com/z3z1ma/singer-rust

In the first example, there's even a toy target published to PyPI thanks to Maturin¹ so it's something you could already add to Meltano! (Taking a personal note to add an example of that to the README).

That said, the main takeaways from those PoCs are:

1. IO is slightly more performant, perhaps in large part because we're deserializing into an struct. The singer-sdk is surely leaving some performance on the table in that regard. See meltano/sdk#1784.
2. Rust and Python are miles apart in terms of beginner friendliness, so it's unclear how much the community could participate in creating novel connectors.
3. The point above makes me think that one interesting approach would be to write the most demanded targets and mappers in Rust with CLIs published to PyPI.

I'm curious what you think of that, and whether you have expectations of what part Rust would play in Meltano plugins.

from meltano.

Actually python is not efficient compared with rust. I know that I can use rust to create tap and target using rust...but I can foresee if we can use rust to build custom mapping and transformation, meltano can run very fast without GC

from meltano.