Containerize test generation and run about conformance HOT 4 OPEN

If this seems like a good approach, I would be happy to work on implementing the solution.
All thoughts and feedback welcome.
cc @echeran

from conformance.

That sounds great to me! I like the listing of the separate incremental steps to make this change (easier to review & adapt to). Overall, this ought to make the project easier to use and maintain.
FYI @sffc @sven-oly

from conformance.

Thanks for the feedback @sffc . Yes, I'm proposing the containerizing of the Python scripts etc for test generation and running, not just the executors. I was planning for the executors to run in the same container as the scripts (at least initially), similar to how GitHub Actions runs them currently. Containerizing individual executors sounds interesting though and could be decided separately. My proposal is just taking the existing test generation+run setup (including its single environment execution) and making it more transparent and easily reproducible - on a developer's machine, for example.

I do still think it is good to have our Python scripts (or the thin .sh files around them) be the entrypoints.

Yes, totally agree. I think that a simple entrypoint to test generation and running would make sense, especially for those without Docker experience. I also think a way to opt out of containerization would be a good idea for developers that have all dependencies locally and don't use Docker. e.g.:

$ python conformance.py test --no-container

This is the basic scope of what I was thinking:

(Your suggestion of containerizing the individual executors would mean that the 'Executor deps' could be moved out of this container.)

Does this address any of your concerns about putting the run into the container?

from conformance.

Things that are very architecture-specific, such as the C++ executor, would be good to containerize. Maybe some of the other executors like Rust or Dart, especially if they depend on very specific compiler setups.

However, I don't think it's a good idea to shove the entire runtime into a container (I'm not sure exactly the scope of what you were thinking). I do still think it is good to have our Python scripts (or the thin .sh files around them) be the entrypoints. But, it makes sense if Python invokes a container as an executor and treats it like a native executor (interacting with stdin/stdout).

from conformance.