As I mentioned in this post on the discourse, one of the tougher things about doing these integration tests right will be ensuring that all the build tools pull in the dev versions of the other relevant build tools. Considering that virtualenv, pip and tox will all be pulling packages from PyPI and will explicitly be trying to build isolated build environments, I imagine this will be kinda tough.

I wanted to start a thread on this repo to discuss it, so we can have a dedicated issue rather than just having all the design decisions hashed out on one mega-thread on discourse.

I think the easiest way to make sure this works correctly is to spin up our own PyPI endpoint locally. We would first use the stable build tools to build wheels and sdists for each of the relevant dev tools, then download all the relevant files we're going to need for the other packages from PyPI before the tests start.

We could also try and use a caching proxy for PyPI where we pre-cache the dev versions, but I'm worried that there could be situations where the resolver ends up falling back to non-dev versions and we're suddenly silently not testing what we think we are.

Should this repo be archived?

From the Discuss, here is what I understood the tests could look like:

• Prepare the artefacts:
  for all the packaging bricks: pip, setuptools, etc:
  • download the latest stable wheel release from PyPI
  • build a wheel from their development branch
• For each brick A, the test would be:
  • launch a local PyPI exposing A master wheel + the stable wheels of the other bricks
  • create a virtualenv and install the different bricks
  • try to install a curated list of packages ? Maybe https://pypistats.org/top ?

What are the bricks we want to test ?
pip, setuptools, pep517, wheel, virtualenv, tox, (should we also include their dependencies ? like pytoml for pep517, etc).

I'm not really sure how to read Azure pipeline configurations, but it seems like @gaborbernat (thanks for setting this up, by the way!) has set it up to build on PR and also daily, which I think is a good cadence. However, I'm wondering if we want to (and also if it's even possible to) try and add some sort of hook that would trigger either on demand or one that can be added in to the repos under test.

The reason I'm thinking this might be useful/necessary is that while most projects would not necessarily want to trigger integration test builds on every PR, generally speaking right before a release (particularly an urgent bugfix), you end up tweaking and/or merging a whole bunch of stuff that you want to get into the release. It would be nice to be able to run the integration test suite once you have a release candidate ready but before a release - seems like "right before a release" would be a good time to run the full integration test suite.

I don't have a great sense of how to actually accomplish this. Some options I'm thinking:

1. Integration tests run whenever a new rc release on PyPI is cut (not sure how to register that as an event)
2. This project is made "portable" and the individual repos can check it out in their own CI on whatever schedule they want.
3. This repo somehow watches the project repos for a specific tag or event, or responds to some event that can be kicked off in the project repo CI.

None of these seem great or particularly easy to accomplish. I imagine for now if you want to trigger an integration test build, you can just make a trivial PR to the repo, or each project can make a single trivial PR and whenever they want a build they just close and re-open it. Still, I thought I'd throw it out there to see if 1. this would be useful to other people and 2. if anyone has ideas on how to accomplish it.