🐛 Bug Report: Timeout error while processing large parent location registered by Github Catalog Discovery Plugin about backstage HOT 1 OPEN

There are a couple of things that get wired together for this ingestion e2e. The entity provider will create a location in the catalog with the wildcard intact, that will in turn be picked up by the UrlReaderProcessor and treated as a "search" target here:

That in turn is gonna take us to the GitHubUrlReader in this case, which in turn does an initial search request here:

If that response is not truncated, it will continue on to do individual fetches for each matching file over here:

This fetch I think we need to make sure doesn't happen for this case. I don't know if it is right now, but based on the error message it seems like it might. What would be better is if it instead falls through to do a full reading of the repo tree so that all files are available locally over here:

If we end up there we don't have any issues with individual file fetches, since it's all available upfront.

So to move this ahead we'd appreciate if anyone wants to validate the above hypothesis that the initial search is not truncated. If that ends up being the case, I think we should probably add some additional check that cases us to fall back to the read tree path in case of very large repos.

from backstage.