stream JSON lines output about scancode-toolkit HOT 2 OPEN

This would be useful indeed but there may be some gremlins along the way as this is the general processing flow:
0. Run pre-scan plugins

1. Run all scanners on one file. Cache the results on disk in a temp JSON file, one such file for each file scanned (this is stored on disk as soon as you have more than 10,000 files scanned)
2. Run the "process_codebase" step of each scanner one after the other that iterate on the whole codebase and can amend the results from step 1.
3. Run post-scan plugins one after the other that iterate on the whole codebase and can amend the results from step 1 and step 2.
4. Run output plugins to create JSON/SBOM outputs

We could technically output some JSON lines early, or some pseudo JSON-like lines, and output the headers and summaries at the end, but there no way to make sure the results are correct and would not be amended after the initial scanner step.

Now, what is your concern? getting some details of the scans early for diagnostic purpose?
Or stream processing large scans?

from scancode-toolkit.

Now, what is your concern? getting some details of the scans early for diagnostic purpose? Or stream processing large scans?

Mostly the latter. Of course, if there are other ways to speed up scancode that would also help a lot.

Another thing is that right now it isn't too easy to extract results for a single file from the larger scancode file. So what I might actually be after is the temp JSON file from the cache (one per file).

from scancode-toolkit.