Backend crawl log streaming API: Stream logs from WACZ files after crawl concludes about browsertrix HOT 6 CLOSED

@Chickensoupwithrice In your court now if you want to try to figure this out :)

from browsertrix.

Alright, after much experimenting I've managed to nail down exactly where we're no longer doing generators and instead load up all the logs. It's in the way we're calling stream_log_bytes_as_line_dicts (which by itself does return a generator) but instead we're extending an array by the output of the generator leading to loading the entire log file into memory and then run out of memory.

Switching extend to append does mean we're generators all the way down, but then when I try to execute on this generator, I get read timeouts on the DO space?

Still investigating.

from browsertrix.

First pass is implemented in #682.

We'll want to move to properly streaming logs, currently blocked by aio-libs/aiobotocore#991

from browsertrix.

Until the aiobotocore is resolved, we may be able to use the sync download option, since we've already implemented this to support collection downloads via https://github.com/webrecorder/browsertrix-cloud/blob/main/backend/btrixcloud/storages.py#L358

from browsertrix.

Implemented as a sync stream in #1168 . Closing for now, though we may eventually want to make this async.

from browsertrix.

Still seems to be a memory issue, looking into it.

Could just fetch from presigned URLs

from browsertrix.