Nightly build failure 2023-01-10 about pudl HOT 8 CLOSED

I downloaded last night's outputs

> gsutil -m cp -r "gs://builds.catalyst.coop/2024-01-10-0603-d62dd5c42-main" .

Then I added a --build-only flag to mirror the flyctl deploy --build-only flag, to see if the build works.
Then I ran

> PUDL_OUTPUT=~/scratch/2023-01-10/2024-01-10-0603-d62dd5c42-main/ python publish.py --build-only
...

which succeeded. Ignore the fact that I got the year wrong in the containing directory 🤦

So I'll re-run it without the build-only flag, and also make a PR for said build-only flag.

from pudl.

Re-ran the full deploy and it all worked! If we keep running into this we should consider how to retry automatically or perhaps bake less data into the docker image.

from pudl.

I'll keep this open until the QOL fix related to this issue gets merged in.

from pudl.

In general when we get a failure like this do we actually want to do the redeploy? How do we ensure that the data that's showing up in Datasette corresponds to the data that shows up under the /nightly prefix in the distribution buckets and was it generated with the code that's on the nightly branch? If something post-etl and testing goes awry, do we just investigate and make it internally consistent by hand?

from pudl.

Your worry is that whatever outputs are in builds.catalyst.coop for this specific build ID doesn't correspond to the outputs that were verified?

I think if you look in the logs, and:

• nothing seems to have gone wrong in the upload to builds.catalyst or pudl.catalyst
• nightly points at the correct nightly build tag

Then it's safe to redeploy - of course, assuming nothing goes wrong in the re-download from GCS / the upload to fly.io.

from pudl.

Oh, or are you asking about whether we should automatically try to re-deploy? I think if the error occurs in flyctl deploy it's worth trying again - everything has passed validation up until that point, you know?

from pudl.

I'm trying to think about how our published outputs / repository could end up being inconsistent with each other, depending on which of the individual sub-tasks has failed, and how we impose consistency after the failure has happened

In this case, the only thing that failed was the Fly.io deployment, and nothing depends on it, so re-deploying Fly.io gets everything back in sync.

But if say, copying from builds.catalyst.coop to pudl.catalyst.coop fails, then the s3 outputs would also be out of sync with what got uploaded to Fly.

from pudl.

Hmm, the fly.io deployment process is from the pre-compressed-outputs days. We could actually have it download the compressed outputs from pudl.catalyst.coop instead of uploading directly from PUDL_OUTPUT - maybe that's more robust?

from pudl.