Lambda execution randomly times out about sidecar-browsershot HOT 7 OPEN

@stefanzweifel This is a weird issue that happens completely randomly, we have no clue what exactly causes it or how to reproduce it. Because when this timeout happens we try it one more time generating the same PDF and everything works... The last time when timeout happened the only clue we got was that CloudWatch log.

Initially, the timeout was 30 seconds, we tried to increase it to 300 seconds, but it still didn't help. When this timeout happens lambda gets "stuck", no amount of timeout helps.

What we did as a workaround:

• lowered browsershot timeout to 30 seconds. when everything works, usually we get the PDF back < 5 seconds
• added a loop of retries up to 5 times, whenever LambdaExecutionException happens

use Hammerstone\Sidecar\Exceptions\LambdaExecutionException;

private int $retry = 0;

public function render(): string
{
    try {
        return $this->browser
            ->setHtml($this->html)
            ->format($this->format)
            ->margins(...$this->margins)
            ->pdf();
    } catch (LambdaExecutionException $exception) {
        if ($this->retry < 4) { // 5 times in total, including the first attempt
            $this->retry++;

            return $this->render();
        }

        throw $exception;
    }
}

I suspect maybe it is related to the underlying library... The CloudWatch log says the deprecation warning is related to fs.rmdir, not fs.rmdirSync which is package is using. Maybe we can try node --trace-deprecation on the layer. There are some reported and closed issues on the puppeteer repository: https://github.com/puppeteer/puppeteer/issues?q=is%3Aissue+rmdir

I also found reported similar issues related to puppeteer-extra, but I believe this library is not being used here.

from sidecar-browsershot.

As you might have seen, my attempt (#112) at updating our internal code to no longer use fs.rmdir was not successful. Have to figure out, what the root issue here is.

In the meantime, I've updated an underlying layer to use the latest puppeteer-core version and updated this package to also use an updated Chromium version: https://github.com/stefanzweifel/sidecar-browsershot/releases/tag/v2.1.0

Can't guarantee that this will solve the issues.

from sidecar-browsershot.

@orkhanahmadov From which version did you upgrade? From v1.13.1? Or from something prior?

In this compare view, you can see that between v1.13.1 to v2.0.0 basically nothing changed, besides the new spatie/browsershot:v4.0 requirement and the change to the config file.

I didn't have yet the time to upgrade my own apps to the latest version, so can't speak from experience. Weird that our test suite in the package, that runs on AWS, is green. 🤔

from sidecar-browsershot.

@stefanzweifel ok, turns out it is not related to v2, I guess.

Recently when the timeout happened again, we noticed a CloudWatch log with the following contents:

Is this useful or related?

from sidecar-browsershot.

Thanks for the update @orkhanahmadov. Seems related to a cleanup bit of code:

Will create a PR with a fix soonish.

from sidecar-browsershot.

@orkhanahmadov Would you able to share some code snippets for this issue. I'm not able to replicate your timeout issue on my machine or in my production apps.

I don't assume you run any JavaScript in your Blade views?
I know this sounds like a not ideal solution, but have you tried increasing the timeout of your function? Does the error still occur?

Maybe this is related to the underlying Puppeteer versions. Will work on upgrading the underlying layer to the latest puppeteer-version.

from sidecar-browsershot.

@orkhanahmadov Thanks! Lowering the timeout makes total sense here. Better fail fast than wait for forever and produce unnecessary cost.

Will do some research. 🤓

from sidecar-browsershot.