Comments (28)
This same error can occur with Gatsby builds for projects that do not include gatsby-remark-images-contentful
from netlify-plugin-gatsby.
Same issue here. Reverting to Gatsby 4.7.2 seems to have stabilized builds for now.
from netlify-plugin-gatsby.
This same error can occur with Gatsby builds for projects that do not include gatsby-remark-images-contentful
Same - I do not think this should be closed?
I am no longer able to deploy my Gatsby v4 app with Netlify at all.
A build that deployed fine on February 14th now no longer builds at all (same exact commit, just cant redeploy).
from netlify-plugin-gatsby.
from netlify-plugin-gatsby.
I just updated my node version to v16 (from v14) in netlify.toml and that seems to be making a positive impact so far.
I'm guessing this is related to this discussion: gatsbyjs/gatsby#34215
from netlify-plugin-gatsby.
It certainly sounds like the same bug in parallel query running. You could try adding those env vars in the UI and see if it helps. Is it always the EPIPE
error during image processing? Gatsby runs image processing in a separate worker thread, and what seems to be happening is that is that thread is crashing. Rather than the error from the worker beign surfaced, you're instead seeing the error from the parent failing to communicate with the worker.
from netlify-plugin-gatsby.
The other thing that can cause an image processing thread to crash is a corrupt source image, but Gatsby is a lot better at catching those than it used to be, and it usually surfaces a more helpful error.
from netlify-plugin-gatsby.
It certainly sounds like the same bug in parallel query running. You could try adding those env vars in the UI and see if it helps. Is it always the
EPIPE
error during image processing? Gatsby runs image processing in a separate worker thread, and what seems to be happening is that is that thread is crashing. Rather than the error from the worker beign surfaced, you're instead seeing the error from the parent failing to communicate with the worker.
Every time it's EPIPE, and always at the image processing step. I looked around at recent publishes in Contentful and changes in the codebase, but there haven't been many changes lately.
Just check out how many failures I'm getting. It's also random, on when main
built there were no code changes, just a retry deploy, same with the feature branch just above it. staging
was also failing at the same time, and the code there was identical to main
.
I've been following/researching two topics related to this:
- Issues with the parallel query running as we said (but I don't really understand what about the parallel thread is causing the error)
- Gatsby v4 taking up far more memory than previous iterations (I think possibly because of the parallel query running?) and hitting the Netlify soft-limit of 3GB, however I've added GATSBY_CPU_COUNT=2 to my env with no affect, so not sure this is part of it.
Any initial thoughts on these two potential causes? Does it seem like they are related? Would you recommend downgrading to Gatsby v3? I can do an image audit and try to move all images that are in the repo to Contentful but it doesn't seem like that's the golden ticket to me, if little content has changed prior to upgrading to Gatsby v4.
from netlify-plugin-gatsby.
Do you have a lot of images locally? I would certainly suggest moving them to Contentful if possible. The benefit there is that they don't need to be resized at build time, and can be served directly from Contentful. The Contentful integration with Gatsby image is very good.
from netlify-plugin-gatsby.
Yes there are around 100 images in the repo that are loaded using gatsby-plugin-image. I'll start there, thanks.
from netlify-plugin-gatsby.
Ok I've started the process of moving the images, but I was looking a little closer at a few things and have questions.
I ran a couple builds locally (I wanted to see if there was output in .cache/caches/gatsby-plugin-image/
after a build), and it failed at the same step as on Netlify... so I'm now thinking this conversation might belong better in the Gatsby PQR area.
Oddly, it fails on querying an image from Contenful at... w=40? I'm not sure why it would ever do that.
I'm thinking I need to take a closer look at how images are being pulled from contentful, especially when inlined in Markdown and transformed using gatsby-remark-images-contentful
.
Here's the error:
from netlify-plugin-gatsby.
Ah, yes in that case I'd take this to the Gatsby issue. I wonder if it's a rate-limiting thing with Contentful.
from netlify-plugin-gatsby.
Hi @ascorbic thanks for talking with me over the last week. I have actually ultimately isolated my problems to the package gatsby-remark-images-contentful
and the Contentful Long Textfield content type. It appears there is a step that happens in that process where the base64 of every image embedded in the Markdown. Since this happens as part of the queries it doesn't actually utilize gatsby-plugin-image for the processing and sourcing, and often makes the Run Queries in Workers
step fail entirely.
That said, I would imagine other people who have this issue likely have some similar plugin that is making extraordinarily long / async requests at the query step.
Thanks for your help and I'm closing this ticket.
from netlify-plugin-gatsby.
Thanks for letting me know! Glad you managed to track it down
from netlify-plugin-gatsby.
@benlavalley Can you share your build logs please
from netlify-plugin-gatsby.
@ascorbic Yep! The good news is I also have my Netlify deploys working again after I locked Gatsby back to =v4.1.3 in my package.json. Here is a snippet of my build log I think you're looking for (I can share the entire thing if need be) - I was actually running gatsby@next here, I believe 4.9:
I also see no such entry for gatsby-plugin-sharp.IMAGE_PROCESSING on my Gatsby 4.1 version.
7:51:42 PM: success onPostBootstrap - 0.876s
7:51:42 PM: info bootstrap finished - 24.515s
7:51:42 PM: success write out requires - 0.016s
7:53:05 PM: success Building production JavaScript and CSS bundles - 83.376s
7:53:54 PM: success Building HTML renderer - 48.486s
7:53:54 PM: success Execute page configs - 0.182s
7:53:54 PM: success Caching Webpack compilations - 0.001s
7:54:20 PM: success run queries in workers - 25.667s - 93/93 3.62/s
7:55:10 PM: success Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs - 209.669s - 102/102 0.49/s
7:55:10 PM: error UNHANDLED EXCEPTION write EPIPE
7:55:10 PM:
7:55:10 PM:
7:55:10 PM: Error: write EPIPE
7:55:10 PM:
from netlify-plugin-gatsby.
This does seem to have been a memory issue related to parallel query processing, introduced in a recent version of Gatsby. What data source are you using for the images? Are they coming from a CMS?
from netlify-plugin-gatsby.
This does seem to have been a memory issue related to parallel query processing, introduced in a recent version of Gatsby. What data source are you using for the images? Are they coming from a CMS?
Jut using flat files for my site content.
from netlify-plugin-gatsby.
Are any of the source images particularly large?
from netlify-plugin-gatsby.
Are any of the source images particularly large?
One was 14MB in size (just reduced to 2MB), a dozen or so are 1-2MB, and the rest are a few hundred KB.
from netlify-plugin-gatsby.
Does reducing the 14MB one help?
from netlify-plugin-gatsby.
Does reducing the 14MB one help?
It's a little tough to test only that change as I since added most of my static content back to my build and it's a bit of a pain to reduce it all again.
I did update to gatbsy@next (4.10 now) and it fails.
1:15:45 PM: success run queries in workers - 148.008s - 348/348 2.35/s
1:20:12 PM: success Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs - 553.793s - 540/540 0.98/s
1:20:12 PM: error UNHANDLED EXCEPTION write EPIPE
Reverting back to 4.1 builds OK.
from netlify-plugin-gatsby.
Depending on your source plugins, you may like to try the new image CDN feature. That will defer image processing, which can save a lot of memory. https://www.netlify.com/blog/gatsby-image-cdn-on-netlify
Another option is to disable AVIF, which is very memory and CPU-intensive. You can change the defaults to ["auto", "webp"]
, which used to be the default. See the docs on how. Despite what that says, the default currently include avif too: https://www.gatsbyjs.com/plugins/gatsby-plugin-image/#customizing-the-default-options
from netlify-plugin-gatsby.
Same issue with Netlify and 28 mega of local picture. Previously the last deploy work fin with Gatsby 3:56:00 PM: > [email protected] postinstall /opt/build/repo/node_modules/gatsby
my current version is :
"dependencies": {
"@emotion/react": "^11.9.0",
"@material-ui/core": "^4.12.4",
"@material-ui/icons": "^4.11.3",
"axios": "^0.26.1",
"gatsby": "^4.12.1",
"gatsby-plugin-image": "^2.12.1",
"gatsby-plugin-manifest": "^4.12.1",
"gatsby-plugin-material-ui": "^4.1.0",
"gatsby-plugin-offline": "^5.12.1",
"gatsby-plugin-react-helmet": "^5.12.1",
"gatsby-plugin-sass": "^5.12.1",
"gatsby-plugin-sharp": "^4.12.1",
"gatsby-plugin-web-font-loader": "^1.0.4",
"gatsby-source-contentful": "^7.10.1",
"gatsby-source-filesystem": "^4.12.1",
"gatsby-source-mongodb": "^4.12.1",
"gatsby-transformer-sharp": "^4.12.1",
"p5": "^1.4.1",
"prop-types": "^15.8.1",
"react": "^18.0.0",
"react-dom": "^18.0.0",
"react-helmet": "^6.1.0",
"sass": "^1.50.1"
},
IF I understand well, I must remove my pics to have something light, is correct ? But what happen for the case where the people don't want remove pics, because for me 28Mg is not so heavy...
from netlify-plugin-gatsby.
I stopped having this error when I downgraded my node version to 16.
Environment settings:
NODE_OPTIONS --max-old-space-size=8192
NODE_VERSION 16.13.1
NPM_FLAGS --legacy-peer-deps
from netlify-plugin-gatsby.
I stopped having this error when I downgraded my node version to 16. Environment settings:
NODE_OPTIONS --max-old-space-size=8192 NODE_VERSION 16.13.1 NPM_FLAGS --legacy-peer-deps
is it not too dangerous to move from node v14.18.2 to node 16.13.1 I ask because I'm not comfortable with the internet jungle programmatic !
from netlify-plugin-gatsby.
from netlify-plugin-gatsby.
NODE_OPTIONS --max-old-space-size=8192
NODE_VERSION 16.13.1
NPM_FLAGS --legacy-peer-deps
where you set that ? Because I update node v16.15.0
and it still doesn't work.
with this gatsby version :
"dependencies": {
"@emotion/react": "^11.9.0",
"@material-ui/core": "^4.12.4",
"@material-ui/icons": "^4.11.3",
"axios": "^0.27.1",
"gatsby": "^4.13.0",
"gatsby-plugin-image": "^2.13.0",
"gatsby-plugin-manifest": "^4.13.0",
"gatsby-plugin-material-ui": "^4.1.0",
"gatsby-plugin-offline": "^5.13.0",
"gatsby-plugin-react-helmet": "^5.13.0",
"gatsby-plugin-sass": "^5.13.0",
"gatsby-plugin-sharp": "^4.13.0",
"gatsby-plugin-web-font-loader": "^1.0.4",
"gatsby-source-contentful": "^7.11.0",
"gatsby-source-filesystem": "^4.13.0",
"gatsby-source-mongodb": "^4.13.0",
"gatsby-transformer-sharp": "^4.13.0",
"p5": "^1.4.1",
"prop-types": "^15.8.1",
"react": "^18.1.0",
"react-dom": "^18.1.0",
"react-helmet": "^6.1.0",
"sass": "^1.51.0"
},
from netlify-plugin-gatsby.
Related Issues (20)
- Test issue
- Background functions HOT 1
- Fix issues with the CI build process HOT 4
- Investigate reducing function .cache folder size by excluding media assets HOT 6
- Update README to include information on adding Background Functions to a site HOT 1
- Issue with linking github organizations HOT 1
- [Bug]: Error when building in netlify HOT 2
- [Bug]: Error: Cannot find module 'common-tags' HOT 6
- [Bug]: netlify build fails with gatsby v5 on node 18 with yarn2 pnp, "Cannot find module 'postcss-color-function'" HOT 1
- [Bug]: Netlify deployment fails with message: "Cannot find module '@gatsbyjs/reach-router/lib/utils'" HOT 3
- [Bug]: Netlify Gatsby plugin causes build failures on Gatsby 2.x site HOT 7
- [Bug] This plugin is running without enabling it HOT 3
- [Bug]: Netlify Gatsby plugin causes build failures on Gatsby 3.4 site HOT 3
- [Test] Verify GitHub workflow HOT 1
- [Bug]: Build has been failing for over a year on live website- only local netlify CLI deploys work HOT 1
- [Bug]: Netlify CI builds failing with error Cannot find module 'common-tags' HOT 2
- [Bug]: Deploying an SSR powered Gatsby app via the Netlify CLI breaks SSR pages HOT 1
- Support gatsby-config.ts
- [Bug]: Image CDN doc has a broken link
- Be more clear that this library is deprecated. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from netlify-plugin-gatsby.