webpack-contrib / cache-loader Goto Github PK
View Code? Open in Web Editor NEW[DEPRECATED] Caches the result of following loaders on disk
License: MIT License
[DEPRECATED] Caches the result of following loaders on disk
License: MIT License
Line 201 in 4e0e3ae
Need a local cache here? Because contextDep.path is mostly the same file.
Instead of ./cache-loader
, it should be ./node_modules/.cache/cache-loader
Consistent with other loaders and to avoid a specific gitignore.
Add an option to control the cacheDirectory
with an env variable (e.g. CACHE_LOADER_CACHE_DIR
).
Useful for keeping the default for local dev environment and controlling the value on CI/CD.
The cacheDirectory
option should still take precedence if specified.
Serialize this.loaders
options to invalidate cache when things change.
The remaining request looks like this:
file??ref--8-0!file??ref--8-1!file??ref--8-2!file
It is not clear if those refs change when loader options change.
As far as I can tell, cache-loader does not account for changes in options passed to loaders in the chain.
Hi,
I'm using cache-loader v1.0.3, and I love it for it saves a lot of webpack build time. (~1min)
The problem is that coworkers randomly experience build error of Empty files
. It happens from our build machines a lot randomly, so we stopped using it recently. Can you find the root cause of it? Stack trace is as below:
Empty files:
ERROR in node_modules/.cache-loader-cache/1ab816c016e1bd0ed02e3712897f9d0d.json:1
1:
^ Unexpected end of input
node_modules/.cache-loader-cache/46d561d9dbc4d7b6bb43238ffd0fea50.json:1
1:
^ Unexpected end of input
node_modules/.cache-loader-cache/879d8d4764e11a424b407b0c5eb9b921.json:1
1:
^ Unexpected end of input
node_modules/.cache-loader-cache/ec76dc763d9fbc22c5e4cc47972e2038.json:1
1:
^ Unexpected end of input
node_modules/.cache-loader-cache/ed921eb0045f85051790b9a0c17f6868.json:1
1:
^ Unexpected end of input
node_modules/.cache-loader-cache/fbb669036963f03d234002a129f75639.json:1
1:
^ Unexpected end of input
The default value for cacheIdentifier
is cache-loader:{version} {process.env.NODE_ENV}
. My project has multiple webpack config files where the step that uses cache-loader gets the same input from previous loaders but performs different transformations (eg. Istanbul instrumentation for test builds only).
I like that the default string includes the cache-loader version number, since it makes sense to invalidate the cache when this loader is updated. When setting a custom value, it is difficult to replicate this behavior since retrieving the cache-loader version number is tricky. It seems like the best options are hard-coding the version number into the string and hoping to remember to update it when upgrading to a new version of cache-loader, or some hacky manipulation of the path returned by require.resolve("cache-loader")
.
Couple of possible fixes:
cacheIdentifier
to the default string instead of replacing the default stringHappy to submit a PR with whatever fix is deemed best!
I would like to set my Cache TTL for my assests to 1 year by recommendation from Google PageInsights.
But I dont see in the docs where to do this. Im using webpack on my react client.
// webpack.config.js
const path = require("path");
module.exports = {
mode: "development",
entry: "./src/index.js",
output: {
path: path.join(__dirname, "dist"),
filename: "index.js"
},
module: {
rules: [
{
test: /\.png$/,
use: ["file-loader", "cache-loader"]
}
]
}
};
// index.js
import png from "./index.png";
console.log(png);
In this case. The .png
is broken because cache-loader
is not a raw loader. Which means the source
is an UTF-8 string
instead of Buffer
. The binary data will be broken during conversion processed.
Some loaders, e.g. file-loader
, added module.exports.raw = true
to solved this issue.
.png
file broken.
cc: @jimexist
This is a really great idea. However, there is an important issue that should be mentioned before anyone uses this in a production environment. At the very least there should be a big bold warning in the readme. Currently, this loader is assuming that the output of all affected loaders is going to remain the same forever for any given input. But updating packages from npm, or modifying options for the loaders can easily change this. To make matters even worse, some loaders read from configuration files for their options (e.g .babelrc). Basically, anything in the build environment that can potentially change the output of a loader needs to be included in the cache key. This is hard to solve generically for all loaders, because you don't know what a loader could be using that could affect its output.
A partial (but not easily usable) solution is to intelligently include what we can in the cache key (e.g webpack version and loader version/options for each cached loader). And then anything else will have to be manually added by the user.
Let's take babel-loader for example. It actually already has its own disk caching, and according to the readme their cache key by "default is a string composed by the babel-core's version, the babel-loader's version, the contents of .babelrc file if it exists and the value of the environment variable BABEL_ENV with a fallback to the NODE_ENV environment variable." So the user would have to tell cache-loader about the babel-core version, .babelrc contents and BABEL_ENV/NODE_ENV.
So this is not so great. The actual good solution to this problem would be to make disk cached loaders a standard thing within webpack. Then loader developers would be responsible for telling webpack about their cache dependencies and the user wouldn't have to do anything. That is the dream!
Please check the declaration of module dependency.
Module build failed (from ./node_modules/thread-loader/dist/cjs.js):
Thread Loader (Worker 10)
Cannot read property 'stat' of undefined
at toDepDetails (C:\agent\_work\2\s\node_modules\cache-loader\dist\index.js:65:14)
at arrayIterator (C:\agent\_work\2\s\node_modules\neo-async\async.js:3780:9)
at timesSync (C:\agent\_work\2\s\node_modules\neo-async\async.js:2292:7)
at Object.mapLimit (C:\agent\_work\2\s\node_modules\neo-async\async.js:3775:5)
at Array.<anonymous> (C:\agent\_work\2\s\node_modules\cache-loader\dist\index.js:88:18)
at arrayEachFunc (C:\agent\_work\2\s\node_modules\neo-async\async.js:2512:19)
at Object.parallel (C:\agent\_work\2\s\node_modules\neo-async\async.js:6867:9)
at Object.loader (C:\agent\_work\2\s\node_modules\cache-loader\dist\index.js:87:9)
Uing nwb for creating library, so here I want to use webpack and babel for some additional packaging functionality.
//package.json
{
"name": "react-app-dashboard",
"version": "1.0.0",
"description": "Describe react-app-dashboard here",
"private": true,
"scripts": {
"build": "nwb build-react-app",
"clean": "nwb clean-app",
"start": "nwb serve-react-app",
"test": "nwb test-react",
"test:coverage": "nwb test-react --coverage",
"test:watch": "nwb test-react --server"
},
"dependencies": {
"@babel/core": "^7.2.2",
"react": "^16.7.0",
"react-dom": "^16.7.0"
},
"devDependencies": {
"babel-core": "^6.26.3",
"babel-loader": "^8.0.4",
"babel-preset-env": "^1.7.0",
"babel-preset-react": "^6.24.1",
"cache-loader": "^1.2.2",
"webpack": "^4.28.3"
},
"author": "",
"license": "MIT",
"repository": ""
}
// nwb.config.js
module.exports = {
type: 'react-app',
babel: {
cherryPick: 'some-module',
plugins: ["babel-loader", "babel-preset-env", "babel-preset-react"]
},
webpack: {
publicPath: '',
extra: {
entry: './src/index.js',
output: {
filename: 'main.js',
path: path.resolve(__dirname, 'dist')
}
}
},
}
There some loaders, (for example tslint-loader), which only emit warnings or errors. It would be useful to cache and re-emit that stuff with cache-loader.
I'm trying to help debug an issue with using Webpack and Karma together. I'm thinking that it could be resolved if each cached module contained its inline source map (when devtool
is "inline-source-map") to prevent Webpack from having to reevaluate. Would this make sense for cache-loader?
Hi, what are the differences between this loader and the hard-source-webpack-plugin?
Also, will the cacheloader speed up paralell webpacks? I'm running webpack-dev-server and karma-webpack at the same time and am looking for something that can help speed up the two.
Regards,
Tarjei
Hi,
I know the relative paths has been a fairly recent feature addition (thank you!)... but I noticed that the cache contents still contain absolute paths.
This sets the correct relative paths in the cache details, but then I see things like this inside the cache file, under the "results":[...]
object:
...,{"version":3,"sources":["src/js/...
Is it possible to parse these "sources" as relative paths too - so that they context stays the same?
This is for javascript files running through uglifyjs-webpack-plugin
(not sure if that is relevant or not).
...,{"version":3,"sources":["/Users/jason/Sites/...
module.exports = (env, argv) => {
return {
mode: argv.mode,
...
module: {
rules: [
{
{
test: /\.m?js?$/,
exclude: /(node_modules|bower_components)/,
use: [{
loader: 'cache-loader',
options: {
cacheContext: path.normalize(path.relative(__dirname, '.')),
cacheDirectory: 'cache/'
}, {
loader: 'babel-loader'
}]
}
}
]
}
}
}
Happens in both modes development
and production
.
Command been using: webpack --mode production
If cache-loader cached files in the .cache-loader folder then on following builds mini-css-extract does not extract css files.
module:{
rules:[
{
test:/\.jsx?$/,
include:[srcPath],
use:[
{
loader:'cache-loader',
},{
loader:'babel-loader',
}
]
},
{
test: /\.(css|scss)$/,
include: [srcPath],
use: ExtractTextPlugin.extract({
fallback: 'style-loader',
use: ['cache-loader','css-loader', 'sass-loader']
})
},
{
test:/\.(png|jpe?g|gif|svg)(\?.*)?$/,
include:[srcPath],
use:[
{
loader:'cache-loader'
},{
loader:'url-loader',
options:{
publicPath:'/',
limit: 10000,
name: utils.resourcePath('img/[name].[hash:7].[ext]')
},
}
]
},
{
test:/\.(mp4|webm|ogg|mp3|wav|flac|aac)(\?.*)?$/,
include:[srcPath],
use:[
{
loader:'cache-loader'
},{
loader:'url-loader',
options:{
publicPath:'/',
limit: 10000,
name: utils.resourcePath('media/[name].[hash:7].[ext]')
},
}
]
},
{
test:/\.(woff2?|eot|ttf|otf)(\?.*)?$/,
include:[srcPath],
use:[
{
loader:'cache-loader'
},{
loader:'url-loader',
options:{
publicPath:'/',
limit: 10000,
name: utils.resourcePath('fonts/[name].[hash:7].[ext]')
},
}
]
}
]
},
Hash: c593f2eeb3a74cc82923
Version: webpack 3.5.5
Time: 40972ms
Asset Size Chunks Chunk Names
./js/vendor.26bfd90b6f42c7d80bfa.js 1.18 MB 0 [emitted] [big] vendor
./js/index.6ae9e86f6a638e7fe8be.js 876 kB 1 [emitted] [big] index
./js/manifest.92cf23c7c278d9152237.js 1.45 kB 2 [emitted] manifest
css/index-c593f2ee.css 342 kB 1 [emitted] [big] index
index.html 510 bytes [emitted]
No picture
e.g. should user reset cache when webpack or other build-lib's config/version changed?
Can you add explanation to readme?
As pointed out in #3, it's important to be able to expire the cache for this loader when the build environment changes. We're currently doing this by naming the cache directory based on a hash of relevant file content (NPM shrinkwrap, webpack config, etc) - however, over time this means developers accumulate a bunch of old cache directories, which need to be cleaned up manually or in a separate build step.
Would it be possible to add a cache-key option that, when changed, would cause the loader to just ignore and overwrite existing cache files?
cache-loader version: 1.0.3
Relevant rule:
{
test: /\.less$/,
use: extractStyles.extract([
'cache-loader',
'css-loader',
'less-loader'
])
}
Just added cache-loader and am finding that the output of my project less files doesn't get updated. Ran into it on multiple machines. No errors reported. It must not be invalidating the cache appropriately in this case. Works fine with JS and TS.
it should not cause memory leak
it causes memory leak
cache-loader babel-loader usage
{
test: /\.(js|jsx|ts|tsx)?$/,
exclude: [/node_modules/],
// use: 'happypack/loader?id=js',
use: ['cache-loader', 'babel-loader?cacheDirectory'],
include: [srcPath, path.join(cwd, '../')],
},
one optimization option that causes memory leak
optimization: {
removeAvailableModules: false,
removeEmptyChunks: false,
splitChunks: false,
}
another optimization option that causes memory leak (create 1 chunk per monorepo package)
const outModulesPrefix = 'entria';
const ourModulePattern = new RegExp(`^@${outModulesPrefix}\/([a-z]+[-[a-z]+]*)$`);
splitChunks: {
cacheGroups: {
default: false,
vendors: false,
vendor: {
name: 'vendor',
chunks: 'all',
test: /node_modules/,
priority: 20,
},
common: {
name: module => `feedback-${module.rawRequest.match(ourModulePattern)[1]}`,
minChunks: 1,
chunks: 'all',
maxAsyncRequests: Infinity,
maxInitialRequests: Infinity,
minSize: 0,
maxSize: 1,
test: module => ourModulePattern.test(module.rawRequest),
priority: 10,
enforce: true,
},
},
},
try to use this babel-loader config with this optimization settings in a big codebase
file-loader
outputs all to assets folder.
cache-loader
used with file-loader
.
While using with file-loader
, if output directory does not content assets folder โ get those files from cache and put in assets folder.
Webpack doesn't output assets folder, that shoul be generated by file-loader
.
{
test: /.(jpg|jpeg|png|svg|gif|ttf|otf|eot|woff|woff2)$/,
use: [
{
loader: 'cache-loader'
},
{
loader: 'file-loader',
options: {
outputPath: 'assets',
name: isDevelopment ? '[path][name].[ext]' : '[hash].[ext]'
}
}
]
}
Add cache-loader
before file-loader
.
Hello!
We've been continuing to tune our build performance, and are happy to see that cache-loader seems to perform slightly better than babel-loader's cache (and are definitely happy to see that it applies to other loaders as well).
We're seeing an issue with it though in that it stores absolute paths in the cache files, which means that the cache cannot be shared across our build workers. I was hoping that I could correct the paths in the cache files directly, but it seems that the cacheKey used for naming the files is also generated using absolute paths.
This means that it's totally unusable for us in CI, which is where I think the primary benefit will be seen.
Any idea if this would be possible to correct?
I notice that webpack has cache system itself, so I'm not sure whether keeping use cache-laoder or not.
I would like to know whether there are any unintended consequences from using cache-loader
from multiple processes writing into the same cache directory?
To give a bit of a background, I would like to create a manager for my project that would handle requests for building checked out project sources in parallel. At the moment, I am constructing a strong hash from the environment for all the loader dependencies that's added as a component to the cache output path.
I am curious if there would be any issues when I try to run two parallel builds when the environments are not different (cache output directories are the same), but my project sources changed? If so, perhaps this is something that might be added to the README
.
same question as #82
It would be great help if someone could figure out whether it's reasonable to combine the cache-loader with the cache feature of Babel. Whether to combine both or prefer one of them etc. I think that's also a pretty fundamental question the readme should answer - Thanks!
I was trying to speed up the cache read/write, so I decided to store stuff in memory with node-cache, instead of in the filesystem (default behavior).
To my surprise, the build time is slower with in-memory cache than with filesystem, it's so slow even a build without cache-loader is faster.
node-cache has pretty much the same function signatures as the example provided in the docs here for Redis. so my implementation is basically carbon copy of that
const NodeCache = require( "node-cache" );
const myCache = new NodeCache();
const BUILD_CACHE_TIMEOUT = 24 * 3600; // 1 day
// Read data from database and parse them
function read(key, callback) {
myCache.get(key, (err, result) => {
if (err) {
return callback(err);
}
if (!result) {
return callback(new Error(`Key ${key} not found`));
}
try {
let data = JSON.parse(result);
callback(null, data);
} catch (e) {
callback(e);
}
});
}
// Write data to database under cacheKey
function write(key, data, callback) {
myCache.set(key, JSON.stringify(data), BUILD_CACHE_TIMEOUT, callback);
}
...
module.exports = {
// rest of config
use: [
{
loader: "cache-loader",
options: { read, write }
},
{ loader: "babel-loader"},
{ loader: "ts-loader" }
]
}
My project is quite massive, hundreds of tsx files that need both babel and ts loaders (which is why I wanted to do caching).
Incremental build times:
See original comment @
070af07#r26826081
Looks like there is some timestamp-fu that is not allowing the .cache-loader/
folder to be cached across VMs.
When using cache-loader
configured with cacheContext
under a webpack in watch
and running a development configuration, it was expected to have the final bundles always in sync with the changed source files.
In the above mentioned cache-loader version when cacheContext
is used some relative paths are being used when calling this.fs.stat
. That call would sometimes return wrong stale metadata and the cache-loader would return the cached results for that file instead of the ones newly changed on disk.
Just use the mentioned cache-loader
version in a simple application configured with webpack in watch mode running a development configuration. Start changing the source files and you'll see that sometimes, despite having webpack running the compilation, the bundles are not updated correctly with the last changes on disk.
This is a bug fixed by #83 but we might want to further investigate the cause for this directly in webpack/webpack dependencies that's why I'm opening that issue.
/CC @evilebottnawi
The cache-loader readme mentions that the cache-loader needs to be added in front of other loaders:
Add this loader in front of other (expensive) loaders to cache the result on disk.
I find the phrasing "in front of" confusing, since the loaders are used right from left / last to first (use docs). So Looking at the code examples they should be used last, but added in front of the use configuration?
i use git for my project . when i checkout one branch to another. it got error in my npm start. but when i deleted this file, i restarted successfully. what can i do?
To allow caching in memory-fs
Here's how I can reproduce this in my private repository:
I have a dependency chain (which is convoluted due to legacy reasons):
application.html.erb
_internal_css.html.erb
app/assets/stylesheets/internal-base.scss
(which is processed via Webpack and its bundled path available in Rail ERB partials).app/assets/stylesheets/internal/base/index.scss
app/assets/stylesheets/internal/base/shared.scss
Here are the relevant contents (with the issue) of my stylesheets:
app/assets/stylesheets/internal-base.scss
:
... a bunch of imports (~38)
@import "./internal/base/index";
app/assets/stylesheets/internal/base/index.scss
:
... ~8 imports
@import "./shared";
... ~4 imports
app/assets/stylesheets/internal/base/shared.scss
:
... ~7 imports
... ~2000 lines of SASS
Any changes to the shared.scss
will not bust the cache! Changes to base/index.scss
will also not bust the cache! However, changes to internal-base.scss
will bust the cache!
In order for changes to get picked up in shared.scss
, I have to provide an the .scss
extension in the @import
statement for internal-base.scss
. That is the only change to get it tracked well by cache-loader
. base/index.scss
remains the same -- which is a bit odd.
I'm still trying to reproduce in a vanilla repository @ https://github.com/juanca/cache-loader-sass-bug
It might have to do with my webpack configuration in my private repository... I can provide snippets from it as requested.
Anyone had similar problems to these? Or are these known issues?
with eslint-loader set up to not fail on errors/warnings, and if cache loader is used with babel-loader, warnings/errors should show up consistently.
with eslint-loader set up to not fail on errors/warnings, and if cache loader is used with babel-loader, warnings/errors should show up consistently.
{
enforce: 'pre',
test: /\.(ts|tsx)$/,
include: [path.join(cwd, 'src/js'), path.join(cwd, '../shared/src/js')],
loader: 'eslint-loader',
options: {
configFile: path.resolve(cwd, './.eslintrc.js'),
emitWarning: true,
failOnWarning: false,
},
},
{
include: [path.join(cwd, 'src/js'), path.join(cwd, '../shared/src/js')],
test: /\.(ts|tsx)$/,
use: [
{
loader: 'cache-loader'
},
{
loader: 'babel-loader',
options: {
cacheDirectory: true,
cacheCompression: false,
compact: false,
babelrc: false,
plugins: ['@babel/plugin-syntax-dynamic-import'],
presets: ['babel-preset-tock'],
},
},
],
},
introduce a warning from linting, such as no-unused-variables, startup webpack with webpack dev server, and see the warning.
stop webpack dev server, then start again. you will not see the error.
Without cache-loader, the expected behavior above does happen. i'm not sure if the issue is with cache-loader, or with eslint-loader
npm 5.8+ set file mtime to October 1985 (See npm/npm#20439 ), it's a feature for them, so the cache-loader does not work anymore.
Furthermore, f24f723 tells to not trust file mtime.
Should we relly on file content hash instead ?
Currently all outputs are retrieved from a single directory. For large projects and certain hardware this might have a negative impact on performances.
Something that might help with that would be to store outputs into subfolders that contain the first two letters of the hash.
Image files emitted by the responsive-loader
should correctly cached or at least be written to disk.
All image files emitted by the responsive-loader
are missing, they are not written as files to disk.
// webpack.config.js
let webpackConfig = {
module: {
rules: [
{
test: /\.(jpe?g|png)$/i,
use: [{
loader: 'cache-loader',
}, {
loader: 'responsive-loader',
options: {
adapter: require('responsive-loader/sharp'),
quality: 100, // (default)
name: '[path][name]-[width].[ext]',
},
}],
},
[...]
}
webpack setup with responsive-loader
and cache-loader.
First leave the cache-loader disabled and let the responsive-loader
resize some images (e.g. during CSS build). Notice that the resized images are correctly written to disk as files.
Now enable the cache-loader, run the build again. Notice that these files are not written to disk anymore, they are completely missing. Also inspect the cache folder and notice there are cache files for these resized images, but too small for an image file and they contain similar base64 encoded payload as above.
The cache files are very small for image files, the base64-decoded payload looks like this:
module.exports = {srcSet:__webpack_public_path__ + "images/layout/header-1557.jpg"+" 1557w",images:[{path:__webpack_public_path__ + "images/layout/header-1557.jpg",width:1557,height:743}],src:__webpack_public_path__ + "images/layout/header-1557.jpg",toString:function(){return __webpack_public_path__ + "images/layout/header-1557.jpg"},placeholder: undefined,width:1557,height:743};
No binary data of the actual image files can be found.
after delete .cache-loader folder all images return 404
what's the difference between cache-loader and happypack?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.