files: list all required files for build tools about spec HOT 71 OPEN

It's a bit rude to link to that. While true, no reason to have to come off as a jerk about saying no.

from spec.

@OmgImAlexis No

from spec.

So the new "files" will hold all source files, and "main" will hold compiled versions again?
If so, I'd vote for proposal 1, it's simple enough.

from spec.

With proposal 2 I think you have the potential to accept valid MIME types as the category list. By doing this, it's not up to you to figure out what kinds of files people need and you offload that to a different organization where it is getting close monitoring. Personally, I prefer proposal 2 as I feel there is more clarity which would be useful to users who are new or inexperienced with a project.

from spec.

Does this mean that the dist/movement.css used in the example of the current spec can be in main? Because if that's the case, nothing really changes. Except the hard-exclude of movement.css is out of the way.

from spec.

@alber70g It doesn't read that way to me — this would put them in a separate files block. main would retain source files only is my understanding of the OP's post.

from spec.

So we can expect half a year of wiredep not working and dependency mayhem... Nice.

from spec.

files is an array of all the files required to use the package. Filepaths may include globs.

files differs from main in that main only lists single files per filetype, whereas files should include every file.

We recommend listing source files in files, rather than compiled distribution files.

The second line implies that we should keep only ONE file per filetype in main
The last line implies that we should put into files what is now in main in a sense that .less (or scss) is not allowed to be in main (which is the case, by looking at the current spec).

from spec.

Who is files meant to be consumed by? Any sufficiently advanced packer should be able to parse the assets listed in main and resolve all other dependencies (scripts, styles, images, fonts, etc.).

The problem is that naive dependency resolvers like wiredep don't have much file specific knowledge, they just want to resolve paths and group assets by extension. Depending on the file extension, they are injected into html or returned in an object (organized by extension/dependency) for further tools to consume.

There are several kinds of information I would find useful from a package (property names aren't important here):

• main: entry points for various file types that can be directly referenced from a browser. Not all file types support imports (e.g. ES5) and therefore concat + minification may be desirable for some.
• metadata: e.g. .d.ts or .less files with typing or variable information, but that compile down to nothing. Useful when doing development.
• sources: a list of source entry point files in the component that are needed for compilation (if no sources use imports/exports, then all source files should be listed). Don't list intermediate representations (e.g. ts -> js -> min.js, the js files should not be listed).
• assets: a list of static assets that are referenced by various other files (e.g. fonts, images, etc.) that should be copied into production so that unneeded files (sources, metadata) can be filtered out.

It may also be useful to offer different collections of files in the package that can be combined together with main as needed by the consumer (think various sizes, legacy support, languages):

{
  "main": [
    "js/motion-base.js"
  ],
  "options": {
    "lite": [
      "js/motion-lite.js"
    ],
    "full": [
      "js/motion-full.js"
    ],
    "huge": [
      "js/motion-huge.js"
    ],
    "legacy": [
      "js/motion-legacy.js"
    ]
  }
}

There would likely need to be some way for the consumer to specify which option(s) they want to be combined with main.

I think this gives enough information for a consumer to take the bower.json and pipe the various file lists into their tools of choice.

• metadata is used by the consumer to build their own projects/do development
• sources can go to a compiler (and then main should be ignored as you don't want to include stuff twice)
• main + options are injected as paths into html (or whatever you want to do with file paths)
• assets are retained when shipping to production so all unneccessary files can be stripped out (I already minified all js and therefore don't need to ship it)

from spec.

• Re: Proposal 1: files flat list:
  • Couldn't tools just do ls -R . and then subtract out the ignore entries and get the same result? If yes, I don't see the point; Bower or the tools could just generate the list programmatically at runtime instead.
• Re: Proposal 2: asset-type object:
  • Better, but I think it would be good to tie precompiled CSS to the preprocessor file that it originated from, which this proposal doesn't do.
  • Example/strawman:

"files": {
  "styles": {
    "dist/css/bootstrap.css": "less/bootstrap.less",
    "dist/css/bootstrap-theme.css": "less/theme.less"
  }
}

Other issues:

• What about bootstrap.min.css.map?
  • Can we assume the <foo>.map naming convention and not explicitly list these and just have tools check for them?
• What about bootstrap.min.css and bootstrap.min.js?
  • Can we similarly assume the <foo>.min.<bar> naming convention?

from spec.

I think proposal 1 is better, build tools will easily adopt it I believe

from spec.

Proposal one works for wiredep 👍 I wouldn't vote for globs, however. It would just slow things down and the publisher should be able to explicitly list what files are important per release. They've been doing it with main already anyway.

from spec.

+1 for Option 1 files, that is compatible with both wiredep and NPM specification...

Mime-types of Option 2 can be easily extracted automatically..

I'd go without globs, but with special behavior of npm:

The "files" field is an array of files to include in your project. If you name a folder in the array, then it will also include the files inside that folder. (Unless they would be ignored by another rule.)

from spec.

@sheerun What becomes the purpose/use of ignore then?

from spec.

@cvrebert It would be the same as .npmignore, but without separate file:

You can also provide a ".npmignore" file in the root of your package, which will keep files from being included, even if they would be picked up by the files array. The ".npmignore" file works just like a ".gitignore".

from spec.

@sheerun Alrighty.

@paulmillr Does Brunch have any thoughts about the proposals?

from spec.

I'm fine with globs.

However... we should note what problems we want to solve here... and which not.

1. What is with minified/optimized code/assets?
2. What is with "dynamic" code/assets as seen in https://github.com/angular/bower-angular-i18n where we only need one locale-specific file?
3. What is our solution to different builds (compatible-modes for older browsers, etc.)? A current problem for RxJS Reactive-Extensions/RxJS#724.
4. What is with source maps?

Do we answer these questions here or in a different place? I would like to solve real problems here.

from spec.

By the way, while we're at it, why don't we choose a more descriptive name, such as "sourceFiles" or "source".

from spec.

I've just read why you want files introduced..

I think Bower made mistake by encouraging to include only source files in main. Maybe we should roll-back a little and accept this pull-request: https://github.com/bower/bower.json-spec/pull/46/files

That said, I think files is good idea anyway, as a more explicit way of listing all files to include in package (not main entry-points, that's what main is for). It can be useful for packages without main files, like CSS libraries.

from spec.

+1 for @sheerun, please do!

from spec.

Would this exclude Less files from "main"? I don't know if I like that :/

from spec.

@donaldpipowitch No, less can be library entry-point as well. If you use less, you probably would be more interested in consuming .less main files. If you use sass, you would be interested in consuming .sass or .scss. If you use plain html, you want to use .css main files. Bower for better of worse tries to be language-agnostic..

from spec.

Okay, thanks. I asked because of this line in the pull request:

"This means that the files can be used without compiling for use by the consuming party."

from spec.

Technically .less and .scss files don't need compilation to be consumed by less, sass tools... But maybe it needs clarification.

from spec.

@sheerun @donaldpipowitch I've updated my pull request to give more clarification about 'high level' sources (less, sass, ts etc): alber70g@a93cd95

from spec.

@alber70g Now it's better, but you probably linked wrong commit. This is correct:

https://github.com/bower/bower.json-spec/pull/46/files

Note that images / fonts are not entry-points because you can't just "use them by default". They always need to be included from another resource (HTML / CSS). That's why they should be listed only in files (my opinion).

from spec.

Thank you all. That's better!

from spec.

@sheerun Yea, I linked the commit instead of the pull request

Well, you can reference fonts as library. For example if you are building a new 'bootstrap' and you want to include 3rd party fonts, they can be part of wiredep when you inject them into your less

For example:

newBootstrap.less

/*inject:fonts*/
/*endinject*/

from spec.

@cvrebert I don't like idea of including minified files in.

Brunch minifies the result. Non-minified ver is useful for debugging.

from spec.

Thanks for all the input.

I'm not keen on rolling-back main. I see a functional difference between main and files:

• main is for entry-point files. Useful for RequireJS, LESS, Sass, and tools that read file source to pull in additional dependencies.
• files is for build tools that do not read file sources, like Wiredep, and need to know all the files that should be included.

Build tools should build from source files.

I see this as the root dilemma, as exhibited by the Bootstrap/Wiredep issue. Bootstrap's source styles are in .less files. Build tools should build from those .less files. Packages should avoid pointing to compiled files — i.e. Bootstrap's dist .css file. Wiredep in fact does work with .less files. A LESS compiler can be added to the build workflow, either by Wiredep, or on the users build process' side.

from spec.

@desandro

.less files are a relatively simple transformation and a quick compilation step. What about languages like TypeScript that are more expensive to compile? If a component author wants to write their component in TypeScript, should all downstream users be required to setup the TypeScript compiler in order to use the component, even though they only care about the javascript?

from spec.

I don't know if it is practical, but maybe we need two bower modules then. One exposing the TypeScript and one exposing the JavaScript...? Or it should just add the .d.ts and .js, but never .ts.

from spec.

If this field, files, has the same behaviour as npm, all minified files should also be included.
It is a inverse of a potential ".bowerignore" file. Anything that's not in this list should be excluded when publishing.
wiredep would not know if a file is minified or not.
Tools shouldn't use this field as files just means all files that are dumped to the registry.

For the main field, I don't think the current spec justifies the definition of a "main" entry. Usually user would expect the main would be able to consumed by the end-user directly. However now we put coffee or sass in this fields so the user cannot use them before they compile them. Think of npm, would you put coffee in the main entry? You only put the compiled js in the field.

I don't think neither these fields are supposed to be used by build tools. A field with more comprehensive information should be defined.

from spec.

Well you could put a .coffee file into the package.json. We did something similar for a short time with .es6 files for non-public modules so we don't needed to use Babel for every release in every module while we were in a prototyping phase.

We currently use "main" in Bower very heavily for Less files.

from spec.

I have a thought. We can defined a dist field, just like the directory name. Simple and easy to understand.

Under the dist field, we can defined each file purpose.

for example:

wiredep install -s index.html -dist product

from spec.

@desandro main is for entry-points of different file formats. Compiled files are another file format so who not put them in main.. If someone has less compiler he can consume main entry of .less format. If someone doesn't want to use it, she/he can consume pre-compiled .css file. I think "source files" requirement can be superfluous, and the only thing that matters is "one file per format".

from spec.

I think we should make things simpler. keep main and files exactly the same as npm (@donaldpipowitch I'm not sure about putting a coffee file into package.json though). adding another field just for defining different types so that tools can better pick up.

Think about this, npm is supporting browser side as well (either with browserify or not), they are not gonna make main an array of files even if they decide to support client side even better. They don't worry about other file types such as css that much. So on top of that we could add another field just for this purpose.

Benefits are:

1. It won't confuse the npm users.
2. Ideas or concepts between package managers are similar.

Once bower matures they will probably learn from bower.

from spec.

Or we make it completely different and look into this discussion again: bower/bower#1520. npm is fine with hijacking it and just add custom bower fields. If we try to align more with npm (which is good), we could just build Bower on top of npm and add frontend related fields (to import Less files or whatever).

from spec.

@sheerun Okay, I see your point. Listing both a .less file and .css that have the same functionality file wouldn't be such a bad thing as build tools can use one or the other. I still worry that we're leading build tools to build with files with duplicate functionality, sounds like that's just me :P

@stevemao #47 (comment) files is not just the anti-ignore. For example, the package (like Packery) may include multiple .js files and a single compiled dist .js. I include both in the bower package because they're both useful in different ways. But I only include one set or the other in files.

@stevemao #47 (comment) Yeah, I'd like to think Bower is the (one of the) first package managers to address this issue. Front-end is messy because of wide variety of tools.

Sounds like things are coalescing. Here's a revised shot at the spec:

files is an array of all the files required to use the package.

files differs from main in that main only lists single files per filetype, whereas files should include every file.

Adding a folder will include every file in that folder.

"files": [
  "js/motion.js",
  "js/run.js",
  "js/walk.js",
  "sass/motion.scss",
  "sass/run.scss",
  "sass/walk.scss",
  "dist/movement.css",
  "img/motion.png",
  "img/walk.png",
  "img/run.png",
  "fonts/icons.woff2",
  "fonts/icons.woff"
]

With this example (using the files listed in the main example), both the .scss files are and the dist .css file are listed. The dist .js file is not listed as it would be duplicated by the other .js files.

dist files concern me because they may include source code from other packages. That's how all my packages work at least. If they contain only source code from that package, they're good, i.e. Bootstrap. But once extra-package source code gets in dist file, it could lead to duplicated code.

from spec.

This sounds reasonable, but I think it should be made very clear in the documentation that files should not include multiple files of the same type that would result in code duplication.

How should dist files be handled by bower? Will there be another property for them, or should they be documented and referenced directly by consumers?

from spec.

I think the reason here we can't agree on certain things is we are trying to solve too many things at once (by changing the things that are already good and stable enough). We also mix up the ideas with other package managers.
@donaldpipowitch bower in fundamentally different than npm no matter what you do. I see two things about the it:

1. npm does WELL with files and main entry. There is absolutely no confusion about them. Since they made it very well we could just borrow the idea (especially now that the bower core team members can't agree on things). It's not wrong to learn and align with something good.
2. npm does NOT do well with other file types. This could be addressed by another field (In my early comments).
   @desandro I think your idea is very clear and makes sense but I wouldn't modify the existing fields that are proven (by other package managers maybe, but doesn't matter) to be good. Again I would put these in a new field.

I still worry that we're leading build tools to build with files with duplicate functionality

This is exactly why I think listing both js and coffee files in main isn't ideal and I can't see your proposed files field can fundamentally solve it.

I think there is nothing wrong to keep main and files the same as npm. We should instead address the problem with npm (and other package managers) by not modifying main, files or other fields that are already good enough. Introduce something new instead. This also makes each entry fields duty very atomic.

from spec.

bower in fundamentally different than npm no matter what you do.

1. npm does NOT do well with other file types. This could be addressed by another field (In my early comments).

Looks like they don't differ that much and we're allowed to add new fields to package.json. :)

from spec.

I don't think bower should breaks it previous rules.
main is good enough, file and main are duplicated.
for entry point for some application, maybe you can change the structure from
main : [ 'file1']
to
main : [ {name: 'file1', entry: true}]
which is not only that rules will be consitent with previous bower version but also new adjustment can be adopted for further use.

from spec.

I don't think bower should breaks it previous rules.

@calidion The whole problem is that its previous rules were so vague that it wasn't clear what the boundary was between correct and incorrect.
Is main.sass "required"? For Sass users, obviously yes. For other users, no.
Is dist.css "required"? For Sass users, obviously no. For other users, yes.
Thus, these two files were simultaneously absolutely required and completely optional, depending on your interpretation, making the correct value of main clear as mud.

from spec.

@cvrebert
thanks for the explanation.

I think user defined configuration may play the main role.

if the users choose to use sass, they may define their configure in bower.conf

files: 'sass'

default styling would be 'default' or none.

then in new introduced files field, we introduce two configuration versions of files.

1. simple version:

"files": [
  "js/motion.js",
  "js/run.js",
  "js/walk.js",
  "sass/motion.scss",
  "sass/run.scss",
  "sass/walk.scss",
  "img/motion.png",
  "img/walk.png",
  "img/run.png",
  "fonts/icons.woff2",
  "fonts/icons.woff"
]

1. complicated version.

"files": {
  'default':  //mostly for js, css only, no sass or less
   [
  "js/motion.js",
  "js/run.js",
  "js/walk.js",
  "sass/motion.scss",
  "sass/run.scss",
  "sass/walk.scss",
  "img/motion.png",
  "img/walk.png",
  "img/run.png",
  "fonts/icons.woff2",
  "fonts/icons.woff"
 ],
 'sass': [],
 'less': [],
 ...
}

which will meet all possible needs for different uses and different configurations.

if the bower.conf is files: 'sass',
then the sass of the files will be choose by wiredep or some other tools.

Thought, I would like bower.json to keep main entry rules from previous version until time is ready to remove other files.

from spec.

@calidion Your "complicated version" is basically #21.

from spec.

@cvrebert

a little different is that it will include all kind types of files.

from spec.

files should just be anti-ignore. A "complicated" field should be defined for different consumers.

from spec.

@stevemao Would files be optional for those of us who already set a proper ignore value? As a package maintainer, I see no point in semi-manually maintaining a field that bower/wiredep could so extremely easily compute itself using ls -R.

from spec.

@desandro While this is still being hashed out, is it worth considering reverting the spec change? I'm happy to tag wiredep along with wherever the Bower boat steers us, but for the sake of the users, it would be great to allow main to live on as it has been interpreted by the community thus far, and consider other metadata options for the gaps.

from spec.

What a mess. Got here from a build failure scratching my head. The de-facto meaning of main has been betrayed! Nooo! Now to wait for other build packages to catch up or patch things with custom main and then wait for the eventual breakage again when those tools use the new files field. Sigh.

from spec.

@calidion maybe a good idea to respect people and their ideas rather than posting what you just did.

from spec.

@calidion You're way off-base. None of that gives you a right to treat people like dirt. Seeing as an apology is unlikely forthcoming, I'm going to mute this issue.

from spec.

@benschwarz I apology for what i have said, but I hope as a base system, considering more on every changes you made, and what would be the subsequences for client projects.

from spec.

Just thought https://atom.io/docs/v1.0.3/hacking-atom-package-word-count#package-json might inspire you guys :)

from spec.

+1 for files with flat list

As an aside - the changes to main also break this grunt task - https://github.com/yatskevich/grunt-bower-task - I think "files" is actually what this copy functionality needs rather than main.

from spec.

After letting things cool off for a while... 🐧 I think ditching this proposal and reverting main back to including all files may be the way to go. This is already the common place practice. This will get Wiredep support back on track. As @notbrain points out, advocating for adopting a new field will likely have its own difficulties.

from spec.

As for me I'm convinced that listing only one file per extension in main, and extracting all the needed dependencies from them (using e.g. webpack) is the way to go. Listing needed files manually is too error prone and doesn't capture e.g. dependencies between assets..

from spec.

Technologies are like products, when they fit for demands, they will grow, otherwise, they will die.
compatibility is the key to success for technologies, Anyone who breaks it will be published.

from spec.

But breaking compatability is how new standards and best practices are made. To only look at the short term is short sightsighted and misses the big picture goals

from spec.

@kkirsche but what this boils down to is a naming convention preference. The main files list can be used as the de-facto standard for compiled/assets (as @stephenplusplus points out) and a new list files or whatever it will be called can be used to list all core source files. I'd call it sources or rawsrc or something since main still makes more sense as a list of files that allow me to use a package as a library, not as a main entry point to files that need further processing. We shouldn't need to process main files, especially if up until this change we didn't need to.

from spec.

@desandro Whilst it might be simple for bower to take this approach, those who utilise bower will have to deal with the breaking change in specification. It doesn't just affect the direct consumers of bower, but those who use components which use bower (think grunt/gulp tasks). This is mostly a pain because there's no good mechanism for communicating the new specification to all consumers.

For me, Bower is a tool which allows devs to get on with the work that really matters to them... If any tool takes more effort to maintain than the primary goal, people just stop using the tool. This will be a real pain for CI builds.

It took a long time to settle on a specification. Personally I feel it would be unwise to change it now rather than extend it as originally proposed in this post.

from spec.

Is there a reason to not just revert this change and re-release this version then do a major version bump to forcefully deprecate the old spec while moving to a new version with a new spec? I can only understand the frustration when there isn't a major version change. With one, sorry but breaking changes are the entire reason for major version numbers.

from spec.

@kkirsche

Is there a reason to not just revert this change and re-release this version

Just to clear something up: we already changed main #43 this change would be add a new field files

@sheerun

As for me I'm convinced that listing only one file per extension in main, and extracting all the needed dependencies from them (using e.g. webpack) is the way to go. Listing needed files manually is too error prone and doesn't capture e.g. dependencies between assets.

Thanks for this confirmation. Current main spec follows this.

Now I'm flip-flopping on my previous comment. "main listing one-per-type" is a good change. It does break backwards compatibility with some tools like Wiredep. However, by adding a new field, these tools can maintain compatibility with packages that use main the old way. A fix would look like checking for files before using main.

// use files field if there, otherwise fall back to main field
var files = manifestJson.files || manifestJson.main

from spec.

I was looking up to see if bower contained something similar to npm's file list to simplify my module (I'd rather be explicit with what I want included versus maintaining an ever growing list of ignores) which ended with my stumbling across this thread. This issue has been open since June and adding a files property would bring yet another thing of bower's to parity with npm; why wasn't that simply done and merged in? This is just a huge issue of arguments over using main or source or sourceFiles or min vs full vs other assets.

Just make a file list that includes everything that should be downloaded by bower. When a developer wants to use one file versus another, in their deployment scripts they can decide what is or isn't taken and that's it. That's how most other package managers do it and it's easy. The simpler the better.

Besides the more parity with how npm works the easier bower/bower#1520 is to close (in one direction or another).

from spec.

npm's files is something different than proposed here (it's simply an inverse of ignore)

We could include it as well, but I don't like attaching it the meaning of "of all the files required to use the package". I especially don't like the idea of build tools using this list to select files to auto-require in project.

If we introduce files as just inverse of ignore, I'm perfectly fine with that.

If wiredep wants to interpret this list as all files to include in project, I cannot stop them from doing so, but I think it's a wrong approach. It's better to extract dependencies with tools like webpack.

How files would be implemented in bower? Any files not listed in files would be removed after downloading project. It means files needs to list even optional dependencies. Just like npm does it.

Additionally we'll make it clear files don't specify the order to load these files (some people assume main does it...). files will be non-ordered list files to include in the package. It can include globs.

Are you OK with it?

from spec.

attaching it the meaning of "of all the files required to use the package"

But that's not what it means in the npm world; those are simply the files bundled with an npm install and nothing more. Yes some are required but not all. You typically pull down more files than what is required (documentation, sometimes examples, etc).

If we introduce files as just inverse of ignore, I'm perfectly fine with that.

👍 ship it :). I'd work on the merge request but I'm unlikely to have time for a few more weeks (though if no one does it I'll certainly just do it).

from spec.

+1 for Proposal 2

from spec.

@desandro after a year and a half has anything happened to this?

from spec.

The reason I ask is I ended up in in a 20+ page rabbit hole to find this page with multiple different projects relying on each other's issues. I wouldn't have asked otherwise.

from spec.

@OmgImAlexis A year later, I'd say the recommended approach for developers is to use webpack to automatically discover assets that really need to be bundled. If you're package author, you can still use ignore field of bower.json to exclude unnecessary files.

from spec.