Originally here: rubygems/bundler#6298

I'd like to propose a new feature that allows naming of gem sources.

Before I start wrestling with the DSL code, I'd like to get some feedback from the community.

What we currently have

source "https://rubygems.org"

gem "other_gem"

gem "my_private_gem", source: "https://my-gem-server.org"

OR

source "https://rubygems.org"

gem "other_gem"

source "https://my-gem-server.org" do
  gem "my_private_gem"
end

What I'm proposing

source "https://my-gem-server.org", name: :my_gem_server

gem "other_gem"
gem "my_private_gem", source: :my_gem_server

OR

source "https://rubygems.org"
source "https://my-gem-server.org", name: :my_gem_server

gem "other_gem"

source :my_gem_server do
  gem "my_private_gem"
end

Why?

Because I'd like to be able to change the source URL in one place and have it update in other places within the Gemfile automatically, keeping things DRY and easy to maintain.

I also feel there's value in being able to see a list of gem server sources at the top of the file instead of trawling through the whole file looking for source URLs.

@indirect @segiddins @hsbt Thoughts?

In the JS ecosystem there is a tool called npx which downloads a npm package and directly executes it.

Read about it here https://github.com/npm/npx

I would love to have this too in the ruby ecosystem, not sure if it needs to be part of gem or bundler. But let me know!

bundle exec -i rails/rails new blog

or

gem exec rails/rails new blog

Let me know what you all think.

I wanted to open a discussion about supporting gem namespaces. Between this repo and the RubyGems.org repo, I wasn't sure where to start this discussion. If this was opened in the wrong place, I apologize, and feel free to close it.

Regarding the issue, though: currently, anyone can go and claim any gem they want, and it is difficult for organizations with a large number of gems (such as rails, aws, etc) to reserve rights to naming. It's not only confusing to customers, but can also be a concern for security if a malicious actor claims a similarly named gem and labels it as official. Other package managers (such as NPM's scopes) have support for namespaces.

My thoughts would be to use a / or :: for namespacing, such as gem install aws::s3 or gem install rails/actionwhatever

Bundler is on step away from being perfect and that would be when Bundler (and RubyGems) was completely from the application runtime.

Lets say we have this simple Rails Gemfile:

source 'https://rubygems.org'

ruby '3.0.0'

# Bundle edge Rails instead: gem 'rails', github: 'rails/rails', branch: 'main'
gem 'rails', '~> 6.1.2', '>= 6.1.2.1'
# Use sqlite3 as the database for Active Record
gem 'sqlite3', '~> 1.4'
# Use Puma as the app server
gem 'puma'

This transforms into Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    actioncable (6.1.2.1)
      actionpack (= 6.1.2.1)
      activesupport (= 6.1.2.1)
      nio4r (~> 2.0)
      websocket-driver (>= 0.6.1)

... snip ...

PLATFORMS
  x86_64-linux

DEPENDENCIES
  puma
  rails (~> 6.1.2, >= 6.1.2.1)
  sqlite3 (~> 1.4)

RUBY VERSION
   ruby 3.0.0p0

BUNDLED WITH
   2.2.3

But what is really needed for application runtime? Just properly initialized $LOAD_PATH. So the Gemfile.lock above could be transformed into lets say Gemfile.rb:

$LOAD_PATH.unshift('/usr/share/gems/gems/actioncable-6.1.2.1/lib')
$LOAD_PATH.unshift('/usr/share/gems/gems/actionpack-6.1.2.1/lib')
$LOAD_PATH.unshift('/usr/share/gems/gems/activesupport-6.1.2.1/lib')

... snip ...

and Bundler also preloads the libraries by default, so there could also be:

requires 'action_cable'
requires 'action_pack'
requires 'active_support'

... snip ...

This way, one could run the application later via e.g. RUBYOPT="--disable-gems -r./Gemfile.rb rails server"

The RFC to require MFA on accounts that own a gem with over 180 million downloads has been accepted and currently is being rolled out 🎉 In the last section of the RFC, it stated that

Once the rollout of the top 100 most downloaded gems is successfully rolled out, a similar rollout will be done for more gems. The details of this phase will be fleshed out in another RFC.

This issue is being opened to create a discussion on how we should implement this phase in the current MFA rollout! The ideas formed here will guide what will be drafted in a formal RFC.

Topics to discuss

• What metric should the rollout be based on? Based on discussions on the original RFC, there seems to be two possible solutions. Open to other ways we can approach this.
  • Dependency chain popularity. Popular gems’ transitive dependencies would require MFA
    • How is this going to be calculated? Number of dependents?
  • Total download count
    • Decrease the current require MFA threshold to desired threshold
• How many gems/accounts should have MFA required? All gems? Exclude personal projects as they won’t be a part of any supply chains?
  • From the original RFC, “We don't need to enfor[c]e mfa on gems if they are just personal projects. Anything below 20k downloads would fall this category for me (includes 83% of gems)” (ref)
• number of rollouts or cohorts

Other package ecosystems

Npm has rolled out the following policy:

In February we enrolled all maintainers of the top-100 packages on the npm registry in mandatory 2FA, and in March we enrolled all npm accounts in enhanced login verification. On May 31, we will be enrolling all maintainers of the top-500 packages in mandatory 2FA. Our final cohort will be maintainers of all high-impact packages, those with more than 500 dependents or 1 million weekly downloads, whom we plan to enroll in the third-quarter of this year…

It would be awesome if bundle list -v would list the actual descriptions of the gems and the repos. This will help when adopting a new project.

bundle list -v 
Gems included by the bundle:
 * rails (5.2.3) - Full-stack web application framework [https://rubyonrails.org]

Etc etc

Problem

When running bundle gem my_gem several files are created, none of which include # frozen_string_literal: true which is a default requirement of some linters like rubocop.

Solution

Include that line by default in generated ruby files (e.g. ), or add a flag to indicate the comment is desired in the new gem.

Some of the files that could include this line:

• my_gem/lib/my_gem.rb
• my_gem/lib/my_gem/version.rb
• my_gem/spec/my_gem_spec.rb
• my_gem/spec/spec_helper.rb

Note: This is a really small change, and might not even consider it an improvement so if there isn't too much support for this we can simply close it.

Introduction

MFA is an ongoing subject of work for Rubygems, and for good reason. Research has shown[0] that account takeovers are the second-most common form of attack on software package repositories (after typosquatting). As a class of countermeasures, MFA is the best single way we have to reduce the probability of account takeovers.

We are not alone in facing this difficulty. Most notably, NPM has had a recent high-profile spate of takeover attacks on popular packages, which were only detected because of errors in the embedded exploit code. Other cases may well be undetected in the wild.

Yesterday, NPM folks announced that they will make MFA mandatory for "a cohort of top packages":

... we will begin to require two-factor authentication (2FA) during authentication for maintainers and admins of popular packages on npm, starting with a cohort of top packages in the first quarter of 2022. We are currently evaluating next steps to ensure that the strongest and most user-friendly authentication options, such as WebAuthn, are available and accessible to developers using npm.

I think Rubygems should follow this precedent.

Some Implementation Questions

I recognise that it's not as simple as flipping a toggle, or even doable in a single PR. But I believe we should at least aim for this policy and take incremental steps towards it.

Some problems that need to be solved for this to become a policy:

• Which gems are selected for the policy? Top n-thousand most-downloaded?
  • What about highly-downloaded gems for which there has been no login in a long time?
• When is the policy enforced? At login? At push?
• How do we deal with the CI/CD usecase, for which human intervention is often seen as undesirable?
• Do we need to finish WebAuthn support first, so that those using USB tokens and biometric tokens are covered?
• How is the policy announced? Blogpost? Email to owners of selected gems?
• When would the policy take effect?
• Should the policy be rolled out in phases? Eg. simple nagging at first, followed by hard enforcement later?
• Should the policy be rolled out in stages? Eg. top 10 gems, then top 100 gems, and so on?

Next steps

The team I belong to at Shopify would like to develop an RFC for this policy change. We are enthusiastic about raising the MFA bar for Rubygems and we'd be prepared to do a lot of the technical work in support of an accepted RFC.

[0] See Towards Measuring Supply Chain Attacks on Package Managers for Interpreted Languages:

We categorize malware by their attack vectors in Figure 8a, which shows that typosquatting is the most exploited attack vector, followed by account compromise and publish. [emph added]

Context

A problem that has always existed forever, but is of course becoming bigger as time passes, is abandoned gems. You may depend on a gem, and one day when trying to upgrade Ruby or another gem realize it now has a very simple compatibility issue, but the original maintainer is no longer active, hence the gem will probably never receive an update.

If the gem is a “leaf”, meaning no other gems depend on it, it’s easy to just fork it and publish it under another name. However if other gems declare a dependency on it, the solutions are much more limited. You can reference a git fork and include the gem that way, which works but tends to be a lone solution rather than a community one. It’s rare, and rather not recommended to point your Gemfile at someone else’ fork of a repository.

Another possibility is to try to reclaim that gem name to become the new maintainer, but it’s slow and rarely works.

Proposal

I believe this community problem could largely be eased if it was possible to declare that a gem is standing up for another one in your Gemfile.

gem "console-formatter" # has a dependency on "left-pad"
gem "left-padder", as: "left-pad"

In the above example we assume that left-pad, a very useful and popular gem that is pulled as a dependency by dozens of bigger gems, is no longer working with a future version of Ruby, and the maintainer is no longer active.

I, as another open source contributor, can fork that gem, give it another name (left-padder) and then instruct users that they can replace left-pad with it, by specifying it in their Gemfile.

If the situation persists and the left-pad maintainer remains inactive, the various gems that depend on left-pad can progressively decide to update their dependency declaration, but in the meantime users have a way out of the problem, and are unblocked.

Additional Use Cases

Such a feature would also allow for secondary use cases.

For instance, maintaining alternative versions of active gems that include features or changes the maintainer doesn’t wish to include upstream.

Another use case is for community maintained versions of EOL releases. If left-pad is currently at version 2, and the version 1 is EOL but a large community of users are still on version 1, they can maintain a left-pad-lts gem if they want.

👋 Hello there! Long time RubyGems user, first time contributor. 😸

My current problem is that, as a Ruby open source developer, I would like to ensure that my projects are receiving funding, so that I can continue to make them available.

I noticed that late last year, npm discouraged the use of asking for funding in a post-install message, in favor of adding a funding option.

I would like to propose adding a similar option here, and, if it sounds good, would be more than happy to open an RFC, and then a PR, to implement ASAP. I am opening this issue to "get feedback on [my] concept before diving into the level of API design detail required for an RFC," as per the README.

Format

I imagine this proposal to add a new key, funding, to the Gem specification:

Gem::Specification.new do |gem|
  gem.name          = 'commonmarker'
  gem.funding       = 'https://github.com/sponsors/gjtorikian/'
  
  # ...
end

funding must be a URI, much like homepage.

Usage

After bundle update or bundle install (or really, wherever post_install_message is also shown), the following message should be printed:

4 gems are looking for funding
  run `bundle fund` for details

fund would be a new command. When a user types bundle fund, the following information should be shown as a bulleted list.

$ bundle fund
  * $GEM_NAME ($GEM_VERSION)
	Funding: <$FUNDING_URL>
  * commonmarker (0.20.2)
	Funding:  https://github.com/sponsors/gjtorikian/

As well, bundle info should also show this information:

$ bundle info commonmarker

  * commonmarker (0.20.2)
	Summary: CommonMark parser and renderer. Written in C, wrapped in Ruby.
	Homepage: https://github.com/gjtorikian/commonmarker
	Path: /Users/gjtorikian/Development/commonmarker
	Funding:  https://github.com/sponsors/gjtorikian/

(Of course, RubyGems.org should also prominently display this information, but that's a different PR for a different repo!)

What do you think?

High-level proposal

Add a new flag to the update command to produce machine-readable output, for better integration of Bundler with tooling. EG:

$ bundle update --machine-readable
[
  {
    "name": "nokogiri",
    "old": "1.13.6",
    "new": "1.13.8"
  },
  {
    "name": "rails",
    "old": "6.1.6",
    "new": "7.0.3.1"
  },
  {
    "name": "tzinfo",
    "old": "2.0.4",
    "new": "2.0.5"
  }
]

Motivation

It would be really useful to have machine-readable output from the update command.

This would make it easier for tooling (CI/CD, security inspections, automated updaters, posting summary messages) to consume & understand the changes made by dependency updates. Currently the only options are parsing the human-readable output from bundle update (which is not guaranteed to stay the same) or diffing the lockfile (which also doesn't guarantee its API, I don't believe?).

For example, I have a Github action that runs daily and checks for Gem updates on my projects. If there are updates, it creates a PR, and writes a summary in the PR text of the change it's making. I could make that summary easier to read if the output from bundle update was more structured. I could also potentially add a feature to auto-merge any update that only has minor version updates and passes tests. I could add an action to post to Slack when major updates are pending, so I know to check changelogs and see if I need to make any changes before updating.

This is a sister issue to rubygems/rubygems#5913, but whereas that would make it easier for humans who are using bundle update directly to understand what has changed, a machine-readable output would make it easier for similar more useful summaries to be produced by automated systems that invoke bundle update.

A machine-readable option would also make it easier for users who are using bundle update directly, but have different requirements or opinions to the solution in rubygems/rubygems#5913, to produce output summaries in a format or slicing of their choosing.

Key Challenges

Figuring out the changeset would already be required for rubygems/rubygems#5913, and I believe that part is relatively easy: we have sufficient metadata hanging around after doing an update to understand which gems changed and what the old and new versions are.

Producing machine-readable output from that data is also relatively easy. JSON seems like a logical choice. Apparently we cannot use the json library? But producing simple json by hand is not hard, so that is not a blocker. YAML is another option, but more tooling uses JSON generally speaking (e.g. jq) and it has fewer footguns both for writing and parsing. Another perfectly decent option would be to produce column output, CSV, or some other simpler format, but I think a more structured format such as json or yaml will make it easier to extend in the future without breaking consumers.

The hardest part would be only producing the machine-readable output, and that's why I am opening an issue to discuss the RFC rather than going straight to creating an RFC.

A lot of different parts of code can and do print to stdout in Bundler, and it would not be trivial to add a flag to bundle update that makes all the output machine readable.

Potential approaches to producing only machine-readable output

There is potentially an easy, albeit quite horrible, way to get machine-readable output that won't have other human-readable messages dropped into the middle of it. We could define some special message e.g. "MACHINE-READABLE-STARTS-HERE" that will be output at the very end, after all human-readable output, followed by the machine-readable summary. This will add an extra hurdle to anyone wanting to consume it: rather than being able to write e.g. bundle update --machine-readable | jq . they would have to do something like bundle update --machine-readable | sed -e '1,/MACHINE-READABLE-STARTS-HERE/ d' | jq ..

The next easiest, and far nicer method for consumers, would be to carefully pass the option to suppress human-readable messages down through every code path that an update calls that might print out a message. This goes a little deep in places but would be somewhat feasible, although not massively tidy — not all of the code passes down options or context, so that will need to be added in places. This, however, might be fragile: new output messages might be added to other functions in the critical path of update which would then break the machine-readable output.

Conceptually the nicest way to handle this would be to upgrade the message output framework itself in Bundler to have two modes. As all output is already mediated through a single output framework within Bundler, this would then mean that a single switch for human/machine could reliably capture all messages, even those added in the future. This would require substantial changes to that framework, though. But with this method we probably have the best chance to not just capture the summary of version changes, but any other messages relevant to the update. It would also be the most robust and least likely to break.

Key questions

In my mind the two key questions here are:

1. Would we want to make all output machine readable (including warnings about gems that failed to update, warnings about gems that went down in a version, post-install messages, messages about not updating particular groups, messages about platform mismatches, etc, etc), or would we settle for just the info on version changes?

If we do want to capture e.g. warnings and all, that pushes me more towards the larger change of updating how messages are output, so these can all be captured — otherwise it will require a lot of changes on an ad-hoc basis that I think will be more work in the long run, and fragile.

If there is less value in anything except version updates, maybe the amount of changes needed for producing only those without reworking how messages are printed isn't too high.

2. Would there be any interest in making other Bundler commands have machine-readable output?

I haven't yet spent any time thinking about whether there might be value for tooling and other applications in other Bundler commands also having a stable, machine-readable interface. If there is, that would be another reason to change the output handling, so this could be done more easily and reliably on other commands as well.

If we think that only the update command would benefit from machine-readable output, then there's less value in reworking how messages are printed in general throughout Bundler.

Answers to these questions will help inform the big decision of how to handle the outputting problem.

Here I will try to make some comments from "the other side" of the argument, that is specifically criticism in regards to reducing or removing functionality from long-term devs in the rubygems ecosystem.

Others could comment on this too, so I would like to ask to keep this issue here open for some time, a few months perhaps, for others to chime in.

In particular https://github.com/rubygems/rfcs/blob/master/text/0007-mfa-rollout.md specifies the phases, and phase 4 is the one that I find most troubling by far.

To avoid people having to jump away from here, let me copy/paste the phase 4 plan:

"Once the rollout of the top 100 most downloaded gems is successfully rolled out, a similar rollout will be done for more gems. The details of this phase will be fleshed out in another RFC."

So, to word this from the other side, it means that long-term devs are now required for mandatory MFA/2FA. If they don't do this, they can not modify their gem anymore. I consider this equivalent to an account attack by the rubygems team, because you deprive of devs the ability to modify their own gems, e. g. if someone wrote the code, you now deny them the ability to modify it anymore - or, more accurately, to push any new updates to this to their own repository. Thus you attempt to force-lock them into MFA. This is unfair, because YOU are not the ones who maintain the code - you actually ADD to the burden of OTHERS who maintain the code. If you guys were to lock them out but at the same time offer to maintain the code that you just removed from them, then this would be less of an issue - it would merely be a hostile account take over. But it is not even that; it is just a lock out.

For those who can use MFA or have no problem with it, this is no issue. For those who, for whatever the reason, don't want to or can not, you actually forbid them from using rubygems.org further IF they want to publish code still. This was what shocked me the most. I recall having used rubyforge and I recall having used rubygems for many, many years before without these issues; suddenly some new accounts come up from nowhere (WHO ARE THESE GUYS?) and the policy is magically changed. Why the sudden policy change? Why such a ruthless policy change that steals gems from accounts of long-term ruby devs? And why is this tied to number of downloads? You actually penalise (!) those who have a certain number of downloads. This is really weird.

Even more interestingly, I can simply remove the old code on any of my own gems, and publish completely new code, under the same name. Is that then still considered the very same gem that has to be MFA protected? I mean, what is the logic behind this, then? Evidently the assumption is that "if the gem name is the same, the underlying code must be the same", but this depends COMPLETELY on the dev who controls the gem. So the assumption by the rubygems.org folks is really weird here. If the code is totally different, it is a different project too, yes? Actually it would be the SAME problem you would potentially WANT to solve by "avoiding account take-over" - but how can you ensure that the dev who maintains the code, isn't doing this exactly? Even without malicious intent? You can not ensure this either. You are basing it on an ASSUMPTION that this is never done by anyone malicious or due to other reasons. What about devs who have not been malicious? What about devs who have not had an account stolen? Why are these suddenly punished and penalised?

If you look at this:

https://github.com/rubygems/rfcs/blob/master/text/0007-mfa-rollout.md

You actually see the attempted strategy revealed:

"The positive of having MFA for all accounts is that we’re setting a higher bar of security if users want to join/stay on the platform."

(Which, by the way, is an ASSUMPTION you do, as my example of exchanging the code shows. YOU CAN NOT ENSURE THIS. YOU ASSUME IT IS THE CASE.)

Meaning this is an invasive policy change. Aka: "if users want to stay on the platform". In other words it is admitted that if they
don't comply, they are being removed from rubygems.org and factually perma-banned as-is (since they can not modify the gem anymore). Now, before mandaty MFC comes I'll remove my rubygems.org account anyway, so the outcome is the same - but I feel as if this is a very unjust treatment coming from rubygems.org suddenly against all ruby devs. Why that sudden hostile policy? Why does it so conveniently and suspiciously overlap with github? They plan the same for 2023. That they need +1 year kind of shows how much opposition they expect. Even Google gmail still does not require mandatory M2A on every account. I can still access gmail fine (I stopped using it for other reasons, though - I am heavily against this global mandatory tracking trend or cohort sniffing via FLOC).

Unfortunately the ruby tracker already went that way and perma-banned me:

https://bugs.ruby-lang.org/issues/18800

Daniel pointed out this issue. Hiroshi said that this was done for another reason, e. g. spam. This is more understandable, but still annoying and unfair. However had, being perma-banned from the ruby bugtracker is still different to being perma-banned on rubygems.org. Even on github, by the way, I understood that you lose only the ability to PUBLISH code in 2023, so you could still use an account for issue requests and what not. But it's really weird how there is a strange, very suspicious move to "you can only write working code if you can be MFA identified". This is weird. At which point did computer systems become dependent on MFA?

Anyway. My hope is that those who are currently hijacking the ruby ecosystem reconsider their wicked plans. There are a MILLION ideas how to work around this to some extent, e. g. encourage people to use MFA without banning them, make "gem" itself and bundler work in that way (e. g. include only projects in your download-chain that are MFA guaranteed), use visual cues, use notifications and what not when accounts change (or code changes), add simple and easy reviews, add badges to encourage reviews of code, and so on and so forth. All of this could be done or intensified before the ""if users want to stay on the platform they now must use mandatory MFA". And while we are on the subject: add a disclaimer to rubygems.org that MFA becomes mandatory AT REGISTRATION, rather than ad-hoc change policies in the middle of nowhere.

My hope is that eventually these MFA proponents stop being so tyrannical and stop crippling the ecosystem from their selfish point of view. If a user wants to have only MFA gems, then you can solve this via the command itself:

gem install foobar --mfa # or any such switch

or via bundler + Gemfile to specify this too. Problem solved.

When I creating a new gem with Bundler, the autogenerated gemspec is not valid. It is generally expected and correct, however it prevents gem from being loaded in development.

Actual behaviour

Given freshly created gem named "store":

$ bundle gem store --test=rspec

When I run specs with:

$ bundle exec rspec

Then it crashes without running any test, printing following message:

Gem::InvalidSpecificationException: The gemspec at <path>/store/store.gemspec is not valid. Please fix this gemspec.
The validation error was 'metadata['homepage_uri'] has invalid link: "TODO: Put your gem's website or public repo URL here."'

Also, when I load IRB console:

$ bin/console

It crashes with following output:

Gem::InvalidSpecificationException: The gemspec at <path>/store/store.gemspec is not valid. Please fix this gemspec.
The validation error was 'metadata['homepage_uri'] has invalid link: "TODO: Put your gem's website or public repo URL here."'

This issue is not limited to homepage_uri metadata. More values are invalid.

Expected behaviour

No matter if all attributes in gemspec have valid values or not, bin/console launches IRB session, and bundle exec rspec runs tests. Moreover, yet another test is added to the autogenerated test suite, which succeeds if and only if gemspec is all valid.

Rationale

Gem validation is a good thing in general. However, in the first phases of development, I may be unwilling to fill in all the details. Possible reasons include:

• I haven't invented a catchy name yet, I use some temporary one which I am going to change later. Hence, some metadata (e.g. home page URL) is unknown at the moment.
• Exact license hasn't been decided yet. Hence, I prefer "TODO change me" rather than an empty entry.
• I am experimenting, I am not sure of exact responsibilities of this gem yet, even some fundamental decissions may be pending. Hence, I prefer to add gem summary and description later, when I exactly know what to write.

Currently, Bundler requiers me to provide all the metadata in advance. But in fact, it actually encourages to either remove invalid entries, or to replace placeholders with gibberish which passes validations, "just for now". In my opinion, both are bad practices, which shouldn't be promoted.

Note that in Ruby Gems these validations can be overriden, as gem executable allows for building from invalid (but syntatically correct) gemspecs with --force option. Perhaps a similar option should be added to the Bundler.require and Bundler.setup methods, e.g.:

Bundler.require :default, :development, validate: false

The exact synopsis is to be defined though, I'm not proficient with Bundler's internals.

Introduction

Lockfiles provide a snapshot of the exact versions of dependencies used to build a gem. Lockfiles allow developers to build a gem using the same dependencies as the maintainer used.

Some package managers use the lockfile to provide security assurances that the dependencies have not been tampered. For example, in addition to Python's Pipfile to specify dependencies, pipenv will generate a Pipfile.lock which includes SHA256 hashes of all dependencies. NPM supports dependency integrity using package-lock.json. Comparing dependency hashes on installation securely guarantees that a package will be built using the exact same dependencies as the maintainer specified.

Lockfiles with hashes mitigate various attacks:

• An attacker in a compromised network delivers malicious dependencies
• A repository of packages is compromised and packages are replaced with malicious instances
• Local packages on a machine are replaced with malicious instances

With hashes in lockfiles, when bundle install tries to install a dependency for a gem, a malicious dependency will not match the hash of the dependency in the lockfile, and the install will not succeed.

Proposal

I propose that Gemfile.lock include the SHA256 hash of each dependency. SHA256 hashes will be included after each versioned dependency. An example Gemfile and Gemfile.lock (Note the example hashes are random):"

Gemfile:

# frozen_string_literal: true
source "https://rubygems.org"

gem "nokogiri", ">= 1.4.2"

Gemfile.lock:

GEM
  remote: https://rubygems.org/
  specs:
    mini_portile2 (2.1.0) (d1bc8d3ba4afc7e109612cb73acbdddac052c93025aa1f82942edabb7deb82a1)
    nokogiri (1.6.8) (dc460da4ad72c482231e28e688e01f2778a88ce31a08826899d54ef7183998b5)
      mini_portile2 (~> 2.1.0)
      pkg-config (~> 1.1.7)
    pkg-config (1.1.7) (8b97bca79750847558d488e2ea4de79903c9c71c9af27ecf1b3dff5dba2abdd9)

PLATFORMS
  ruby

DEPENDENCIES
  nokogiri (>= 1.4.2)

BUNDLED WITH
   1.12.5

Implementation details

Hashes will be parsed in the lockfile parser. Hashes should be compared to downloaded dependencies before installation, such as when checking specs compatibility. I'm not familiar with the codebase, so please let me know if there's a simpler way to verify hashes.

For backwards compatibility, hashes will need to be optional initially, as previously generated lockfiles will not contain them. When hashes are present, they should be enforced. Hash enforcement should be all-or-nothing - All dependencies must include hashes, since any dependency missing a hash becomes a target for compromise. In addition, a flag can be added to bundle, such as --hashes, to require that hashes are present in the lockfile. In a future version, Bundler could mandate that hashes are present in the lockfile, but this would require maintainers to release gems with updated lockfiles or users would need to use an older version of Bundler.

Next steps

After a discussion on this issue, I'd like to propose this as an RFC.

Problem

Occasionally, when maintainers publish new gem versions, they make mistakes.

Examples:

1. rails Ruby version constraint mistake that broke 5.2.4.3-5.2.4.5 on Ruby 2.2. See https://github.com/rails/rails/blob/v5.2.4.3/activesupport/lib/active_support/cache/redis_cache_store.rb#L323
   It has been fixed in May 2020, but only released nearly a year later in Rails 5.2.4.6 (May 2021).

2. rspec-rails Ruby version constraint mistake that broke rspec-rails on Ruby 2.2.

3. diff-lcs issue with older Ruby versions

4. cucumber 4.0.0 broke compatibility due to diff-lcs dependency, fixed in 4.0.1 by pinning diff-lcs version to ~> 1.3.

Suggestion: Soft-yank

What soft-yanking means?

Gem maintainer scenario

The maintainer can soft-yank a gem version, just like they can yank it:

gem soft-yank GEM -v VERSION [-p PLATFORM] [--key KEY_NAME] [--host HOST]

Server/CI scenario

It remains possible to install the soft-yanked version of a gem with bundle install from Gemfile.lock.
Bundler emits a warning.

Developer scenario

Bundler excludes soft-yanked versions from dependency resolution.
bundle update/bundle lock show an error, just like for a yanked gem version or a removed gem.

Could things have gone better?

rspec-rails 4.0.0 could have been soft-yanked.
cucumber 4.0.0 could have been soft-yanked.
diff-lcs 1.4.3 could have been soft-yanked.

I have no such certainty regarding Rails, since it took a year to release the fix.

Misc

Related: rubygems/rubygems#1506 (comment)

#26 is semi-related, a proposal to prevent the only cause I'm practically aware of, weak Ruby version constraint. There might be others, like adding extra runtime dependencies, but I have not seen this in the wild.

cc @halostatue @JonRowe @marcandre @mattwynne @aslakhellesoy.

Problem:

At the moment the ~/.bundle/config stores both credentials and global configuration as well.

This is not ideal for versioning purposes. It does makes sense to version the file because global configurations like BUNDLE_PATH are useful across multiple setups, but it's not safe to do it right now because credentials for accessing private gems (like BUNDLE_GEMS__CONTRIBSYS__COM for Sidekiq) would also be versioned, which is not desirable.

Proposal:

These credentials should go to their own ~/.bundle/credentials file which would greatly help to avoid them getting leaked from public repositories, since there would be no point to version this specifically file.

I'd gladly jump into this if the core team finds it worthy.

Let me know what you think.

Summary

This is request to allow bundler to be able to support optional dependencies with an opinionated syntax and behavior.

Use case

A creator of a gem might want to declare that their package has some additional functionality if the client wants to include an additional dependency. For example, a string formatter gem might want to optionally add string coloring functionality if the client is okay with including the rainbow gem.

Another use case is sorbet, a type-checker for ruby. More info on my specific use case here.

What we'd like is for a gem to include sorbet-runtime (method type annotations for sorbet). However, we do not want to burden the client with this gem if they do not want to install it. Instead, we'd like to fall back to a separate gem, which is a shim of this runtime.

Desired syntax

This is the desired syntax for this feature. Very open to other options here.

Take no action of the primary dependency is unavailable

spec.add_optional_dependency 'sorbet-runtime', '~> 0.5.5585'

Sub in a separate dependency if the primary dependency is unavailable

spec.add_optional_dependency 'sorbet-runtime', '~> 0.5.5585', else_add_dependency: 'sorbet-runtime-stub', '~> 0.14'

Desired behavior

Case 1: The user does not include the optional dependency

Behavior: If the gem uses else_add, it will include the given argument (sorbet-runtime-stub above) and reconcile the fall back against the client's other dependencies, and include it in Gemfile.lock of the client (i.e. the fallback is required and behaves as if its any other required gem).

Case 2: The user does include the optional dependency

Behavior: The fallback is ignored, and the optional dependency is required, and the versions are reconciled against the client's version, and the standard reconciliation process applies (i.e. bundler errors if there are irreconcilable dependencies).

Optional additions

Allow the client of the gem to specify the optionality inline.

For example, the client might have:

gem 'my_special_gem', include_optional_dependencies: 'sorbet'

This would allow the client to explicitly tell the gem to use that dependency. This is essentially syntactical sugar over:

gem 'sorbet'
gem 'my_special_gem'

It makes it explicit that the dependency is specifically to satisfy the optionality of the sorbet dependency.

If the client uses include_optional_dependencies, it uses the version specified by the gem. If the client wants to specify the version of sorbet, they use the second approach (i.e. simply declare the two separate gems with the desired version numbers).

Allow the gem to declare a fallback require

spec.add_optional_dependency 'sorbet-runtime', '~> 0.5.5585', else_require: '../lib/sorbet-runtime-stub'

This would allow us to explicitly handle the case where the client did not include the optional dependency.

Other options

This behavior is today able to be emulated, but the emulation is fraught with concerns.
Here is my best attempt at emulating this behavior.

There are many issues with this:

1. The client's Gemfile.lock will not reflect that the optional dependency is actually a dependency of the gem.
2. Bundler will not reconcile versions at the time the user executes bundle install. Instead errors come only during runtime, which conflicts with the overarching philosophy of bundler of being able to reconcile dependencies statically in a separate process (i.e. before runtime).
3. This relies on possibly unstable API (specific errors raised by the gem command and the specific messages in those errors).
4. Depending on the load order, it might be possible that the fallback overrides the optional dependency in an unexpected way.
5. In general, we are moving dependency management away from bundler and into the client. This can potentially have a lot of undesirable downstream effects. One simple example is if you are constructing a dependency tree of your application, it would not reflect these optional dependencies using this workaround.