regression: encountered mutable pointer in final value when "outer scope" rule applies in const/static with interior mutability about rust HOT 29 CLOSED

A full revert will be hard since there's a bunch of follow-on work that landed since then that either further changes the interner, or requires the new interner.

I think it's possible to just turn that error ("encountered mutable pointer in final value") into a future-compat lint. Other than this bug, it can only be triggered by unsound flags/features (const_heap, miri-unleashed). Then we also get an idea of whether there is more code out there that relies on this pattern.

from rust.

We should still track this issue, it's just not a stable-to-beta regression any more.

from rust.

bisection seems to point to #119044 ? cc @RalfJung @oli-obk

searched nightlies: from nightly-2024-01-01 to nightly-2024-02-26
regressed nightly: nightly-2024-01-24
searched commit range: d5fd099...5d3d347
regressed commit: 6265a95

cargo bisect-rustc --start=2024-01-01 -- test 

from rust.

Oh strange, this landed a while ago and so far we got no reports I think...

Boa-0.13.0/src/builtins/map/mod.rs:341:9

That seems to be a deprecated crate, the repo has moved on to a new version. Not sure how to get source code for it.

The epo crate also regresses via boa

[INFO] [stdout] error: encountered mutable pointer in final value of constant
[INFO] [stdout]    --> /opt/rustwide/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/boa_engine-0.14.0/src/builtins/mod.rs:205:9
[INFO] [stdout]     |
[INFO] [stdout] 205 |         const UNDEFINED: &JsValue = &JsValue::Undefined;
[INFO] [stdout]     |    

That code I found here. (The current version of the crate on master doesn't have this any more, at least not in that file.) But so far my attempts at extracting an example that reproduces the issue failed. I assume it has to do with that JsObject type, that mentions some Gc things so it might be particularly tricky.

from rust.

I can reproduce the crash by compiling an old tarball of the boa crate (v0.13). But it also crashes stops compiling on rustc stable (1.76) for other reasons (glob imports), so unsure how helpful can that be.

I have the feeling that trying to get to a reproducer would be expensive in terms of time vs. benefit 🤔 But I am unsure if behind this crash compile error lies something useful for us to know

from rust.

"crash"? It doesn't crash for any of them I hope, just show this error?

I am certainly surprised that there is a regression at all, in particular given that an earlier version of that PR (#116745) has been cratered and didn't find anything. So in that sense it would be interesting to extract a testcase here.

from rust.

"crash"? It doesn't crash for any of them I hope, just show this error?

I stand corrected. Compile stops and exits with the error reported in this issues (sorry for mixing up apple and oranges).

from rust.

I copied more of the type definition into its own file, still no luck.

But I think the only way this can happen is if different parts of the compiler disagree on whether a type has interior mutability, which is quite concerning.

from rust.

Aah! I finally got it:

use std::cell::Cell;

pub enum JsValue {
    Undefined,
    Object(Cell<bool>),
}

impl ::std::ops::Drop for JsValue {
    fn drop(&mut self) {}
}

const UNDEFINED: &JsValue = &JsValue::Undefined;

fn main() {
}

The Drop is key. In boa it's generated by a derive which made this hard to spot...

So, I think what happens here is that we don't do promotion because of the Drop, but the "outer scope" rule still applies. The reference is created to JsValue::Undefined and some analysis concludes that this is fine since while the type has interior mutability, the value doesn't. But when we actually evaluate the initializer for UNDEFINED we just see a shared reference created to a type that has interior mutability so we mark it as potentially mutable, and then interning sees that and rejects it.

Honestly my preference would be to reject this in const-checking, IMO we shouldn't allow references to interior mutable types under the "outer scope" rule. Note that both Stacked Borrows and Tree Borrows actually currently allow writing through &JsValue::Undefined (or they would if we turned off promotion, which changes the code in a way that UB arises without even involving the aliasing model). In that sense both promotion and const-checking are really playing with fire when they accept enum variants of enums where other variants have interior mutability.

from rust.

So... actually allowing this code again will be quite tricky. Not sure if we want to do that.

What I would prefer to do is make this check entirely type-based, and thus give a nicer error for when this happens:

But that is quite the departure from our attempts to handle interior mutability in a value-based way, i.e. with these "qualifs".

EDIT: That's now implemented in #121786.

from rust.

So... what shall we do here? Retroactively ask T-lang to approve the part of #119044 that is a breaking change? That PR went through FCP but we didn't realize it would be a breaking change.

from rust.

So, I think what happens here is that we don't do promotion because of the Drop,

I'm assuming changing that, specifically for promotion in consts/statics is not gonna fly? We are already trying to remove/limit the method call exception that promotion has there

from rust.

You mean making the code work by having promotion kick in regardless? That sounds very tricky, we'd have to ensure this happens only when it's the final value of the const being promoted, not some intermediate value.

If we absolutely must accept this code then we can make const-eval do a type-driven traversal of the x in &x expressions to check whether there's any UnsafeCell anywhere. But that would introduce UB into code that I don't think should have UB (when x is not valid at the given type) and it could be terrible for performance...

from rust.

If we absolutely must accept this code then we can make const-eval do a type-driven traversal of the x in &x expressions to check whether there's any UnsafeCell anywhere. But that would introduce UB into code that I don't think should have UB (when x is not valid at the given type) and it could be terrible for performance...

Yea, i discarded this option right away and was looking for alternatives

from rust.

Yea, i discarded this option right away and was looking for alternatives

I think that option is still better than making promotion cover this.^^

from rust.

I guess we each hate each other's preferred solution, and neither of us love our own preferred solution. So...

@rust-lang/lang We did an accidental breaking change in #119044

The following example does not compile on beta anymore, but does compile on stable:

use std::cell::Cell;

pub enum JsValue {
    Undefined,
    Object(Cell<bool>),
}

impl ::std::ops::Drop for JsValue {
    fn drop(&mut self) {}
}

const UNDEFINED: &JsValue = &JsValue::Undefined;

with

error: encountered mutable pointer in final value of constant
  --> src/main.rs:12:1
   |
12 | const UNDEFINED: &JsValue = &JsValue::Undefined;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^

what happens here is that we don't do promotion because of the Drop, but the "outer scope" rule still applies. The reference is created to JsValue::Undefined and const checking concludes that this is fine since while the type has interior mutability, the value doesn't. But when we actually evaluate the initializer for UNDEFINED we just see a shared reference created to a type that has interior mutability so we mark it as potentially mutable, and then interning sees that and rejects it.

The reason this worked before was that before #119044 we did a value based (not just type based) interning traversal, so we picked up on the fact that the specific value has no interior mutability.

We have come up with two ways to get back the previous behaviour:

1. we can make const-eval do a type-driven traversal of the x in &x expressions to check whether there's any UnsafeCell anywhere.
2. special case promotion to trigger for these values, even though their type is Drop and !Freeze

The issues with those are respectively:

1. that would introduce UB into code that should not have UB (when x is not valid at the given type) and it would be (very) terrible for performance...
2. terrible extension to an already fragile part of the compiler: promotion.

So we would like to just keep rejecting this code, as it requires special cases to support instead of falling out of a set of simple rules.

from rust.

I guess we each hate each other's preferred solution, and neither of us love our own preferred solution. So...

Well, "hate" is a strong term.

I did think some more about this, and I think if we can make promotion entirely take over the "enclosing scope" rule then maybe it's worth it. That will simplify const-checking. However, I think this means duplicating and carefully aligning the "enclosing scope" logic inside promotion which has a huge risk of going wrong -- in particular as both of them keep evolving.

The reason this worked before was that before #119044 we did a value based (not just type based) interning traversal, so we picked up on the fact that the specific value has no interior mutability.

The point to emphasize is that "value has (no) interior mutability" is not necessarily a meaningful concept, as far as t-opsem is concerned. There has been a lot of discussion around exactly how far interior mutability "leaks" into its surroundings when a type contains some UnsafeCell and some data outside the UnsafeCell, but all the options that t-opsem was still seriously considering would all agree that there is some leakage, i.e., that at least some of the bytes in *UNDEFINED would actually be considered mutable no matter its value.

This is also tied up with rust-lang/unsafe-code-guidelines#493. There are two places where rustc does value-based reasoning for interior mutability: promotion and const-checking. This issue here is about const-checking. No matter what we do here, we'll have to figure out what to do with rust-lang/unsafe-code-guidelines#493 eventually... I don't know if we can ever get rid of value-based reasoning for interior mutability, but I also don't think we want to have an operational semantics that would permit value-based reasoning for interior mutability.

So we would like to just keep rejecting this code, as it requires special cases to support instead of falling out of a set of simple rules.

To add to that, that doesn't mean we consider the issue resolved, we want to keep tracking it somewhere. But right now we feel like the cure for this regression is worse than the disease, so if we can get away with this breaking change then that seems like the least bad option.

The regressions listed above all come from a single crate, boa. The problematic code has been removed from boa at least a few months ago; the latest released version builds fine with current nightly, modulo some unrelated issues.

from rust.

We discussed this today in triage without resolution. We're planning to take this question back up on the 20th.

from rust.

link to Wednesday's T-lang discussion: https://hackmd.io/U-CKiZx_RKiaANAPXtWf7g#regression-encountered-mutable-pointer-in-final-value-rust121610

from rust.

Discussed in the compiler team triage meeting. The 1.77 release ships on the 21st so there won't really be time for anything to happen after the T-lang meeting on the 20th so we're going to attempt a revert targeted at beta to give T-lang more time to reach a conclusion.

from rust.

I was wondering why promotion doesn't trigger this error; turns out the interner simply disables it for promoteds. I guess we have that case covered in our test suite (or in the standard library itself).

I think increasingly my tendency is to fix this by changing &x semantics such that under specific conditions it traverses x to determine whether there is actually an UnsafeCell there. Ideally those conditions are "only for borrows in the tail expression of a const/static that got elevated to module scope". I don't know how easy it is to determine this; one extreme option would be "only in const/static initializers" (but not in const fn bodies).

from rust.

This was downgraded in #122204 and beta-backported in #122524.

from rust.

This was downgraded in #122204 and beta-backported in #122524.

Tracking issue for that: #122153.

from rust.

This was discussed in a T-lang meeting.

There was general sympathy with the t-opsem desire to not do value-based reasoning for interior mutability. But there also wasn't a clear idea for how value-based reasoning could be removed from both promotion and tail-expression checking. The one thing everyone agreed on was to at least do a crater run to see how bad it'd be. Then based on that, maybe there could be an edition-based transition plan.

For this concrete issue, the goal is to monitor the tracking issue of the future compat lint (#122153) to see if anyone speaks up -- to see if there's notable code out there not covered by crater that would regress if we just made the current check a hard error. Depending on that we could either take the hit and break old versions of boa, or do value-based reasoning in const-eval when determining the mutability of shared references (at least in the "root frame" of const-eval, i.e. the initializer expression of the const itself).

from rust.

@rustbot labels -I-lang-nominated

Thanks to @RalfJung and @oli-obk for joining us in the meeting to talk through this. As @RalfJung mentioned, there are some next steps here, so please renominate when we have more data and a new proposal for how to move forward based on that.

from rust.

The crater experiment in #122789 which entirely removed value-based reasoning for interior mutability had almost 4000 regressions. I think we can conclude that this is not an option.^^

from rust.

I'm proposing we fix this by removing the lint and just accepting the code, unfortunate as that is. See #128543.

from rust.

Mutating a non-internally-mutable type is UB, but that doesn't necessarily mean that mutating an internally-mutable type is not UB.

Presumably there could just be a second rule that mutating read-only memory is also UB, even if the type being mutated is internally mutable.

As long as safe code is prevented from mutating these values, then the only danger is that of people not realising this is UB when writing unsafe code - this could be mitigated by a compiler lint, which does a "best effort" attempt at detecting writes to read-only memory.

from rust.

Presumably there could just be a second rule that mutating read-only memory is also UB, even if the type being mutated is internally mutable.

Yes, we already have such a rule. I was just hoping it would not be necessary...

Also see rust-lang/unsafe-code-guidelines#493, which has the same root cause but is worse because it is not at all clear to the user that there's any "read-only memory" here.

from rust.