Support replaying with less memory about cardano-ledger HOT 1 OPEN

I am wondering if there is a real reason to pursue this approach with UTxOHD being around the corner? I assume it would solve this issue of unnecessary memory overhead, right?

An open question is if this can be supported without major changes to ledger. My assumption is that it can.

Unfortunately support for this would require significant changes. It could theoretically be done, but it would add quite a bit of complexity to the already complex rules, because we would have to track in the ledger rules all the parts that would need to not be applied to the ledger state on a specific flag. The biggest reason why I would oppose this change is that, besides ignoring certain state modifications, we would also have to perform some validations conditionally on this flag, which all together sounds too dangerous to me.

I think I can suggest an alternative approach that might work for db-sync. Namely to employ some postprocessor that would strip out the unnecessary parts from the state after applying a block. The only required trick for this to work is that all ledger validation has to be turned off, i.e reapplyBlock is not sufficient, because it still performs some validations. So applyBlock has to be called with NoValidation, cause otherwise predicate failures like ValueNotConservedUTxO will be triggered. In other words it would work like that:

1. apply a block without validations to a stripped down state
2. look into transactions in the block and collect redundant data that they could have added to the state
3. remove that data from the state.

My suggestion would allow for ledger not to worry about this optimization that is not relevant for the chain safety and performance and it would allow for the team to stay sane working on those rules.

The ledger reapplication should work and reach a correct ledger state without relying on unecessary data. Also making it easier to create a ledger state could be used for debugging.

This use case is not accurate at all. We do need all data in order to reach the correct state. It is just that db-sync's view differs from the one that node and ledger have about what correct state actually means 🙂

There is plenty of data on chain that is not relevant for ledger, eg aux data, anchors, etc. Almost none of it is stored in the ledger state precisely because it is not relevant for ledger.

from cardano-ledger.