[Katana] persistent state about dojo HOT 11 OPEN

I could try to take this!

from dojo.

i dont think readability of the dump file is necessary.

maybe something like bincode might be helpful?

from dojo.

Hello! Thanks @Eikix that gives a great start!

@tarrencev I have an idea that might work, I'll try it and get back here to post result if I can achieve it! 👍

from dojo.

Ok I'll investigate with the blockifier team for a bit and then try both approaches

from dojo.

@Eikix i think it is worth talking a look at how foundry's anvil does this. are they dumping to json?

edit: Hmm yeah looks like they serialized it to bytes which will be a much more efficient representation than json

https://github.com/foundry-rs/foundry/blob/master/anvil/src/eth/backend/mem/in_memory_db.rs

from dojo.

@Eikix i think it is worth talking a look at how foundry's anvil does this. are they dumping to json?

edit: Hmm yeah looks like they serialized it to bytes which will be a much more efficient representation than json

https://github.com/foundry-rs/foundry/blob/master/anvil/src/eth/backend/mem/in_memory_db.rs

It definitely sounds better to do it with bytes,
Does this deserve its own issue after we do the json dump as "first implementation"?
as in staggering difficulty?

I'm guessing in any case, the difficulty might lie in ContractClass as I understand it's not just a standard object, it's a bit more complex?

Mhm will explore how easy it is to do local kv db dump as first implementation

from dojo.

i think its worth mentioning that katana is loosely based of the python devnet, so maybe also worth looking into how they handle this feature https://github.com/0xSpaceShard/starknet-devnet/blob/master/starknet_devnet/dump.py

from dojo.

@Eikix i think it is worth talking a look at how foundry's anvil does this. are they dumping to json?
edit: Hmm yeah looks like they serialized it to bytes which will be a much more efficient representation than json
https://github.com/foundry-rs/foundry/blob/master/anvil/src/eth/backend/mem/in_memory_db.rs

It definitely sounds better to do it with bytes, Does this deserve its own issue after we do the json dump as "first implementation"? as in staggering difficulty?

I'm guessing in any case, the difficulty might lie in ContractClass as I understand it's not just a standard object, it's a bit more complex?

Mhm will explore how easy it is to do local kv db dump as first implementation

I think that makes sense, if the json dump is simple lets land that first

from dojo.

I guess pickle in Python is super powerful, as it ser/de objects into bytes, which makes it very portable,
We'd be the only ones doing JSON dumps, which is kinda bad;/

The only upside I see is readability, and chances are dumps aren't too big anyway, so i/o won't be too expensive, I may be very wrong.

If the dump includes every tx since inception and the genesis state, it'll be way too big to kinda "run" a projection, but if it's a snapshot object, whereby you only have the latest state and not a huge list of tx, should be ok.

from dojo.

Hi all,
An update on my PR

I've managed to do go as far as:

• Run Katana and dump some state in a local file named by user, every dump_interval seconds (defaults to 10s) with the following command:
  cargo run --bin katana -- --dump .katana/dump.bin

This required me to implement Serialize and Deserialize inside katana-core quite a bit, as well as inside blockifier
Here is the PR to dojo's fork of the blockifier: dojoengine/blockifier#1

• ❌ Run Katana from a dumped state:
  cargo run --bin katana -- --state .katana/dump.bin

Problem encountered:
It seems I cannot load Katana from a previously dumped state due to this error:

thread 'main' panicked at 'Failed to load KatanaSequencer from dump: Bincode does not support the serde::Deserializer::deserialize_any method', crates/katana/src/main.rs:34:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I looked it up, and it seems some particular inner field of the StarknetWrapper struct in the project uses deserialize_any to deserialize its fields.

Tried options:

• install bincode 2.0 RC -> does not seem to work with Serialize and Deserialize traits out of the box (need to implement Encode and Decode on each struct)
• try with serde_json -> run into an error linked with struct keys needing to be strings:

thread 'tokio-runtime-worker' panicked at 'Failed to dump state: Custom { kind: Other, error: Error("key must be a string", line: 0, column: 0) }', crates/katana/src/main.rs:69:26
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

• investigate where the deserialize_any is, without success yet

My branch:
https://github.com/Eikix/dojo/tree/feat/persistent_state

PS: I still need to rebase on main

from dojo.

Hi all,
I can't manage to find a satisfying solution in the time that I have for this issue, I'm going to have to drop the issue and let someone else try:), thanks a lot for your patience and help so far

Tried solutions:

• serde_json as library to dump and load state -> a lot of Error("key must be a string") when trying to dump state
• bincode library (1.3.3) -> dump state is fine, cannot load state due to deserialize_any errors -> i suspect ContractClass's dynamic deserializing to be at fault

#[derive(Clone, Debug, Default, Eq, PartialEq, Deserialize, Serialize)]
pub struct ContractClassV1Inner {
    #[serde(deserialize_with = "deserialize_program")]
    pub program: Program,
    pub entry_points_by_type: HashMap<EntryPointType, Vec<EntryPointV1>>,
    pub hints: HashMap<String, Hint>,
}

With:

/// Converts the program type from SN API into a Cairo VM-compatible type.
pub fn deserialize_program<'de, D: Deserializer<'de>>(
    deserializer: D,
) -> Result<Program, D::Error> {
    let deprecated_program = DeprecatedProgram::deserialize(deserializer)?;
    sn_api_to_cairo_vm_program(deprecated_program)
        .map_err(|err| DeserializationError::custom(err.to_string()))
}

And DeprecatedProgram located in starknet-api

/// A program corresponding to a [ContractClass](`crate::deprecated_contract_class::ContractClass`).
#[derive(Debug, Clone, Default, Eq, PartialEq, Deserialize, Serialize)]
pub struct Program {
    #[serde(default)]
    pub attributes: serde_json::Value,
    pub builtins: serde_json::Value,
    #[serde(default)]
    pub compiler_version: serde_json::Value,
    pub data: serde_json::Value,
    pub debug_info: serde_json::Value,
    pub hints: serde_json::Value,
    pub identifiers: serde_json::Value,
    pub main_scope: serde_json::Value,
    pub prime: serde_json::Value,
    pub reference_manager: serde_json::Value,
}

from dojo.