I'm on commit hash 79f91aa5eb606a1fa453db18e4fcd5dc45fe0d8b and, after running the agent for about an hour, I tried cancelling it and it wasn't working. You can see the ^C in the logs here:

I'm quite sure I'm up to date with the latest code, so I think this issue persists. I believe it was addressed before but perhaps there's a new issue when it's more long-running.

Requirements

• A new tokio task that gets the last seen block height, and attempts to index the blocks since now.
• Also have another task that will periodically fix gaps in the data (also have a time limit for finding when the gap occurred):

Example gap finder query:

select time, height + 1 as gap_start,
       next_nr - 1 as gap_end
from (
  select time, height,
         lead(height) over (order by height) as next_nr
  from block
) nr
where height + 1 <> next_nr;

2022-10-15 22:19:21.556055,339186,339192
2022-10-15 22:22:03.791004,339213,339221

Think!

There are two aspects to the CronCat indexer:

1. Rust indexer
2. Indexer ✌️ Sweeper ✌️

Rust indexer

The Rust indexer (the repo where this issue is created, obvs) is in charge of being fast, efficient, and able to handle the scenario of a world where we have hundreds, thousands, or more transactions per block.

There are a ton of smart contract queries needed to fully index what's happening with CronCat, but those don't need to be done in Rust. Instead, the Rust indexer will be ingesting information about new blocks coming in.

When a new block comes in — either detected via websockets or polling — there is a limited amount of information we get. We get the block height, timestamp, and transaction info in the block. The transaction info doesn't have everything we need, but it does have info on what messages are in the transaction.

For our purposes, we only care about blocks where end users/apps/protocols/agents are calling CronCat contracts. (The primary one being the CronCat Manager)

So we'll want to ingest a new block, filter to see if there are transactions with messages that are WasmExecute ones going to one of our contracts, and then capture the message info.

Once we've written (limited) rows to our database, the NodeJS Indexer Sweeper will use the handy tools from CosmJS in order to query the blockchain for account balances and contract state at that given height.

Talking brass tax, here's the only thing we want the Rust indexer to be writing, and nothing else:

Writing a row to the blocks table, and getting the id that's returned from that INSERT, but only if this block interacts with one of the CronCat contracts.

Writing row(s) to the transactions table using the id from the block a second ago, similarly returning the transaction tables' primary key, 'id.

Writing message(s) to the messages table, using the transaction id

Writing the contract address and chain ID prefix (like uni for uni-5) to the contracts table.

Note: we just added the smart contract's address but no other information about it. The sweeper will fill in the rest, making sure all the database tables have useful foreign keys to enable useful and deep querying later. It does this for the contracts table as described just now, but will also fill in the entities inside our smart contract storage.

The entities in our smart contract storage are:

1. Config
2. Agents
3. Tasks

Tasks also contains what's called multi-valued attributes. For instance, a task that has multiple rules will need to store those in another database table, and relate to its task. So there are a few entities inside of tasks, you could say, and they all get their own database table. It looks like this:

Roughly the top half of that chart are tables the Rust indexer will to insertions on. It will not touch any tables including or below agents, config, and tasks, as those will be filled in by the indexer sweeper CosmJS code. (There are many helpful libraries in CosmJS that make this part of the process easier to encode/decode messages, protobufs, oh my. The Rust indexer gets the initial block info written as efficient and scalable as possible.)

Regarding the database and the relationships, here's the schema that's fairly mature:

https://github.com/CronCats/indexer-sweep/blob/6b170d007f599ac2f30d2e2113f2e6036822946d/db/schema.sql

Indexer Sweeper

The indexer sweeper lives in this repository:

https://github.com/CronCats/indexer-sweep
Please see the README there for usage instructions.

It's a simple TypeScript project that uses CosmJS to query the blockchain for smart contract state and protocol-level values. Example: we keep track of each CronCat agent's protocol balances as well.

Aim and direction

At this time, there are a handful of useful queries we can use to start capturing important information. One of the first things we'll look for is:

Are all the fees working exactly as we expect.

We can test this out as follows:

1. Instantiate a new CronCat Manager on Juno testnet, one that only we care about.
2. Run the indexer(s) so they're "watching" the chain
3. Run an agent that (first registers and then) starts running, pointing at the our new contract.
4. Create a recurring task, maybe a bank payment message to one person.
5. Let the task run out naturally, either because the agent runs out of funds (this might be fixed now, not sure) or the task is removed because it has depleted the balance that was initially sent to it upon task creation.

Then we can analyze the results of the payments, and learn more about how much is going to gas, how we are adding up the rewards to the agent, and more.

I think we might need to be careful about using the ? operator for long-running tasks like the agent and indexer.

This is a pretty good rundown:
https://doc.rust-lang.org/book/ch09-02-recoverable-errors-with-result.html#a-shortcut-for-propagating-errors-the--operator

It's basically like, "something happened that means we gotta stop"

If the value is an Err, the Err will be returned from the whole function as if we had used the return keyword so the error value gets propagated to the calling code.

In the function below, for instance, here's an example:

This is in a while loop, and if a block variable comes in with an error, we're instructing the application to throw an error and exit.

I think we'll need to replace this with a match statement that checks for Ok or Err and on error, perhaps log it (optional) or just continue on with the while loop.

I see in that same function ws_block_stream there are a few more places where we have the pattern to throw an error and crash the program, but I think those also need to be dealt with in a way that doesn't stop execution.