For the following list of actions - we should make them idempotent otherwise we end up with ever appending lists:

• resourceProviderAddTrustedMediator - if the mediator is already trusted don't call this
• jobCreatorAddTrustedMediator - ^ same
• postResOffer - if there is aready a resource offer that has not been matched - don't add another

• point solver to internal node
• https://gist.github.com/frank-borkin/8141db320cc88c5b487a468adcee9392
• ALSO: application level logging events directly into GA

check verification mechanism and ensure it's working

hash based verification based on outputs should be how it works - we'll have to wire up the bacalhau result hasher to modicum

Currently - the solver works by keeping an in memory map of resource offers and job offers - as well as trusted mediators for both parties.

The probem is if the solver restarts then everything breaks because it relys on it's in-memory map to match jobs.

Let's add some persistance to the solver service so it can survive restarts otherwise guaranteed this will break the testnet at the worst possible time :-)

useful to be able to reuse cids between runs (especially with more interesting modules that use outputs from one step as inputs to another e.g. LoRA)

as seen below, when there's contention on the smart contract because of the short block times we are more likely to see this, we just need to retry in the client

2023-07-17 12:19:02,606;ResourceProvider;🟠 post resource offer
Exception in thread Thread-1 (platformListener):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/app/modicum/ResourceProvider.py", line 303, in platformListener
    self.postDefaultOffer()
  File "/app/modicum/ResourceProvider.py", line 90, in postDefaultOffer
    self.postOffer({"request": "post",
  File "/app/modicum/ResourceProvider.py", line 78, in postOffer
    ).transact({
  File "/usr/local/lib/python3.10/dist-packages/web3/contract/contract.py", line 479, in transact
    return transact_with_contract_function(
  File "/usr/local/lib/python3.10/dist-packages/web3/contract/utils.py", line 172, in transact_with_contract_function
    txn_hash = w3.eth.send_transaction(transact_transaction)
  File "/usr/local/lib/python3.10/dist-packages/web3/eth/eth.py", line 362, in send_transaction
    return self._send_transaction(transaction)
  File "/usr/local/lib/python3.10/dist-packages/web3/module.py", line 68, in caller
    result = w3.manager.request_blocking(
  File "/usr/local/lib/python3.10/dist-packages/web3/manager.py", line 232, in request_blocking
    return self.formatted_response(
  File "/usr/local/lib/python3.10/dist-packages/web3/manager.py", line 205, in formatted_response
    raise ValueError(response["error"])
ValueError: {'code': -32000, 'message': 'nonce too low: next nonce 120, tx nonce 119'}

as well as our geth testnet, we should add an ipc one so users can spend tFIL

When running in dev mode (no DATA_DIRECTORY) on a new checkout, if you get

Traceback (most recent call last):
  File "/usr/local/bin/modicum", line 33, in <module>
    sys.exit(load_entry_point('Modicum', 'console_scripts', 'modicum')())
  File "/usr/local/bin/modicum", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 957, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/lib/python3.10/importlib/metadata/__init__.py", line 548, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for Modicum

then running pip install -e . inside /app inside one of the containers will fix it (permanently, even though the container restarts -- the change that this causes which fixes it, whatever it is, must persist inside the bind-mounted directory)

as soon as we have random users connecting nodes, the solver will have to get a bit smarter about only routing jobs to nodes that can run them

gpu is just one parameter of course - architecture and mem/cpu requests are also a factor

you'd have to have the client check the object versions and ideally sha256 hashes of all objects in a bucket... and then ship that object version manifest along with the input data, and verify it

see https://docs.bacalhau.org/data-ingestion/s3/

important for verification of filecoin data prep

can we document how a user can call e.g. the cowsay module from a contract on the same chain as the modicum contract?

2023-07-19 05:36:10,449;ResourceProvider;🔵🔵🔵 RUN JOB NOW
2023-07-19 05:47:40,621;ResourceProvider;🟢🟢🟢 Posting result: 
{"matchID": 0, "joid": 1, "contractStatus": 0, "resultHash": "QmTocYmkghKwXf4UPEpt4HRS1G8saACfCqWdEZNgJ1GKaA"}
Exception in thread Thread-1 (platformListener):
Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/app/modicum/ResourceProvider.py", line 197, in platformListener
    events = self.ethclient.poll_events()
  File "/app/modicum/EthereumClient.py", line 272, in poll_events
    log = self.filter.get_new_entries()
  File "/usr/local/lib/python3.10/dist-packages/web3/_utils/filters.py", line 181, in get_new_entries
    self.eth_module.get_filter_changes(self.filter_id)
  File "/usr/local/lib/python3.10/dist-packages/web3/module.py", line 68, in caller
    result = w3.manager.request_blocking(
  File "/usr/local/lib/python3.10/dist-packages/web3/manager.py", line 232, in request_blocking
    return self.formatted_response(
  File "/usr/local/lib/python3.10/dist-packages/web3/manager.py", line 205, in formatted_response
    raise ValueError(response["error"])
ValueError: {'code': -32000, 'message': 'filter not found'}

we might miss some events.. but probably better than doing this

currently we're pushing different architectures to different tags, we should instead use docker manifest to combine them proper like

https://stackoverflow.com/questions/66337210/is-it-possible-to-push-docker-images-for-different-architectures-separately

apply all the tips in https://huggingface.co/docs/diffusers/using-diffusers/reproducibility

ipfs is struggling to load, maybe changing this will make it better

which makes my laptop warm, and i don't like it

(this is when running

./stack lilypad-modicum-process runAsMediator

)

one day it would also be nice to have docs for how to add new modules, a bit like

• YOU MUST MAKE THEM DETERMINISTIC - see this for AI/ML, otherwise, just make sure you always return the same output with the same input, don't log timestamps in the output, read from /dev/random, be careful with concurrency, map iteration order in go etc
• get it working in bacalhau first, for example bacalhau docker run --gpu 1 ghcr.io/bacalhau-project/examples/stable-diffusion-gpu:0.0.1 -- python main.py --o ./outputs --p "cod swimming through data"
• then run the above command with --dry-run, then port the resulting yaml to a new module file in https://github.com/bacalhau-project/lilypad/tree/main/src/python/modules and add them to https://github.com/bacalhau-project/lilypad/blob/main/src/python/modicum/Modules.py

might as well

make chain persistent... then do analytics on the chain

so that when you put that into metamask it's correct

We made this change in order to fix a bug with the solver but it now means that the mediator address will always be the same as the job creator address - which will change when the user supplies their own private key.

We need to deloy the mediator against a fixed account and then both the resource provider and job creator should reference that static mediator address

Reference modicum in the readme and give them proper credit and mention gplv3

move lilypadnetwork.org dns (name servers) over to cloudflare

since we're going with simple fixed pricing for now, we should enforce the prices in the contract (currently, a user can send too little and they will still get the work done if other nodes have jobs in progress 🙀)

The purely onchain job creator currently accepts every job without verification, unlike the CLI which will verify 10% of jobs

Add verification to the onchain job creator contract. This will require a source of entropy/randomness

to set expectation

shouldn't be too hard, i know for sure it installs the x86_64 version of kubo though

currently if we fail a job for any reason, we also fail to repost our resource offer, so the network ends up stared of resource offers

check payment mechanism and ensure it's working

if we don't have cpu usage checking in bacalhau now, fix job price per module (e.g. stable diffusion inference = 10p, fine tuning = £1, cowsay = 1p)

at least for the client

for some reason, if you start the ResourceProvider, and then run

./stack lilypad-modicum-process runLilypadCLI --template cowsay --params "HAHAHACOWS"

twice, it only works the first time - the second time everything hangs

failing to find nvidia-smi means we don't find gpu nodes, need to install nvidia-smi in our container and set --gpu=all in the outer container

E_MISSING_EMOJI

luke@floating:~/pb/lilypad$ lilypad run --template cowsay:v0.0.1 --params "hello lilypad"

🌟 Lilypad submitting job cowsay:v0.0.1('hello lilypad') 🌟

🔗 Connecting to smart contract
🔑 Loaded private key for 0x2cebeea7E84E3746b991E2B7bf32D9587aCB64E8
🔌 Registering job creator
🧘 Adding mediator 0xcD9379EEDcBb9a077025Fbdf5AE81F1399e43F2C
💌 Posting offer for cowsay:v0.0.1
🥰 Running job... Sending deposit of 1 ETH to contract
ResultsPosted Fetching results... https://ipfs.io/ipfs/QmNjJUyFZpSg7HC9akujZ6KHWvJbCEytre3NRSMHzCA6NR
🍃 EtherTransferred Released deposit of 1 ETH to compute provider

🍂 Lilypad job completed 👉 Released deposit of 1 ETH to compute provider

luke@floating:~/pb/MODICUM$ ./stack lilypad-modicum-process runLilypadCLI --template cowsay --params "HAHAHA COWS"
I am about to invoke runLilypadCLI
Usage: modicum runLilypadCLI [OPTIONS]
Try 'modicum runLilypadCLI --help' for help.

Error: Got unexpected extra argument (COWS)

somehow the quotes get stripped off, is it something missing in how we're doing "$@" in stack?

so that results persist across dockerized node upgrades

Once the solver matches a job to a resource provider - it wipes the resource offer from the in memory state of the solver...

This means the resource provider will only do one job until we either:

• change this behaviour (e.g. keep hold of the resource offer but then how do we rate limit)
• re-register the resource offer once a job has completed

Ability to see transactions and blocks on the Lalechuza testnet would be great to have (perhaps ethernals? - it appears you can "turn your explorer public" here)

See thread https://bacalhauproject.slack.com/archives/C05FWMSJ3B6/p1689343663397499?thread_ts=1689343162.181219&cid=C05FWMSJ3B6

Also: Expedition - https://github.com/xops/expedition Blog: https://www.quicknode.com/guides/infrastructure/blockchain-data-tools/how-to-run-the-expedition-block-explorer-for-ethereum

then expose bacalhau get, logs, describe, dashboard etc via a lilypad wrapper that simply looks up the node address

also set --wait=false and then publish the job id immediately so that long running jobs can have a better UX

and time them out in case of ungraceful node shutdown/disconnection from chain

there's a lot of complexity in the smart contract around keeping track of user provided ids, we don't need to do this, so change the CLI so that it doesn't provide an id and then simplify the smart contract code

when private key is unset, and we're in production-client trim, error rather than using unlocked account zero

we should also move all the funds out of the unlocked account

I am a compute node - I start lilypad, it registers a resource offer.

The solver is now looping waiting for job offers to match.

I then stop my lilypad node so it's no longer waiting for match events to arrive to trigger jobs.

However - the solver still thinks I am there and ready to run jobs.

Do we need a removeResourceOffer workflow?

check if it works, if it does then deploy it

give the resource provider a way to write an output to a location which is (by CID) served over native HTTP, for efficient publishing for the filecoin data prep use case

we can't write back to S3 because we can't consume secrets in the public network

Presently - we use a single admin account for:

• deploying the contract
• running the mediator and solver

The user will supply their own keys (ergo addresses) for:

• resource provider
• job creator

This means the mediator will always have a different address than the resource provider and job creator.

This issue exists to replicate that same setup in local development - i.e. the mediator must use a different address than the RP and JC to replicate production.

given direct access to the bacalhau API, it should be possible to tail logs (with the bacalhau CLI) from a node

this is an optional service that compute nodes can offer for better DX and is not incentivized or guaranteed to be correct...

looking at the code, I wonder if changing

    wg.Add(3)

to

    wg.Add(1)

might make the code deterministic at the cost of performance?

https://github.com/anjor/go-fil-dataprep/blob/main/cmd/data-prep/fil-data-prep/fil-data-prep.go#L69

that should make it only do one thing at a time

currently if we fail a job for any reason, we also fail to repost our resource offer, so the network ends up starved of resource offers

e.g. lilETH = hypothetical $1

cowsay = $0.01
SD FT = $1
SD inference = $0.10
filecoin data prep up to 1GB = $0.50
arbitrary wasm for up to 10 minutes = $0.20