Flipside/Dune Integration about gitcoin-grants-data-portal HOT 10 OPEN

I just learned we can in fact select assets using groups, compute kind or other properties.

This means we can probably construct assset job definitions in such a way, that say CI run would only trigger default assets while weekly fetch would trigger everything including actually updating Dune or Flipside.

from dagster import AssetSelection
AssetSelection.groups('group_label')

from gitcoin-grants-data-portal.

Dune has an upload limit of 1MiB...

We can upload probably some daily metrics but not much more.

@DistributedDoge, do you know how the Flipside external data works?

Seems their live query stuff would work for JSON data but I'm not sure if it can read Parquet files

from gitcoin-grants-data-portal.

1. From what I see in docs+UI Dune is telling me upload limit is 200MB per .CSV table. See here, no clue if it is accurate though.

2. Last time I tried, Flipside could not read anything other than JSON, it will most certainly work with ipfs-hosted file though.

EDIT: For 2, to avoid making things messy we could e.g. push a subset of tables as json to different Filebase bucket.

from gitcoin-grants-data-portal.

Dune is telling me upload limit is 200MB per .CSV table

I think that might be correct. The limit comes from the Free plan, which limits storage to 1MB.

For 2, to avoid making things messy we could e.g. push a subset of tables as json to different Filebase bucket.

That could work! We'll need to check how efficient those queries are though. Querying uncompressed JSON is not very efficient.

from gitcoin-grants-data-portal.

Confirmed empirically that Dune does not let me upload 25MB file, bait and switch at its finest.

from gitcoin-grants-data-portal.

They're thinking about increasing it though.

I can poke them a bit more and check if that's a "future" plan or actual thing happening in the following weeks/months.

from gitcoin-grants-data-portal.

Left a comment with a potential implementation approach.

from gitcoin-grants-data-portal.

Sounds like the perfect approach!

from gitcoin-grants-data-portal.

For Flipside, a new option can be to push data to a spreadsheet and have Flipside read from it.

from gitcoin-grants-data-portal.

Seems like Dune live fetch is now open to all free users!

from gitcoin-grants-data-portal.