A lot of unneccessary strings are created when handling the errors, using the pattern

e.unwrap(anyhow!("{}", "x failed")

This will always evaluate the anyhow, even when not used. This is a lint in clippy, which also picks up some other "errors"

The server on dars.met.no sometimes says

We are currently serving 0 datasets.

while it actually has 1 dataset. This message usually appears after reloading the page a few times.

We should introduce a CI/CD workflow which build and deploys this image.

This can either be done under https://hub.docker.com/r/gauteh/dars or on a mirrored copy to gitlab.met.no which can deploy to the internal registry.

https://github.com/tokio-rs/console

Does also have grafana plugin, but requires third-party grafana extension.

This is a tracking issue for thread-safe and concurrent reads from netCDF files. Either using a pool of file-handles, or even better concurrent read support.

older netCDF formats depend on the netCDF library, while newer depend on HDF5. the following issue will presumably be the best reference for this issue:

• georust/netcdf#43

hyrax has their own HDF5 implementation, which might be useful to us (maybe through rust-netcdf):

• OPENDAP/hdf5_handler#13

I am not sure if it is permitted to include these fonts without a privacy policy statement or similar. Probably it is okay for the source code but not for running a public server (e.g. an unmodified docker container). I would suggest to drop these lines. I do not expect any readability problems when using a different font or simpler CSS.

When is a good time ..?

In gauteh/hidefix#8 a couple of different DBs have been benchmarked. The deserialization of the full index of a large file (4gb) takes about 8 us (on my laptop), its about 8 mb, and takes about 100-150 ns to read from memory-mapped type local databases (sled, heed). Reading it (8 mb binary) from redis, sqlite or similar takes about 3 to 6 ms which is maybe a bit too high. It would be interesting to also try postgres.

1. We need to keep data-discovery and dataset removal/update in mind:

• I think datasets should be registered, not auto-discovered by the data-server: the registration could be run by another dedicated service that auto-detects/scrapes sources.
• When a data-file turns out to be missing, or mtime has changed, we return an error, possibly notifying the scraper-service.

2. I think that we have to assume internal network-latency is OK, I don't see how we can do much about that, except keeping communication at a minimum.

A solution could be:

• Keep a central db with the index, DAS, DDS and list of datasets. This could be an SQL server or whatever, it is only written to by the scraper.
• Each worker has a local cache of datasets (index, DAS, DDS) (e.g. heed, or maybe even just in-memory): to avoid having to verify that a dataset still exists it checks the mtime of the source on request. If the mtime is changed: Update cache from server. In the case of NCML-aggregates this will not be discovered.
• When the central DB is changed, cache clearing is triggered at the workers. Retrieving new data from the central server is pretty cheap. This will handle NCML-changes.
• This will make it possible to extend to cloud data-sources since the central-db then would point to e.g. an s3 URL.

Unfortunately this complicates things significantly, but I don't see how to avoid it when scaling up. It would be nice to still support a stand-alone server that does not need a central db, but just caches locally and discovers datasets itself in some way. That would make it significantly easier to test the server out.

Some reasons:

• Storing the full index of all datasets on every worker takes a lot of space and needs to be kept in sync
• Network disk of index is slow? Embedded databases like SQLite still too slow, so then need a memory mapped DB anyway
• Indexing on-demand too slow, especially for aggregated datasets.

Since data is usually on network disks, caching data could possibly be done using large file system cache or maybe something like https://docs.rs/freqfs/latest/freqfs/index.html.

@magnusuMET

There might be a memory leak related to the number of requests made. After testing rust-netcdf, and not being able to provoke the issue there, I believe the leak could be this bug: hyperium/hyper#1790 (comment)