skip query db if the cache size is not reached

docker-compose requires all environment variables to be set, even when building the application, which is unnecessary. We should move the environment variable check to the be done in the application itself, rather than in docker-compose

A couple of significant requirements:

• Some tests are written with data sources (such as an zone file accessed via HTTP) in place, which must either (1) be mocked or (2) these tests must by skipped by GA.
• Some tests require interaction with a postgres database in place, which should be provided by GA.

• fix posthook error
• new domain error

There exists a "zonediffer" and a "zones" collector and one of them is supported whereas the other is not.

I think there is a problem by retrieving the last stored certificate from the DB. I guess this function dosen't work very well (https://github.com/aau-network-security/gollector/blob/master/app/ct/main.go#L135).

The way ct should work: it should insert in the DB (n) certificates every time is run. the function linked above should get the last entered certificate from the DB in order to scan the next (n) certificates from that one.

The way ct is working right now: the first time i run ct it enters the first 100 certificates. Running it again it dosen't store the next 100 certificates in the DB.

The function linked above give back an error in the tests too
https://github.com/aau-network-security/gollector/pull/39/checks#step:6:111

With some high-performant IPC communication between a process and a cache container (high throughput => 100,000 messages per second, low latency => (far) under ms), it would be possible to run multiple measurements simultaneously.

Current situation

+------------------+           +------------------+
|                  |           |                  |
|     Process 1    |           |     Process 2    |
|                  |           |                  |
|  +------------+  |           |  +------------+  |
|  |            |  |           |  |            |  |
|  |   Cache    |  |           |  |    Cache   |  |
|  |            |  |           |  |            |  |
+--+------+-----+--+           +--+-----+------+--+
          |                             |
          |                             |
          |                             |
          |                             |
          |                             |
          +--------------+--------------+
                         |
                 +-------+--------+
                 |                |
                 |   Persistent   |
                 |                |
                 +----------------+

Suggested alternative

+------------------+           +------------------+
|                  |           |                  |
|     Process 1    |           |     Process 2    |
|                  |           |                  |
+-------+----------+           +---------+--------+
        |                                |
        +--------------+    +------------+
                       |    |    High performant IPC
                       |    |
                 +-----+----+-----+
                 |                |
                 |     Cache      |
                 |                |
                 +-------+--------+
                         |
                         |  Asynchronous, but reliable
                 +-------+--------+
                 |                |
                 |   Persistent   |
                 |                |
                 +----------------+

Use just a library to interact with the DB. pq should be the best on

I am not convinced the current implementation is correct. The only zone file accessible over FTP is the .com one

We should probably consider using an LRU cache, instead of caching everything.

Right now, I believe gollector does not handle renewal of certificates by certmagic. Only relevant when the cache is running for a long time

Somehow Golang does not manage to read environment variables

• delete previous implementation not used
• delete benchmark code

Error messages are somewhat difficult to debug otherwise

• Wrapping errors to make error prints more clear
• Report messages to Sentry/Rollbar and send emails when errors are occuring

When running the cache over TLS, the certmagic lib automatically obtains certificates. The issued certificates are stored on disk, but because we currently do not mount a volume to persist those certificates, they disappear whenever the container is closed, and with a restart a new cert is issued. As a result, the rate limit of Let's Encrypt may be hit, locking us out of running on TLS for a few days.

The docker-compose config should mount a volume to the correct location where the certs are stored by certmagic

Make sure the implementation about CT logs works for all the collectors component we have

Instead of relying on a precompiled list of TLDs, use the CZDS api to retrieve all TLDs to scan

We must be able to distinguish between multiple measurements from different vantage points, knowing exactly which data point belongs to which data set/vantage point.

It might be useful to retrieve the current state of the entries contained in cache process via an gRPC. To go even further, a monitoring tool can read this state and visualize the growth of number of entries over time.

This results in the failure to download some zone files, and wrong last_seen dates

As suggested from @mrturkmen06 i am reporting this problem.
Running tests on CI with this parameter run: go test -v --race ./... give back the following error :
https://github.com/aau-network-security/gollector/runs/689914296?check_suite_focus=true#step:6:68

add the item to the cache:

• when i find them in the db
• after i wrote them in the db (could be done in the posthook)