Apparently gitlab creates one event when you push multiple tags in one go.

That event won't have a ref referring to the created tag, but instead has the ref_count set to however many you pushed.

This means the private dub registry needs to recrawl that one project when it sees ref_count != null

Specifically the getPackages endpoint should not (in)-directly use it.

The event crawler only looks at tags, not imported projects, which are thus skipped.

This is an event for an imported project:

  {
    "id": 1,
    "project_id": 1,
    "action_name": "imported",
    "target_id": null,
    "target_iid": null,
    "target_type": null,
    "author_id": 1,
    "target_title": null,
    "created_at": "2021-02-23T04:25:36.531Z",
    "author": {
      "id": 307,
      "username": "asdf",
      "name": "asdf",
      "state": "active",
      "avatar_url": "<url>",
      "web_url": "<url>"
    },
    "author_username": "asdf"
  }

It is unsure if this event is generated when the project is imported, or when it is finish importing. E.g. can we already look at the tags and pull dub files?

Also, this event doesn't show up in the event api when action=pushed, but only when action=created.

Besides cleaning up stuff, otherwise we can never reuse the package name

Otherwise it messes up the index

Otherwise we can shadow public packages on the private registry.

Not really though. The name needs to be unique across all registries.

Dub will download packages to its global packages dir, and if a package conflicts with another from code.dlang.org, it will bring chaos.

Every package needs to be prefixes else it cannot be safely downloaded by dub (for fear of global conflicts).

or less at least

The current synchronization mechanism uses the gitlab event api to fetch events since a particular date. The events are filtered for new tags which update the registry.

The after date is set to yesterday, but close after midnight that can mean just a couple of minutes. That way it might miss tag just before midnight.

The dub-packages-index trails 15min behind and it uses an awful sync mechanism. Also it runs on github actions.

Right now reading has no locks, which might be an issue since we add items to the AA from another thread.

useful for loadbalancers to signal the backend is synced up

Sometimes it isn't master.

https://docs.gitlab.com/ee/api/projects.html#get-single-project

gitlab doesn't like paths like ./fx/dub.sdl. Instead they need the ./` stripped.

Allow the initial sync to happen against a mirror of the same private dub registry.

This is useful in immutable-infra environments where a new version is rolled out. If the registry can sync up with the currently running one (over https) it can speed up the sync (it takes about 8-10min on a 900 project gitlab instance).

Need to supply token to the sync endpoint (which we validate against the gitlab version endpoint), then dump everything over via multiform.

Instead we should skip and ignore the version.

It seems the version contains the commitId and the commitId is empty.

Registries:

1. https://code.dlang.org/api/packages/infos?packages=%5B%22symmetry-imap%22%5D&include_dependencies=true&minimize=true
2. https://dub.symmetryinvestments.dev/api/packages/infos?packages=%5B%22symmetry-imap%22%5D&include_dependencies=true&minimize=true

The package in question is symmetry-imap. In the first registry, all configurations specify sil-lang as an optional dependency. In the second, this is not the case.

And have failing Senders not kill the server.

For now we will assume nice citizens and for people to reintroduce the tag on another commit.

yet code.dlang.org gives latest beta