innogames / serveradmin Goto Github PK

I wanted to give Serveradmin a try this weekend. The new interface looks neat. Good job!

I added a few attributes, servertypes, and objects. Object creating went smooth. Then I realized the "attr" command not working. Is it a known issue?

I looked at my JavaScript console, but haven't seen anything other than "Failed to load resource: the server responded with a status of 404 (Not Found) http://127.0.0.1:8000/static/js/popper.min.js.map".

This is Safari Version 14.0.2 (16610.3.7.1.9) on MacOS 11.1 here.

The Problem

Sometimes one object needs to contain a bunch of information on the same topic, we do this by attaching multiple attributes to an object. Example: One vm has backup_host and backup_window.

Sometimes one object needs to have multiple values on the same topic, for this we offer the multi flag on attributes. Example: One vm can relate to many loadbalancers

Sometimes we need both of these qualities in one attribute though.
For instance protocol_port on loadbalancer is currently a multi string attribute, containing values like tcp443t. This encodes protocol is tcp, port is 443 and the flags are ['sticky']. Serveradmin doesn't really offer a great way to check this, except one giant regexp on the attribute, or a pre commit signal aborting the commit. We also lose the data type safety.

Another example of the same problem is a hypothetical aaaa_record on the public_domain object type. Currently:
a) we could either make it a multi string attribute (like protocol_port) and encode the information de1.the-west.net has the IPv6 address 2a00:1f78:fffd:1::d448:bf3f with a TTL of 60 minutes by adding a line like de1;2a00:1f78:fffd:1::d448:bf3f;3600 to the list of aaaa_records on the public_domain the-west.net.
b) or we could make it a multi relation attribute (like loadbalancer) and create one DatasetObject per record. The aaaa_record object type could have the required attributes with proper data types. The problem here is: What's the name of this thing? DatasetObjects require a unique and meaningful name, preferably a FQDN. We would probably end up with something like de1.the-west.net_aaaa_1.

My proposed Solution

To solve this I propose a concept of Anonymous DatasetObjects:
We would add a new anonymous column to the Servertype database table. Checking this implies:

• Objects of this object type do not have a hostname
• Queries never return objects of an anonymous object type directly. They can only be queried by getting a named object and joining one of its attributes, which is of type relation to the anonymous object type
• Unreferenced anonymous objects are garbage collected
• Servershell needs to learn how to do at least one level of joining
• The api currently doesn't support committing joined attributes at all. That should probably be implemented first.

TODO:

• How are these objects actually created? Both API and UI wise. I suppose anonymous object types are not shown by servershells new command?

The Problem

Currently we use plain django signals. These have a few problems:

The signals are completely best effort. We don't note down which signal receiver actually reacted to the signal and if its action was successful. We don't retry anything either.

Further we block the commit from going through and the http request from returning before the signals are finished. The rational is, we want people to be sure their DNS change is already live before returning their commit as successful. The downside is that we block the django worker for seconds at a time to execute the flush by connecting to places via SSH (our preferred method of RPC for some reason) and that's the success case. In the failure case a hook takes a really long time. We may have to wait until the request timed out at 60 seconds. If we have enough requests in the queue making such a change, all workers will become busy. Subsequent requests will queue up until we reach net.core.somaxconn. Once that happens the health check endpoint will start failing, resulting in this web server getting kicked out of the loadbalancer.

My proposed Solution

To solve this, we should move the signal handler out of the serveradmin django. Signals should have no external dependencies and only check data integrity. Serveradmin already keeps track of commits. When making a commit via the API we should return a handle to this commit. If a client expects a certain hook to be performed, the commit object should offer an API to wait for the hook to be finished.

Further we need to extend the API to allow retrieving commits that have happened. Using this API an external DNS flusher, reusing the code currently living in serveradmin, could notify when it started and finished working on a hook for a commit and even report errors back. This hook information should then be transmitted back to the client that made the commit and is waiting for this hook. Further the hook information could be persisted in the database to have a record of which hooks were performed.

Another upside to this design is, it allows for very long running hooks. Even building a VM would be a perfectly plausible hook to implement this way.

TODO:

• I'm deliberately vague about the API here, especially on the side working an the hook. The simplest implementation would offer all the commits that are made on serveradmin, but we could also be smarter. Emre already proposed[0] extending the adminapi Query API to subscribe to changes made on serveradmin after the initial retrieval of a Query. That proposal would complement this proposal well.
• Django signals only happen within one web server. We have multiple web servers though. If somebody commits a change on web server A, but I'm subscribed via web server B we wouldn't know about it. Hence we either need to constantly poll B to requery the database or we need to inform all web servers about a commit that just happened. I suggest looking into postgresqls NOTIFY/LISTEN features for this. There's even already a django plugin for it.

[0] https://github.com/innogames/serveradmin/pull/52