Plan for this project about aegir HOT 16 CLOSED

I don't agree with trying to sync with our main guild. The test guild should be generated purely from the config. I believe this will be feasible if the config format/system is changed to accommodate for easy distinction of types for IDs (i.e. channel vs. category vs. webhook vs. role). I proposed some ideas before: using type annotations, a naming scheme (e.g. "role_" prefix), and nesting them under aptly-named keys.

Channel order doesn't matter. Role colours don't matter. Missing emotes don't really matter, but it may be useful to get an authentic preview of the front end for a feature. If it's important enough, we can look into always including the URL for emojis.

Keeping a guild in sync would be a matter of checking if anything is missing when the bot starts. It can then either automatically create missing things or show a log/message that says "run this command to sync things". What you propose sounds more like the user/role sync confirmation that was recently removed. That's fine too, though relatively more complicated to implement. I see no reason for a web interface.

Keep in mind a large portion of channels are only there for whitelist/blacklist purposes, so they aren't strictly necessary. In fact, many if not all features don't need separate channels; they are only separate for organisational purposes. We may be able to cut down on generated channels by re-using some channels for multiple config fields. I don't have any concrete ideas yet, but one that comes to mine is setting a flag somehow that indicates which channels should preferably be separate.

from aegir.

Why would we maintain two versions of the "required" state? We already keep a lists of configs for our main server that could serve as a model for a test server. It would mean that you don't reuse channels, but it saves us on having to maintain two constants states and it decouples the state of our test server with the model we use for other test servers. The state of our main guild should always be in sync with the HEAD of "master", while our test server could be the home of experiments for things yet to be merged.

If we were to have a config file that looked something like:

server:
   id: 12345566

   categories:
       information:
           id: 12344235

           channels:
                announcements: 230409432
                news: 349910832970842

and so on for the main guild, you'd have everything you need to set up a test server with the current state of "master" without actually "looking" at the main guild directly. You just generate everything based on the configs we necessarily maintain for required features anyway.

A side-effect is that you could maintain the same category/channel order, but that's by no means required. It does mean that the right channels would be in the right categories, which would make sense, as we're starting to use categories more to whitelist/blacklist things.

from aegir.

@MarkKoz

I don't agree with trying to sync with our main guild. The test guild should be generated purely from the config. I believe this will be feasible if the config format/system is changed to accommodate for easy distinction of types for IDs (i.e. channel vs. category vs. webhook vs. role). I proposed some ideas before: using type annotations, a naming scheme (e.g. "role_" prefix), and nesting them under aptly-named keys.

Yeah, this sounds like the best approach so far, by "syncing with the mean guild" I didn't mean to grab the status from Discord directly.

Channel order doesn't matter. Role colours don't matter.

It doesn't matter, but it is still useful to have. I mean, it is still better to feel "at home" in the dev server.

Missing emotes don't really matter, but it may be useful to get an authentic preview of the front end for a feature. If it's important enough, we can look into always including the URL for emojis.

It actually is. The most notable is probably the trash can emoji since it can create quite a lot of exceptions when playing around with the bot. Also, I think they have to be actual emojis because they are used inside embeds, but I might be wrong about that.

Keeping a guild in sync would be a matter of checking if anything is missing when the bot starts. It can then either automatically create missing things or show a log/message that says "run this command to sync things". What you propose sounds more like the user/role sync confirmation that was recently removed. That's fine too, though relatively more complicated to implement. I see no reason for a web interface.

So we would have this feature directly on the dev bot? The idea was to automate things as much as possible, and we need a (simple) web interface to get OAuth tokens, that was @jb3 idea, I quite like it personally.

Keep in mind a large portion of channels are only there for whitelist/blacklist purposes, so they aren't strictly necessary. In fact, many if not all features don't need separate channels; they are only separate for organisational purposes. We may be able to cut down on generated channels by re-using some channels for multiple config fields. I don't have any concrete ideas yet, but one that comes to mine is setting a flag somehow that indicates which channels should preferably be separate.

Sure, we could remove quite a few channels, but while we are at it, I don't see why we wouldn't have every channel.

@SebastiaanZ

Why would we maintain two versions of the "required" state? We already keep a lists of configs for our main server that could serve as a model for a test server. It would mean that you don't reuse channels, but it saves us on having to maintain two constants states and it decouples the state of our test server with the model we use for other test servers.

That was the idea, using what is already here.

The state of our main guild should always be in sync with the HEAD of "master", while our test server could be the home of experiments for things yet to be merged.

That is why I think we should be able to choose if we want to sync based on python-discord/master, or we can specify another remote/branch. Also, let's not forget that while this tool is intended to be used by us in our current workflow, It will mainly be used by external contributors that don't want to maintain a dev server themselves.

I quite like the config idea personally.

from aegir.

Why would we maintain two versions of the "required" state? We already keep a lists of configs for our main server that could serve as a model for a test server. It would mean that you don't reuse channels, but it saves us on having to maintain two constants states and it decouples the state of our test server with the model we use for other test servers.

Sure, we could remove quite a few channels, but while we are at it, I don't see why we wouldn't have every channel.

Because having channels that aren't strictly necessary just adds clutter and makes the test guild more cumbersome to use. As an example, consider the channels recently added for blacklisting duckpond. One only needs 1-2 channels to test if the blacklist works, and those channels can be existing channels used for other features.

So we would have this feature directly on the dev bot? The idea was to automate things as much as possible, and we need a (simple) web interface to get OAuth tokens, that was jb3 idea, I quite like it personally.

Seems like a lot of complexity just to save someone having to create a blank server and invite the bot. It could be a standalone CLI app, but I think it's also a good idea to have the bot be able to detect something out of sync too. Otherwise, people will forget to sync.

It doesn't matter, but it is still useful to have. I mean, it is still better to feel "at home" in the dev server.

If we were to have a config file that looked something like:

Unfortunately, trying to do this doesn't fall in line with my idea of a config overhaul, which was basically the following:

Define a config class in any module. A decorator or superclass will register it as a config class in a global table. In this case, the names' prefixes are used. It could be done with type annotations instead.

@config
class Config:
    maximum_thing = 1
    channel_general = 12345656778
    voice_general = 1223445423
    category_help = 2342342348
    role_admin = 2343234

Once registered, read from the config file and see if there are overwritten values that need to be assigned. A config file would look like this.

guild:
    id: 12312313
    categories:
        help: 2342342348
    text_channels:
        general: 12345656778
    voice_channels:
        general: 1223445423
    roles:
        admin: 2343234

bot.exts.my_ext:
    maximum_thing: 1

Note that a config file wouldn't be necessary if one wishes to use purely the default values as defined in the config classes. There's no order/category information for the channels because it would be cumbersome to require that to be specified for every channel defined in a class.

from aegir.

@MarkKoz so you’re saying that it should just be a cog on the development bot?

from aegir.

I think that should be one component of it. The cog can just be used to check if the guild is out of sync and resync it.

However, this project can do more than sync the server. It can have an interactive setup prompt that automatically sets the right config values based on questions like "are you using docker compose?". In this regard, a CLI app has the advantage of being simpler to implement and the ability to directly modify local files. A website would have a nicer UI, enabling us to show screenshots of how to fork, create a bot app, etc. as well as get OAuth2. Such website could technically run locally too but this sounds error prone and is relatively heavy on dependencies.

from aegir.

I think that should be one component of it. The cog can just be used to check if the guild is out of sync and resync it.

I feel like keeping everything in this project is a better idea, or we would have a lot of duplicate code between this project and the bot.

However, this project can do more than sync the server. It can have an interactive setup prompt that automatically sets the right config values based on questions like "are you using docker compose?". In this regard, a CLI app has the advantage of being simpler to implement and the ability to directly modify local files. A website would have a nicer UI, enabling us to show screenshots of how to fork, create a bot app, etc. as well as get OAuth2.

I was thinking about having an API so we can have bootstrap scripts like described in python-discord/organisation#221, but we will have to see about that later.

@MarkKoz would it be possible for you to draft an implementation of this new config format soon-ish?

from aegir.

I feel like keeping everything in this project is a better idea, or we would have a lot of duplicate code between this project and the bot.

I was imagining this to be a dependency, so the cog could be imported from this package.

would it be possible for you to draft an implementation of this new config format soon-ish?

Yes

from aegir.

I was imagining this to be a dependency, so the cog could be imported from this package.

I don’t think it is a good idea for two reasons, it adds unneeded complexity for both production and development, and if the config format changed and the bot can’t boot anymore, you’re pretty much screwed. But I’m not really strongly opposed to that.

from aegir.

I wrote up a demo here.

It works based off attribute names. This isn't the final decision, but it felt the easiest to implement for the sake of the demo. That being said, I do feel we're going to end up naming attributes with such scheme anyway just for the sake of readability, so why not leverage that further? For example, Config.help would be vague, but Config.help_channel is clear. Could also go with Config.Channels.help (similar to the system in production currently) but that seems more cumbersome to define.

As things currently stand with the demo, it's nice that there are no imports required to define the config; there are no custom types for annotations or any config base classes. Something else to note is that it's still possible to have e.g. a channel constant that is put under the extension's key - just don't use the _channel suffix. Unfortunately, this will not work the other way around - using the suffix for something that isn't a channel. I don't foresee it being hard to come up with alternate names to work around that.

The config class is in the extension testbot.extensions.sample:

class Config:
    my_channel: int = 400121469641818124
    my_category: int = 400121469641818123
    my_role: int = 400499713570242571
    my_voice: int = 647317854978310157
    my_webhook: int = 690000402552455210

    enabled: bool = True
    role_whitelist: List[int] = [400499713570242571, 400121903420801033]

Assuming the user wanted to customise all values, the config file would be

guild:
  categories:
    my: 11111
  text_channels:
    my: 2222
  voice_channels:
    my: 3333
  webhooks:
    my: 4444
  roles:
    my: 5555

testbot.extensions.sample:
  enabled: false
  role_whitelist: [111, 222]

but it can be shorter if default values are desired for some of the attributes.

It's possible to change it to keep channels, roles, etc. under each extension but I thought it was nicer to group those together. One thing that needs to be worked out is loading the config for extensions which are not loaded by default, but may be loaded via a command later. I have some simple ideas for that but didn't get around to it tonight. I also haven't worked out anything for a global config yet.

from aegir.

it adds unneeded complexity for both production and development,

Developing and deploying a website sounds just as complicated, if not more, than a PyPI package.

and if the config format changed and the bot can’t boot anymore, you’re pretty much screwed

I'm not sure what you mean. Keep in mind, with my proposal, everything would have a default value without any YAML config needed. Existing configs would only break if an attribute was renamed or moved (or we completely overhauled the format again, which is unlikely), but this seems unavoidable.

from aegir.

from aegir.

That's a great point. While it can easily be fixed by extending the default list, how would one distinguish between wanting to extend and wanting to overwrite?

from aegir.

from aegir.

Sorry, I must have misunderstood the point of your original comment. Can you rephrase it?

from aegir.

Alright, so after some discussions in #dev-core, here is the current plan:

• This project will host a PyPi package for every bits and pieces shared across the bots that use this config system
  • Although, should we make the code loading and updating the configs inside this package? I'd say yes
• We put the website as optional, and we will probably work on it after the first release, we'll see

from aegir.