godotmodding / godot-mod-loader Goto Github PK

as per the info by thunderstore, a "full id" would be {namespace}-{name}-{version_number}
I'm not sure what we would call this id, maybe something like "full_name_id"?

the validation does address the composition, but not the elements of that composition. following thunderstore,
namespace and name are are validated with this ^[a-zA-Z0-9_]$ (where _ can only be replaced with space for display purposes)
and the version_number with this ^[0-9]+\.[0-9]+\.[0-9]+$
Godot does have a regex class you can use

Originally posted by @Qubus0 in #51 (review)

Brotato uses a custom version of ModLoader that has support for loading from Steam Workshop folders.

We should implement their work, so that Brotato can be updated to use the latest version of ModLoader without needing custom edits.

The modified version of ModLoader is attached, decompiled from the current Brotato PCK (v0.8.0.0-beta). It's based on version 4.1.

ModLoader-v4.1--Blobfish-Edit.zip

We have to check if we can overwrite an existing ModLoader Instance in a pck file.

So the version for Thunderstore can overwrite a build in one.

KANA — Yesterday at 16:46
I need a way to tell a mod ( Darkly77-ContentLoader ) that it has to wait for my mod ( KANA-WhatAmILookingAt )

Ste — Yesterday at 16:49
technically only by dependencies.. we could introduce a "load-before" to store more mod ids that should load after this mod
load before would essentially copy over the importance score of the other mod and add it to the own importance

The small utility program GodotPCKExplorer can extract an embedded PCK from its EXE.

And as @Qubus0 discovered, it also has CLI support.

Could we add support for doing this as part of the setup process?

The CLI args aren't explicitly documented, but they can be found in the source code -- see here (ctrl+f: "-e").

Note: This can come after PR #81 #89, which doesn't need to be held up by this implementation

Ste — Today at 16:23
should we have different logs levels for mod_log? INFO, WARNING, ERROR and just add those at the front of the line? would make reading the logs easier if you use a highlighter

Something like:

11.1.2023 - 08:28:20 ModLoader-INFO: Initializing -> Cube-HarvestCalc

It looks like there's a lot of new stuff in the latest release that's not documented.

https://github.com/GodotModding/godot-mod-loader/releases/tag/v5.0.1

• New CLI args @KANAjetzt
  • --only-setup
    #94
  • --setup-create-override-cfg
    #99
• New auto setup process @KANAjetzt @Qubus0
  #75
  #89
  #94
  #99
  #105
  #110
• overwrites.gd @KANAjetzt
  #74
• overrides.cfg @Qubus0
  #75

I haven't worked on these features so I think they might need to be added by the person who worked on them

*Edit by @KANAjetzt
Added @
*Edit2 by @KANAjetzt
Added PRs

this will make everything easier to maintain and simplify merges
the readme can link to pages in the wiki instead of containing everything.

Load Mods into a sub folder

3. Have a 2nd root-level folder for unzipped mods (eg "res://mods-unpacked"). The folder structure inside them can stay similar to now, just with the mod files being inside the unpacked directory, eg "res://mods-unpacked/Darkly77-Invasion"). ModLoader then iterates over each directory within "mods-unpacked" to find meta.json and [ModMain.gd](http://modmain.gd/) files, and inits them from there
https://discord.com/channels/1028191864966361178/1028381928275058838/1058717881346560061

• At the moment every mod gets loaded into res://Mod-Folder
• Let’s make that res://mods/Mod-Folder

List of things that need to be added to the wiki. To prevent this issue becoming infinitely expanded, this issue focuses solely on adding docs for mod developers.

• Using zipped mods in the editor stops mods from res://mods-unpacked from working
• godot.log / mods.log
• Adding child nodes to ModLoader.{mod_id} - eg
• Using child nodes - eg
• Decompiling games - eg
• GodotSteam - eg

Any other suggestions please keep 'em brief so I can just add them to this list 🤗

Darkly77:
Another use case for replacements: You could do stuff like make a balance pack for Brotato [...]

I've found that replacing files can be a bad idea because it can mess up extended scripts.

I was trying a few things with the file character_selection.tscn, and after replacing it I found that the extended scripts for character_selection.gd (which is referenced in character_selection.tscn) no longer worked. Replacing .tres files likely has the same negative effect, where any .gd scripts that are referenced in that .tres file will ignore their extensions.

This needs further testing though.

Originally posted by @ithinkandicode in #14 (comment)

(this is mostly copied from Discord)

It would be cool to have a JSON config loader. Eg. my DebugLoader mod already uses a config JSON, and WikiTools has hardcoded settings that I need to replace with JSON-sourced settings instead. So it would be great if we had a standardised system for handling config files, that passed loaded data to the mods when they init.

I'm thinking the config implementation could be this:

• Each mod can potentially have its own config file, in a folder called config, with the filename matching the mod ID.
  • Eg for Invasion, its config file would be here: res://config/Darkly77-Invasion.json
• We can store the settings in a variable on modloader, eg ModLoader.configs["Darkly77-Invasion"]
  • That way we can pull the settings from the file (if it exists) by only knowing the mod_id, which we'll know before we run func _init_mod(mod) -- which means that, when the mod's _init fires, that mod can access its config data.
• We can also add a simple utility func to get settings from a mod's own JSON

Example code for the setting utility:

func get_setting(mod_name:String, setting_name:String):
    return configs[mod_name][setting_name]

Example JSON:

{
	"enable_potatoes": {
		"description": "Adds more potatoes",
		"type": "bool",
		"default": false,
		"current_value": true
	}
}

Misc Ideas:

• Include a load_from option (string), which is a path to a different config JSON file. I did this in DebugLoader, it lets you override the settings with settings from a different file (non-recursive ofc). This lets you have multiple config files you can easily swap out, depending on the situation
• Include a debug_log option (bool). This can be checked by mod_log, by matching the log_name against the mod's own debug_log option. It would let you selectively enable verbose logs for a specific mod. Could also be called verbose_log.

• Try out the .import folder at root of mod folder approach
  • That would save use from rewriting the .import files

• Works fine 👌

• Now do we want to add the mods import files right into the games .import folder.
  Or keep it separated and close to the mod folder?
• I think it’s fine to copy them in the base .import folder
  • It’s the default spot for imported files
  • The files are hashed so there should be no overwrite issues?
  • Whit that we just need to copy the files into the mods .import and don’t have to mess with rewriting the .import file of each import.

Make sure that there is a translation resource on the given path.

If no translation resource is at the given path, the translation server is not happy.

Add some validation to this:

func add_translation_from_resource(resource_path: String) -> void:
	var translation_object: Translation = load(resource_path)
	TranslationServer.add_translation(translation_object)
	ModLoaderUtils.log_info("Added Translation from Resource -> %s" % resource_path, LOG_NAME)

I think the auto setup process will need to be updated to include the file script_extension_data.gd, which as introduced in this PR:

• #124

Should we move the custom resource classes to their own directory? Eg. mod_loader/classes. It would contain:

• mod_data
• mod_manifest
• script_extension_data

I think it would clean up the root mod_loader directory a bit and help newcomers understand what each file is doing

          One mod can also add itself as a dependency

Originally posted by @KANAjetzt in #111 (comment)

for ease of setup for everyone

• There is a helper function to add a node to modified scene but you cant define the position of the node

• add add_child_below_node() to ModLoader.append_node_in_scene()

ModMain.gd Script Extension:

func _init(modLoader = ModLoader):
    modLoader.installScriptExtension("res://mods-unpacked/KANA-MultiRes/singletons/utils.gd")
    modLoader.installScriptExtension("res://mods-unpacked/KANA-MultiRes/singletons/progress_data.gd")
    modLoader.installScriptExtension("res://mods-unpacked/KANA-MultiRes/main.gd")
    modLoader.installScriptExtension("res://mods-unpacked/KANA-MultiRes/ui/menus/run/end_run.gd")
    modLoader.installScriptExtension("res://mods-unpacked/KANA-MultiRes/ui/menus/shop/[shop.gd](http://shop.gd/)")
    modLoader.installScriptExtension("res://mods-unpacked/KANA-MultiRes/ui/menus/title_screen/title_screen.gd")
    modLoader.installScriptExtension("res://mods-unpacked/KANA-MultiRes/ui/menus/pages/menu_general_options.gd")

shop.gd

class_name Shop
extends Control

signal item_bought(item_data)

export (Array, Resource) var combine_sounds = []
export (Array, Resource) var recycle_sounds = []

var _reroll_price: = 0
var _last_reroll_price: = - 1
var _shop_items: = []
var _go_button_pressed: = false

var _initial_free_rerolls = RunData.effects["free_rerolls"]
var _free_rerolls = _initial_free_rerolls

var _need_to_set_locked: = false

...

The issue is the

var _initial_free_rerolls = RunData.effects["free_rerolls"]

RunData is not initialized when the script extension is called

Originally posted by @KANAjetzt in #3 (comment)

Make it possible to mark a mod as optional dependency.

• loop through optional_dependencies
• check if mod_id exists
• if so increase importance score by 1

Ste - January 3, 2023 11:37 AM
Hey btw, back to the idea of only needing the modloader and not some edits to the source code (like adding it as auto load): godot can override project settings with a override.cfg placed in the same directory as the exe. One problem being that it completely replaces it and can’t just ‘add to it’.. though there might be a way, since you can save the project settings as a custom file (https://discord.com/channels/212250894228652034/342056330523049988/873285063805124678)

I can save these config settings very easily using the following:

var path = ProjectSettings.globalize_path(OS.get_executable_path().get_basename() + "override.cfg")
ProjectSettings.save_custom(path)

That means we could possibly run the project once, make it add itself to the auto loads, create the override and close again

Ste - January 3, 2023 11:51 AM
Adding it as auto load might be a bit annoying though, since it is added as last in the list. So get existing, hold them, remove them, add loader, add them back..
also, save_custom saves all of the settings, which might be bad.. and projects using this mechanic might be affected (though i've never seen a project use that)

In mod_loader_utils.gd, we need to use OS.get_datetime instead of Time.get_datetime_dict_from_system.

This is because the Time singleton was only introduced in Godot 3.5, so it won't work for older games (eg. Dome Keeper, which is on 3.4).

The code we need to update is in our custom func get_date_time_string.

We should make a switch from .zip files to .gdmod (PCKPacker) or whatever we want to call the mod file extension, to not confuse users that might wanna manually install mods, so there would not be an issue with them assuming it's something they might want to extract

In #138 I added the array variable logged_messages. I think this is a potentially very useful approach that could be applied it to all logged messages.

Implementation:

• In _loader_log, we save logged messages to an array.
  • This is the same as what I did for #138,
  • but with the array variable being in ModLoaderUtils instead.
• For each message, we can track all the data parts that are passed as variables to _loader_log
  • Ie. mod name, message, and type.
  • Actually, it might be better as a dictionary, with these keys:
    • all = All logged messages
    • by_mod = Notices logged via a certain mod
    • by_type = Notices of a specific type

API Methods:

• get_tracked_messages_all()
• get_tracked_messages_by_mod(mod_name: String)
• get_tracked_messages_by_type(type: String) -- where type is one of the current log types (fatal-error, error, etc)

Use Case:

This would be a massive help to players who are using mods, as it would save them needing to check the logs (and save them needing to know how to do that). This does depend on the game or a mod presenting this data, but this is the first step towards supporting that.

Note: This isn't related to UI at all. It simply provides data that a UI could interact with, which would be the responsibility of the game developer or modders -- like how Brotato's Mods screen interacts with ModLoader's current data (see example with otDan's mod BetterModList). So on that screen, it could include a console-like tray with any errors or warnings.

If it's not the first one, something has probably gone wrong, or the user needs to set up their autoloads. Using assert would tell the user about this issue immediately.

Snippet to get autoload order, via here

# Log Autoload order
var autoloads := {}
for prop in ProjectSettings.get_property_list():
	var name: String = prop.name
	if name.begins_with("autoload/"):
		var value: String = ProjectSettings.get_setting(name)
		autoloads[name] = value

ModLoaderUtils.log_debug_json_print("Autoload order", autoloads, LOG_NAME)

We may also want to do something like this to restart the setup process, as it may mean the the PCK got a vanilla update.

• Note: If we do take the approach of "if not first autoload, run setup" then we will need to be careful, because if a bug occurs that technically completes the setup but doesn't successfully add ModLoader as the first autoload, then we could enter an infinite loop, ie:
• setup > bug occurs but setup completes > our check sees that modloader isn't the first autoload >
  setup again > bug occurs again > etc

Add an option to _meta.json to allow a mod manager app to enable/disable it, eg:

"enabled": true

Check if the file structure of a mods zip file is correct:

Structure
Mod ZIPs should have the structure shown below. The name of the ZIP is arbitrary.

yourmod.zip
├───.import
└───mods-unpacked
    └───Author-ModName
        ├───ModMain.gd
        └───_meta.json

We could add support for checking a file called mods.json in the user data folder, and when ModLoader inits, we could check that file and skip any disabled mods.

We could also provide a func to save the settings to this file.

This would let games implement a toggle to enable/disable certain mods from within the game itself, so users don't need to unsubscribe from them in the workshop/Thunderstore, or manually delete/rename mod ZIPs from their game folder.

*I just remembered that you will have to check for duplicate mod IDs since those are just user-made*

https://discord.com/channels/1028191864966361178/1028381928275058838/1056907018583154780

• Maybe we can also add something generated, after the user based ID, to prevent duplicate IDs.

Atm, if you retrieve a Config JSON setting and there's no custom JSON file, it logs this every time to try to retrieve that setting. This can flood your debug log.

I propose that it only logs once. The log message can be useful, but once it's been stated the first time, there's no need to repeat that statement.

Add an arg to all the log methods that let you only log the message once.

This would provide a standardised way to implement what I did for #138.

Doing this may depend on implementing #140 first.

Related:

• #138
• #140

We need to provide some changes to have thunderstore support our loader and godot games in general.
Currently we want:

• then mods path that can be provided as a cli argument
• a config folder that can be provided as a cli argument

Edit by @ithinkandicode to add checklist:

• CLI arg for mods path -- #28
• CLI arg for configs path -- #31
• Tightly couple mod ID to package ID -- #51
• A standard way for mods to handle configs, tied to mod ID -- #31
• Move non-Thunderstore metadata keys under their own object -- #27

I seem to have encountered a bug when using the get_mod_config method, where it's not returning the expected value.

I need to investigate and fix it if needed.

Darkly77 — Today at 04:39
Should we start using versions for dependencies? Eg.

"dependencies": [
    {
        "name": "Dami-ContentLoader",
        "version": "2.0.0"
    },
    {
        "name": "Darkly77-BFX",
        "version": "1.0.0"
    }
],

Image
We'd need to use the semver compare approach I mentioned in this issue:
#13
Only issue is, all ModLoader mods would have to use semver -- ie x.y.z ( major.minor.patch) -- for it to work correctly
which might be too strict
though we could probably account for major.minor, as the .patch version in that case can be presumed to be .0
same with minor version actually

• We might be able to use the load_resource_pack() with replace_files set to true, to easily overwrite for example textures.

• That would make a mod like Brotato-Explosion-Mute as easy as locating the image file you want to replace and adding a new one at the mirrored location in the mod zip.
  ( In this case res://projectiles/rocket/explosion.png )

• If not handled with care, this has potential for carnage

If 2 mods depend on each other the dependency check will run into infinity.

Atm there's a requirement with Brotato to launch the game with --script run.gd. Can we remove the need for the cmd line arg, and potentially for the run.gd file?

I'm not sure what it does/why it's needed so if anyone can illuminate me I'll be happy to discuss options and workarounds and potentially code a fix.

Problem

As discussed on Discord today (here), if we want parity between the Thunderstore manifest.json and our current one, then we need to ensure that dependencies have version strings, eg:

"dependencies": [
    "Darkly77-ContentLoader-5.0.0"
]

But we also need to support dependencies without them, because Brotato's workshop support has launched without this requirement. This means that this also needs to be valid (like it currently is):

"dependencies": [
    "Darkly77-ContentLoader"
]

Suggestion

• Add a new mod ID validation check to ensure that a version string is present.
  • If they are missing, trigger a fatal error in the editor (to notify the mod creator).
  • However, outside of the editor, allow normal execution of the code (ie. proceed to the next step of validation, below).
• Keep the legacy validation, which doesn't check for the version string.
  • An error here should prevent the mod from loading, whether you'r in the editor or not (just like it currently does).

PR #145 puts various settings into ModLoader.ml_options. I've put a bit more thought into that than what I've covered in the PR, so I'll cover it here.

TL;DR: Dictionary = JSON, user configs, standardised code 🤗

Dictionary

We can use the dictionary to keep track of data that's currently loose variables. This would help differentiate data that can be user-controlled vs. internal data that's only changed programmatically. Eg:

• os_mods_path_override      -> ml_options.path_to_mods
• os_configs_path_override -> ml_options.path_to_configs

JSON

Doing this also lets us work towards letting these settings be overridden via JSON too. And having settings that can be controlled via JSON would allow a game or mod to provide a GUI to do that (though ofc only a couple of settings would benefit from being exposed like this, namely log_level and enable_mods).

This can also be related to #109, as we can store all the user options (including the active mods from #109) in a single JSON file, eg called mods-options.json, as keeping all the settings in one single file would reduce the number of file IO operations.

ML Options

And taking #109 further, we could also add support to disable certain mods during mod development via ML Options, by building off of #145 (though I haven't mentally planned out the implementation of this yet).

CLI Args

We may also want to revisit the CLI args we currently have. If our CLI args affect the options dictionary, we could standardise them to follow a certain format, like --setcustom-*. For example, using the variables defined in the ML Options PR (#145), the refactored CLI args might be:

• --setcustom-path_to_mods = --mods-path
• --setcustom-path_to_configs = --configs-path
• --setcustom-enable_mods = --enable-mods
• --setcustom-log_level=1 = --v
• --setcustom-log_level=2 = --vv
• --setcustom-log_level=3 = --vvv

Related

• #109
• #145

Once the basics of #14 are implemented, I would like to see if its fessable to track witch mod is overwriting what resource. To potentially log a warning if 2 mods overwrite the same thing.

To do this with the implementation from #74 the overwrites.gd or the overwrites folder has all the information we need to do this. If the mod creator followed the recommendation to mirror the games folder structure inside the overwrites folder.

So 2 possibilities I can think of:

A) Regex the overwrites.gd for paths starting with res://
B) Use get_flat_view_dict(overwrites_folder)

Then save the paths to the original resources in mod_data and compare.

I think the valid status codes (0/1) should grouped, so that statusCode > 1 always means an error.

Atm to check if a status code is OK, you have to check for "is status 0 or 2", which isn't very elegant.

Current status codes:

Proposed:

to prevent logs being overridden immediately, I propose that we keep 4 (or what is set in ProjectSettings logging/file_logging/max_log_files) older logs, with the time they were created in the name

Ste — Today at 00:24
Ugh we have one break between 3.4 and 3.5.. mod_log uses Time which does not exist in .4 (OS is used instead)

• Maybe we can progressively enhance by checking if the Time Class exists
• If not we can just use OS.get_datetime() in 3.4 and 3.5

_Dictionary get_time ( bool utc=false ) const

Deprecated, use Time.get_time_dict_from_system instead.

Returns current time as a dictionary of keys: hour, minute, second.
https://docs.godotengine.org/en/stable/classes/class_os.html#class-os-method-get-datetime

Show the version in the main file, mod_loader.gd.

Suggestion: Add a required setting in _meta.json to set the required version of ModLoader.

Requirements:

• 1 - Add a key to meta.json that specifies the required version of ModLoader.
• 2 - Add support for this to ModLoader: If ModLoader's version is below the required version, don't load the mod.
• 3 - Add a const to ModLoader for its current version, eg const MODLOADER_VERSION = "2.0.0"

Since we're using a semantic version string for the version, we will need a custom compare func to check if the ModLoader version matches or is below the mod's specified version.

If it's below, we can check the major versions based on semantic versioning (major.minor.patch, eg. 2.1.16). For example, a mod that requires ModLoader v2.1.0 should still work in v2.2.0, but won't work in v3.0.0.

Using Config JSON currently has 1 method, which loads configs. But there's currently no way to save data.

It would be good if we could implement this, as it would save mods (or vanilla games) from having to implement it for each game that uses it.

My approach would be to create 2 methods. One saves the entire config, the other saves a single setting:

func save_mod_config_dictionary(data:Dictionary):
    #...

func save_mod_config_setting(key:String, value):
    #...

Might be a good idea to return a bool for success/failure (or a status code, if more info can be returned regarding errors).

To help with this, we could also add a generic utility method, simply for saving JSON file data (or maybe saving a string to a file? I'd need to get started on the code to see what's possible/optimal).

This discussion has come up a few times so I've made a proper issue for it. The loader can support either ZIP or PCK, so it's not a decision we need to make in a hurry, but it would still be good to discuss this to completion.

So: What are the cases we can make for publishing as ZIPs vs. PCKs?

ZIP:

• Universally understood.
• Very easy to make with existing tooling: Zipping is built into Windows, and there's lots of software for other OS's.
• No extra build steps or setup required.
• Easier to extract source code from published mods than don't have a repo, as the code is 1:1.
  • This makes them easier to maintain -- eg. if the dev leaves the scene or takes a hiatus.
  • And makes them easier to remix -- ie. new modders taking a new spin on a mod.
  • It also makes creating translations much easier (as GDRETools' CSV extraction doesn't include the keys).
• Can be packaged by any tool, so not tied to Godot.
  • This is good for devs who primarily work in other languages, as they can produce personal packagers particular to their preference (eg. I could easily make a ZIP packager with node.js + Electron because I'm very comfortable with those).
• ZIPs are better for Thunderstore for a few reasons -- see Mythic's feedback on Discord.

PCK:

• As far as I understand things, the sole benefit of publishing PCKs is based on potential options for creating Godot-based tooling:
  • Can be packaged via GDScript, whereas ZIPs can't be. So if we develop tooling to package mods, they'll have to be PCK.
  • Tooling will need docs and an interface that anyone can use, and since it's an extra step it needs to be more convenient than just zipping up a folder.
  • The tool(s) would ideally be easy to update by anyone in the scene, both for maintenance, and to make it easy to expand with ideas from other contributors (it's good to keep in mind that to making mods for games like Brotato doesn't require experience with GUI editing).

Based on this, it seems like ZIPs are the most convenient option.

What are the other benefits of using PCKs?

Related: #22

empty strings and any other other non-string values should be ignored