ackwell / manacutter Goto Github PK

As it stands, gqldn is doing some very naive naming tweaks on top of my shoddy-at-best name sanitisation. Should shore up this logic a lot to make it more consistent.

Possibly worth pulling in a lib for this, https://github.com/Humanizr/Humanizer seems to be a well rounded approach.

To support the relay pagination structure (connections), we'll need to finish implementation of the plural fields on the sheets query, as well as flesh out a lot of intermediary structures/interfaces in the schema. This will in part include bringing in a "node" interface and hooking it into all the sheet schemas, as well as generating globally unique IDs for them (probably b64 json).

references:
collection spec: https://relay.dev/graphql/connections.htm
node id spec: https://relay.dev/graphql/objectidentification.htm
gqldn base relay stuff: https://github.com/graphql-dotnet/graphql-dotnet/tree/master/src/GraphQL/Types/Relay
gqldn fleshed out relay stuff: https://github.com/graphql-dotnet/relay

The game data natively supports separate values for translated text.

Currently, we've got a hardcoded default to the english text. It will be necessary to support some way to specify the language that text should be fetched in, as well as a potential configurable default for the server.

This will likely play into #9 somewhat, as many shared-id sheets are for translated text.

Looks like NUKE is a popular build system. And by popular, I mean universalis and dalamud use it. 2 is a big sample size, okay?

https://nuke.build/
https://github.com/goatcorp/Dalamud/tree/master/build
https://github.com/Universalis-FFXIV/Universalis/tree/v2/build

Would be good to support patching the game files at runtime, possibly storing a few back compat versions.
Dats can be relatively limited, 04 and 0a should be enough for a baseline.

ref. https://github.com/pmgr/ZiPatchLib

Being a somewhat-relational database, numeric columns can be interpreted as a foreign key field, joining to rows in other sheets.

Some complications arise from the game's current usage of these fields:

• If the target sheet supports subrows, the field will only reference the primary row id - effectively making it a many:many join. We'll need to return a list of all subrows within the primary row in this situation.
• Some joins target different sheets depending on another column. We'll need some way to link to multiple sheets in this case. For GQL, this will probably involve unions or similar.

Some sheets share their row IDs 1:1 with another sheet. This is most common in the "Transient" pattern, where a table with a lot of data as well as bulk translatable text is split into TableName and TableNameTransient, with the latter containing the bulk text only, and the former containing the rest of the data.

Querying both of these tables for e.g. an action and it's description is undesirable at best.

Consider how these can be defined and/or derived.

To ensure cached data that relies on a schema does not become stale, a mechanism to tie cached data as a dependant of the core schema cache should be added.

MemoryCache actually supports this in a naïve fashion through cancellation tokens, we can probably build an API around the existing infra it has to make it easier to consume in the cross-package cross-cache nature we want to expire things.

Ref. https://docs.microsoft.com/en-us/aspnet/core/performance/caching/memory?view=aspnetcore-6.0#cache-dependencies

For feature parity with existing XIVAPI, and general usage, manacutter will need to be able to execute full text searches, as well as filter results within tables based on a few criteria.

This issue documents ideas and thoughts around potential implementations of the above.

Requirements:

• search index must support more than one game version / game edition at a time
• search index must support more than one active definition / definition source at a time

Game versioning
On the game version / edition side, this will likely necessitate seperate indexes / namespaces. Game data rarely changes dramatically, so there may be some disk space optimisations to be had in combining in some manner, but that may be non-trivial.

Definition versioning
For definitions, storing seperate indexes is likely not viable. Definitions can change regularly, and multiple sources would quickly cause significant disk space usage. As such, the best course will likely be to limit data indexing to purely what is available in the excel headers, i.e. column numbers and types.

At runtime, queries against the search index can then be built based on the choses definition mapping.

This does have the immediate repercussion that any structural data that exists purely within definitions (arrays, substructs, foreign keys, etc) cannot be represented directly within the search index. At current, I'm not sure if that's a problem.

Potential implementations

• ElasticSearch is proven-capable for this sort of workload, being used for current XIVAPI
  • ES is its own server software though, which complicates server structure slightly
• Lucene is the underlying logic found in ES
  • Has an official .NET port with support for .NET6
  • Can be integrated directly into manacutter, at the cost of a lower level api to manage

References
https://github.com/xivapi/xivapi.com/blob/master/src/Common/Service/ElasticSearch/ElasticMapping.php
https://github.com/xivapi/xivapi.com/blob/master/src/Common/Service/ElasticSearch/ElasticSearch.php