mtvirux / ffxivmt Goto Github PK

Should include:

• Project Description
• Tech stack
• Minimum reqs for hosts
• Current avaliable tools
• Emojis as that's the 2024 Github meta (?????)

Currently when query-ing for gilflux crafted only it will not show items from crafter collectibles
This can be achieved by utilizing the Scylla "from_scripts field"

Implement client cookies to remember the previous chosen option on dropdowns, in the various tools

The websocket worker container should look to heal the missing information from an outage or error.
This should be done by saving the timestamp of the last websocket message which will then be used on server start to know the time frame from which to request data from the Universalis API.

• Save timestamp from the last recieved WS message
• Request data from the Universalis API based on the the time gap
• Configurable maximum limit of timegap (e.g.: If outage was longer than 3 days, do not attempt to fill the gap)

Improve readability of the scylla prep script by making the wait lines overwrite themselves and stop presenting nodetool failed messages

• Wait message overwrites itself in the terminal
• Hide Nodetool expected errors (connection refused due to scylla_server not being ready yet)

Add configurable caching options for the API Gilflux controller

This should use the built-in Codeigniter caching options

By adding cache per location / per world, server responses should improve significantly at the cost of around 1.4GB of disk space (at based on the number of items at the time of writing)

The configurable caching options should be as follows:

• Disk / Memory (needs research) [currently only disk implemented]
• Per world / Per location ( w/ Per location being default as it's the most efficient one)
• Cache TTL (Default to 3 min. Currently lowest timeframe is 1h so this should be insignificant)

Create documentation for the functions under Application/Controller/API

Make the first start script auto-update the DB as to load static or low-volatility information such as:

• Items
• Worlds
• Recipes
• etc.

• Add .env variables
• Make script recognize .env variables

While the 6h in the Chaos DC is valid the 6h in the Light DC is not

The model fetches the highest timestamp (in this case Chaos') and tries to check if it's valid ignoring that Light had an invalid timestamp at the time of the DB fetch

Setting the query to get the min timestamp would cause the Europe results to get zeroed out

Fetching by world and then summing would be too costly as you'd also have to filter per world

This issue is happening on a datacenter scale as well

--------------------EUROPE-------------------------
{"log_timestamp":"2024-01-12 15:07:08","message":"Gilflux ranking for item 16564 on world  is OK on range of 7d."}
{"log_timestamp":"2024-01-12 15:07:08","message":"Gilflux ranking for item 16564 on world  is OK on range of 3d."}
{"log_timestamp":"2024-01-12 15:07:08","message":"Gilflux ranking for item 16564 on world  is OK on range of 1d."}
{"log_timestamp":"2024-01-12 15:07:08","message":"Gilflux ranking for item 16564 on world  is OK on range of 12h."}
{"log_timestamp":"2024-01-12 15:07:08","message":"Gilflux ranking for item 16564 on world  is OK on range of 6h."}
{"log_timestamp":"2024-01-12 15:07:08","message":"[13065000 > 1705061228000] Gilflux ranking for item 16564 is outdated on range of 3h."}
{"log_timestamp":"2024-01-12 15:07:08","message":"[13065000 > 1705068428000] Gilflux ranking for item 16564 is outdated on range of 1h."}
--------------------LIGHT-------------------------
{"log_timestamp":"2024-01-12 15:07:30","message":"Gilflux ranking for item 16564 on world  is OK on range of 7d."}
{"log_timestamp":"2024-01-12 15:07:30","message":"Gilflux ranking for item 16564 on world  is OK on range of 3d."}
{"log_timestamp":"2024-01-12 15:07:30","message":"Gilflux ranking for item 16564 on world  is OK on range of 1d."}
{"log_timestamp":"2024-01-12 15:07:30","message":"[48684000 > 1705028850000] Gilflux ranking for item 16564 is outdated on range of 12h."}
{"log_timestamp":"2024-01-12 15:07:30","message":"[48684000 > 1705050450000] Gilflux ranking for item 16564 is outdated on range of 6h."}
{"log_timestamp":"2024-01-12 15:07:30","message":"[48684000 > 1705061250000] Gilflux ranking for item 16564 is outdated on range of 3h."}
{"log_timestamp":"2024-01-12 15:07:30","message":"[48684000 > 1705068450000] Gilflux ranking for item 16564 is outdated on range of 1h."}
--------------------CHAOS-------------------------
{"log_timestamp":"2024-01-12 15:15:25","message":"Gilflux ranking for item 16564 on world  is OK on range of 7d."}
{"log_timestamp":"2024-01-12 15:15:25","message":"Gilflux ranking for item 16564 on world  is OK on range of 3d."}
{"log_timestamp":"2024-01-12 15:15:25","message":"Gilflux ranking for item 16564 on world  is OK on range of 1d."}
{"log_timestamp":"2024-01-12 15:15:25","message":"Gilflux ranking for item 16564 on world  is OK on range of 12h."}
{"log_timestamp":"2024-01-12 15:15:25","message":"Gilflux ranking for item 16564 on world  is OK on range of 6h."}
{"log_timestamp":"2024-01-12 15:15:25","message":"[13562000 > 1705061725000] Gilflux ranking for item 16564 is outdated on range of 3h."}
{"log_timestamp":"2024-01-12 15:15:25","message":"[13562000 > 1705068925000] Gilflux ranking for item 16564 is outdated on range of 1h."}

Add a branch for staging so no more testing in production 😅

Currently the Scylla keyspace has to have a set replication factor that's input in the "1-create_keyspace" shell script.

This should be configured dynamically through the .env file
Or even better making a dynamic script that doesn't require the users to edit files 👍

Make sure that the whole system recovers from a complete outage (not including the import of missing data as that is handled in another issue)

• Backend Container
• Websocket Container
• Local Scylla Container
• External Scylla

Gil trading is when two characters trade gil by utilizing buy/sell mechanics either from a Mannequin or the market board

While the system currently ignores Mannequin trades when adding to the sales ScyllaDB, some people appear to no be hindered by market tax and do it from the MB anyways.

Universalis has a Diff field based on the sale price compared to the Average Price Per Unit and this appears to be a good metric to detect these cases.

• Implement Average Price Per Unit fetching from Universalis
• Comparison with Average PPU at the time of filtering sales history from Universalis API

Creation of a frontend in whatever well known modern JS framework (e.g.: React, Vue)
UI details are open to whomever takes up the task but are still subject to review

Infrastructure:

• Docker container / image
• Correct connectivity with outside(HTTPS and take port 443 external connectivity from backend container)
• Correct connectivity with backend container

Content:

• Homepage
• Gilflux
• Item Profit Calculator
• Currency Efficiency Calculator

Make an appealing home page so it's not just text and a navbar 👍

Make the first start script interactive as to keep user from having to edit it according to their infrastructure layout

Table currently doesn't load as a datatable like Gilflux
The Universalis links are also missing

• Fix datatable loading
• Add Universalis link column