hiyori-api / hiyori-cli Goto Github PK

• Allow resume of source
• Allow resume of failed list
• Maintain last entry data (page/offset, index)

Cache HTTP requests to speed up the ingestion process if on re-try or just wanting to update meta-data.

• Symfony HTTP cache
• ENV variable support

• Maintain Failed List

Could be maintained through failed_queue collection on MongoDB

It will be required to run this tool to get the best weightage to assign to the Hiyori Schema before running the Combiner with the following strategy: #18

However, this utility will initially be used to create a default weightage which will be added to a config.

php hiyori utility calculate:schema_weightage {source1} {source2}

Implement Symfony Logging via DI.

Levels of logging

1. Debug
2. Warning
3. Error

Features

1. Rotating Logs
2. Utilize ENV

• Accept option flag
• Update rather than skipping entries

• Go through the merged collection and check the IDs from available sources to see whether the IDs are valid or not anymore

php sweeper -- Default goes through all sources
php sweeper {source name} -- Go through specific source

Notes

• Clean titles. eg1 and eg2

e.g MongoDB

Related to #16

Probably due to the serializer. will need to look into.

Export combined metadata for HiyoriDB

Export Options

• Default
• Default Minified
• References only
• References only Minified
• Default minus Tags
• Default Minified minus Tags

Merge metadata from all sources into HiyoriDB.
Put unmergable entries into an unorganized collection.