• getter/setter problem - kotlin call from java link
• configuration properties data class - use init{} and the primary constructor should work
• Gradle? no, use Maven to speed up IDE's project build time
• JSON parsing, tree node approach, data class by map approach, or swagger with our own fixing approach?
  • if tree node, can we have [] operator working as getter and setter?
  • if data class by map, how to deal with field?? there is no tool to generate that kind of class. maybe we could write an issue for a JSON to kotlin plugin
  • if swagger, how is the maintainability? can it save time from the other two approaches?
  • An: swagger approach. it is good at having enums, proper time format. by doing our own fix, we help to make a great open-source vocadb client.
• so kotlin has special definitions for properties, does Java getters return type? or type? it is type, honor project poc proved it
• but with Xjsr305=strict, all @Nullable java function (including getter) wil return Type? in kotlin, which sometime is useful, sometime is annoying
  • An: depending on situation, either write orThrow() or just !!

what features do we support? why

• all features from the vvd-downloader-old include
  • global retry mechanism
  • different downloader for different PV service
  • custom command line for how to launch a downloader
• retry setting on the single downloader, instead of global retry setting
  • this allows us to use a downloader that can continue if failed, by setting a large value
    • but this will falsely make other exceptions retry and retry unnecessarily
  • An: if a downloader relies on a retry-and-continue mechanism to successfully download a PV, such download must handle this retry-and-continue mechanism internally, instead of relying on the global retry mechanism
• retry mechanism in #12
• dynamic load downloader mechanism #13
• if pv-preference doesn't list one PV service, filter out all PVs from that service, and no need to load downloaders for that PV service

• instead of scanning -songInfo.json, we should use a dedicated JSON file with all other necessary info for the next module to proceed. The JSON info also includes any filenames associated with the song (including songInfo.json file)
• this can also fix the issue: to implement an advanced audio extractor choosing strategy, we need information from the previous module.
• don't write too many util functions for labels, instead, each module has its own helpers (reason: for better extendibility)
• also don't be confused that this label JSON is used to record necessary info and parameters that the next module needs
  • it is not the aggregated parameter object in #8

let's only worry about documents until we are ready to release this product

Problem: easy-batch doesn't support Listener class on a single intermidate processor, how to handle error path.
Also for writers, since it already group them in Batch, we can't do much for it

Possible solutions:

• just write error path logic inside the normal path business code
  • use an ErrHandler class
  • should we use Listener and reply on throwing exps to guide to err path? or should we just use conditions to guide to err path? An: long functions will split, and it will cause return value headache 😫
  • don't write delegate RecordProcessor class with exp handling. It probably won't work for all scenarios
• let writer class implemented in RecordProcessor class, and use Listener to handle exceptions.
  • This is not a good usage of easy-batch, although it can solve problems easily because with batch writing, we can support writing to DB or cache if we want

^{Created from JetBrains using CodeStream}

Instead of generating the whole client, just generate the DTOs is better and allow users to use whatever client they want

However, it is shown that our hack of allowing Enum class name ended with "s" to support multiple unique enum value is necessary. Because in API call, such as GET /api/songs with the parameter fields, It must be a column separated list with unique elements like fields=tags,pvs. Using something else like fields=tags&fields=pvs won't work

We are also fixed in Jackson because the model is problematic with Gson, see https://github.com/CXwudi/vocadb-openapi-java-client-autofixer#before-generate

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Pending Branch Automerge

These updates await pending status checks before automerging. Click on a checkbox to abort the branch automerge, and create a PR instead.

• ⬆ upgrade spring boot to v3.2.5 (org.springframework.boot:spring-boot-maven-plugin, org.springframework.boot:spring-boot-dependencies)

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• ⬆ upgrade dependency com.lmax:disruptor to v4

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

This will allow us to run tests with various dependencies on CI server.

Will create a single docker file that adds all dependencies in one docker image, and a base template docker-compose file for mounting either the project sources code or jar packages for testing and running.

Optionally, we can have another docker-compose file for building the jars

Since #3, we are only using the reader, processor and writer classes, and manually build and run the batch processing.

Then there is no need to use explicitly easy-batch's reader, processor and writer classes as it introduce too much overhead of wrapping and unwrapping the Record class.

Current naming is [vocalist]song name[producers], but if a producer self-remix a song with same vocalist, two songs will have exactly same name, and the -label.json and -songInfo.json will override another.

Solution: use [vocalist]song name[producers][vocadb-id] as the name

currently we just do a fix order of pv type like

  companion object {

    private val pvTypeToIntMap = mapOf(
      PVType.ORIGINAL to 0,
      PVType.OTHER to 1,
      PVType.REPRINT to 2
    )

    /**
     * to make sure that reprinted types goes behind original and others type
     */
    private val pvTypeComparator = ToIntFunction<PVContract> {
      requireNotNull(
        pvTypeToIntMap[requireNotNull(it.pvType) { "the pv type is null for ${it.name}?" }]
      ) { "More PV Type?" }
    }
  }

let's make it configurable by changing

[vocadb-video-downloader-new] vvd-downloader/src/main/resources/application.yml (Lines 22-23)

    try-reprinted-pv: true #to improve: can let users decided on pv type, similar to how pv-preference works, make sure add similar validation as pv-preference too

setting

• no spring-batch, in spring-batch, you need a job repo to store steps
• with easy-batch, follow #10 on how to handle exceptions in intermedium steps and in the writing step
• don't use easy-batch JobBuilder and JobExecutor, instead, we run the reader, all processors, and writer ourselves, using our favourite framework (currently using kotlin coroutine for parallel processing, and simply forEach for iterative processing)

Currently, no way for us to config two same downlaoders with different command line for even same pv service.

So in config.downloader, instead of having each pv service listing all available downloaders, lets it be a list of config with name. And the enablement will use the name as the order of downloaders for that pv service.

[vocadb-video-downloader-new] vvd-downloader/src/main/resources/application.yml (Lines 37-65)

  enablement:
    NicoNicoDouga:  # enable downloaders from NicoNicoDouga's available downloaders. e.g. nndownload
    Youtube:
    Bilibili:


  downloader:
    NicoNicoDouga:
      # settings of youtube-dl/ytp-dl for downloading niconico videos
      # settings must make sure nothing blocks from downloading the video (e.g. don't put --version on yt-dlp)
      # do the same for all other downloaders.
      youtube-dl:
        launch-cmd:
        external-args:

      nndownload:
        launch-cmd:
        external-args:

    Youtube:
      youtube-dl:
        launch-cmd:
        external-args:

    Bilibili:
      youtube-dl:
        launch-cmd:
        external-args:

Open in IDE · Open on GitHub

^{Created from JetBrains using CodeStream}

and also consider cutting down unneeded features

when we have more and more downloaders choice in the futures, we can't force users to fill in all downloaders config, so we need:

• multiple downloaders for one PV service
• validate that a downloader for one PV service can not be used in another PV service
• load downloaders base on enablement and pv-preference
  • will probably use spring @ConditionOnBean to check if the corresponding downloader config existBIG NOTICE: base on Spring Doc, Spring Boot issue 1 and issue 2, any @ConditionalOn annotations that rely on knowledge about Spring beans will NOT work outside of Spring auto-configuration (PS: @ConditionalOnProperty would probably work)
  • downloader config bean is @ConditionOnExpression with the check if its name is in enablement config map Because of this, SpEL can not resolve properties (except @Value). So we need another solution to load only enabled downloaders
  • other solutions, see below comments

instead of thinking about what parameters does each method need, just uniformly make methods receive and output a giant parameter object only. Each module will have their own parameter object, and it contains anythings needed to process one data through the entire batch processing procedure

It doesn't really care if u rename them all as mkv and jpg, but instead we should properly handle it by either

• convert it right away
• record what format is it (and such info is readable by the next module)

The current way of sorting and filtering PVs are sucks and buggy. Here is a way better mechanism:

config:
  preference:
    # a list of <PvService>.<PvType>
    chosen-pvs: NicoNicoDouga.Original, NicoNicoDouga.Reprint, NicoNicoDouga.Other, Youtube.Original, Youtube.Reprint, Youtube.Other, Bilibili.Original, Bilibili.Reprint, Bilibili.Other

This single option can cover three old options altogether: pv-services, try-reprinted-pv and try-all-original-pvs-before-reprinted-pvs

notes for retry in downloaders: (TODO)

• if one downloader failed, try the next downloader on the same PV
  • allow users to choose multiple downloaders for one PV service
  • allow us to try the faster risky one, if not successful, use the safer slower one
  • make sure that the global retry count is independent for each downloader
• if one PV service failed, try the next PV service
  • not all the time that VocaDB info of PV availability can be up to date, so we need our side to handle it
• if all original PVs from all services are failed, try download reupload PVs
  • this could be useful if a Vocaloid producer deleted all his/her contents, but other ppl upload a copy
• all retry features are configurable
  • how to resolve PV preference sorting base on current enabled retry features
  • An: sort and filter PVs base of all setting aboves

There is no need to write a new extractor, just the python mutagen tagging code

Make sure do the POC first

basically an implementation of
https://kotlinlang.org/docs/delegated-properties.html#storing-properties-in-a-map but in java

So that we can prefer over PVs from 无色透明祭 or from producers themselves

The proper way is to use YAML's list, see example

Using any other solutions (e.g. space/comma separating) will cause command line injection issue

#2, #3, #5 and #8 must be followed
when #2 is followed, make sure the output stays compatible with other modules.

use https://github.com/spring-projects-experimental/spring-boot-thin-launcher to share similar jars

won't fix until we done all the developing