I would like to update the existing ODD, in two steps, and this ticket is meant for the first and gentler of them, namely for a rewrite of the current ODD to the current TEI idiom, which should ideally mean just a cosmetic change without affecting the extension (i.e., the patterns/grammars defined by RNG, XSD, DTD), but in practice, the extension is going to be affected due the the changes in the TEI that have happened over the years, so some tinkering may be in order, and a lot of test runs across all the databases.
In doing that, I would like to add two files to our version control. For strictly internal purposes, so that we can trace the changes in the TEI internals without investigating the git history of the TEI itself, each time.
Let me sketch some background:
- the TEI ODD mechanism is in essence a customization / documentation mechanism that targets a set of all the definitions encoded by the TEI Guidelines.
- that set is not present in a cloned TEI repository, but rather gets derived by the make system (via TEI Stylesheets, which is a set of tools that accompanies the TEI Guidelines) and resides in a cryptically named document called
p5subset
. It is called an 'integrated ODD'.
- any typical ODD document created with the appropriate TEI tools is meant to tailor the integrated ODD down to a particular purpose: manuscript description, corpus encoding, dictionary encoding, etc.
- the application of the Freedict ODD to the integrated ODD (
p5subset
) silently creates something that can be called Freedict integrated ODD; it is not visible to the outside eyes, because it is regenerated each time that the Freedict ODD is manipulated by the TEI Stylesheets.
- the 'Freedict integrated ODD' is used (or rather: was used) to derive the schema documents: RNG (of primary use for us), but also XSD and DTD (which we provide more or less out of courtesy -- but I can imagine us not providing these two, to avoid having to address the potential issues if someone decides to use those instead of the RNG)
- I stress the "was used" because, simplifying the history slightly, that happened once, years ago: I ran the TEI tools on the current Freedict ODD and created the three schema documents. Note the crucial issue: they were ran on the
p5subset
as it was defined by the TEI years ago. So while the Freedict ODD hasn't been modified since then, the result of its application on the current p5subset
is going to be extensionally different from what was used years ago. I don't think it's a major issue (because we only use a very small subset of the TEI), but it's definitely something to be aware of.
- one more relevant issue and an argument for 'freezing' the
p5subset
in our version control is that, if one doesn't have full control of the TEI environment, their ODDs may reference the current 'blessed' TEI ODD, recreated after each release in the TEI Vault, or the current snapshot of the TEI under control of their Jenkins environment, or the local p5subset on the user's hard drive; what I propose reduces this potential complexity and adds a lot of transparency.
A hopefully minor complication is that our RNG was edited by hand since it got derived. Since it is version-controlled, I can extract the modifications and reapply them at the ODD level.
Another hopefully minor issue (but actually part of a larger issue suitable for a separate task in a separate ticket) is the way to make sure that the newly derived RNG is still valid for all the dictionary databases. I seem to recall that the Freedict make system had a 'validate' target, so I imagine that, after regenerating the RNG, I would only have to run make with the specific parameter, and watch for error messages. @humenda , do you sense any trouble in this regard, please?
EDIT: this is now the topic of freedict/tools#28 and I have an interim solution
I mentioned adding two files to the version control. I meant the current p5subset
and the Freedict integrated ODD (call it... freedict_p5subset
?). The first one freezes the current state of the TEI, so that, in the future, we can diff that. The second is to expose the Freedict integrated ODD for similar comparisons. I could probably live without the latter, since it depends on the former, but it also depends on the TEI stylesheets, and those are under constant development as well. Bottom line: it's far more convenient in case one has to investigate some schema-related issue across time, to have both these files handy, because both of them can only be recreated in the future after tinkering with two very dynamic repositories (TEI Guidelines and TEI Stylesheets).
Envisioned action sequence:
- derive the current p5subset (on my disk, against the current snapshot of the TEI and TEI Stylesheets)
- freeze the
p5subset
by adding it to Freedict version control (where? under shared/
or elsewhere?)
- derive the current
freedict_p5subset
by using the current Freedict ODD, with one change: its @source
attribute will now point at the p5subset
frozen at step (2)
- derive the RNG and check if all the databases validate against the RNG
- freeze the newly derived
freedict_p5subset
next to the p5subset
; this one should be regenerated by hand after each modification of the Freedict ODD (one has to remember about that); recall: it's frozen for convenience, to shield it from any ensuing modifications in the TEI Stylesheets
- rewrite the current Freedict ODD, just for the syntactic sugar
- (recurring step) derive the RNG and check if all the databases validate against the RNG
- commit the newly created
freedict_p5subset
just to document any modifications that could have crept in at step (6)
- check our RNG version history for potential modifications introduced by hand, and see if they need to be handled at the ODD level (it might be that the underlying TEI has caught up with them, during the years that passed), if an ODD rewrite is necessary, then repeat steps (7) and (8)
At this point, after all the above actions, we should be still at the status quo, except with (a) 2 new files, kept for reproducibility checks and (b) a newer Freedict ODD, ready to be modified further.