These scripts and auxiliary files are used to extract the documentation from the wiki.rptools.info website and store them here under the processed/ directory.

The MediaWiki software adds some boilerplate that is removed in these files – we only keep the contents of <div id="mw-content-text"> but even that has pieces removed, like the table of contents and all HTML comments.

The end result is a Java properties file that we can manually copy to ../../src/main/resources/net/rptools/maptool/language/macro_descriptions/ under the name i18n.properties.

Hopefully, at some point these scripts can be modified to handle other languages!

I recommend that you execute these scripts as shown and compare the output to the tests/ directory.

It’s possible that changes to the wiki dump file format or the content of the pages themseelves could cause the scripts to break. The easiest way to detect that is to run the script(s) and capture the output, then compare that output to the previous version.

In my latest tests, this detected a change to the Python lxml module that cause the script(s) to produce incorrect output but without any errors. If I hadn’t done that, I could’ve spent a lot of time trying to figure out what went wrong with the wiki data.

By producing intermediate files, a human can look at the content — and even modify it — before proceeding to the next step.

1. Extract a list of pages that appear to be macro functions and save the list in the file wiki-has.txt

2. Read that list and retrieve the specified pages, storing them under wiki.rptools.info/.

   In the past, the list needed to be tweaked manually as some pages were not appearing for various reasons. The pages have since been updated so that valid pages are included and invalid pages are not.

3. Convert the wiki page HTML into small snippets of HTML with just the page content.

   The HTML for the wiki pages can be messy, but it includes all of the formatting that we want (bold, italics, etc) and some things we don’t want (like the table of contents and the "stub" template content). So we run the HTML through an XSLT that extracts just the div#mw-content-text text and throws away everything else.

   The results are stored under the processed/ directory under the name of the page.

4. Create a one-line description for each macro function.

   The content under the processed/ directory contains everything needed for the i18n.properties file, but the macro editor also wants a one-line description for each function. So we read those files, extract the first sentence, and write the result to processed.txt.

   This is a great place to eyeball the content and see if anything looks out of place. For example, if MediaWiki changed the structure of the generated page, you might see the wrong description lines show up here. But if everything looks good, continue to the next step.

   (Note that some parts of this should probably change. Right now, the macro editor doesn’t support links in the help doc in all cases. But if it does in the future, we’ll want the links in the href attribute to point to the corresponding wiki page. Until then, we should probably remove all attributes for anchor tags; that allows them to continue to display as links, but there’s no chance of them being activated.)

5. Combine the one-line description and the full description into the i18n.properties file.

   The lines are read from processed.txt and combined with the full description from the file under processed/ to create the properties file.

All done!