dishmint / markdown2wl Goto Github PK

This is generally working outside of MarkdownParser.wl. Needs to be integrated into the package.

Implement Import and Export converters for markdown files

Forego the use of the 'exhibit of markdown' example markdown page and write one up from scratch. It will be much easier to assess the effectiveness of the parse with succinct examples.

With MarkdownLexer implemented, I should make sure a comprehensive set of tests are added.

Make sure all exposed symbols are reflected in the documentation.

Add CopyAsMarkdown menu item so one can copy a cell into markdown formatted text

Add parser support for SubItems etc.

Some nested MarkdownElements are not rendering correctly

branch here: https://github.com/dishmint/Markdown2WL/tree/feature/renderer/dishmint/MarkdownRender_Init

The parser tests need to be verified. I constructed them by running ImportMarkdown on the test input from the markdown field in the commonmark v0.30 JSON tests. There's no guarantee they are correct right now. A brief look at some of the initial tests suggests they probably all are not correct. (or at least not expected)

Recognize HTML and parse as HTMLElements.

It may be as simple as detecting the html snippets and importing them as symbolic HTML.

The parser refines tokens into their final symbolic form

• Parser
  • Emphasis (Italics, Bold, InlineCode, LaTex)
  • Blocks (CodeBlock, Tables, Lists)
  • Links (Hyperlinks, Footnotes, Images)

In the default stylesheet item indentation is limited, so I think this warrants a custom stylesheet.

Stylesheets to reference:

• Outline

The \\s{2} could be a problem if people use different indentation-lengths.

(* OrderedListItems *)
    RegularExpression[ "^((\\s{2}|\\t)*)((\\d\\.)+\\d?)\\s(.*)$" ] :> $TokenLevelData[ <| "Token" -> "OrderedListItem", "Level" -> GetIndentationLevel["$1"], "Data" -> "$5" |> ]

There are three possibilities: (1) set the indentation-length with an option, (2) set the indentation-length per flavor, or (3) normalize all leading indentation to spaces or tabs

For some reason ExtractMarkdownFootnoteURL enters the Private context when extracting the 3rd footnote from test.md

 First[Private`ExtractMarkdownFootnoteURL[3, Private`footFile]]]

MarkdownElement["Item",
    <|
        "Type" -> "Unordered",
        "IndentationLevel" -> 0,
        "IndentationType" -> None, 
        "Content" -> {
                "A named link to",
                MarkdownElement[Hyperlink, "MarkItDown", First[Private`ExtractMarkdownFootnoteURL[3, Private`footFile]]], 
                ". The easiest way to do these is to select what you want to make a link and hit ",
                MarkdownElement["InlineCode", "Ctrl+L"], "."
                }
    |>
]

Parser test failure

TestID:

OrderedItemTest2

Input:

"  1.1 here is an ordered sub item"

Expected:

MarkdownElement["Item",
    Association[
        "Type" \[Rule] "Ordered", "IndentationLevel" \[Rule] 1,
        "IndentationType" \[Rule] "Whitespace",
        "Content" \[Rule] "here is an ordered sub item"
    ]
]

Output:

"  1.1 here is an ordered sub item"

Convert current implementation to a paclet

Here's the link to CommonMark .30 tests

An example test:

VerificationTest[
    ImportMarkdown["< test string >"],
    ...
]

I should also note the version being tested, I could do that in the TestID, or have different test files for different CommonMark versions.

Use the markdown fields from these tests as input, and by default set the expected output in the VerificationTest to be whatever the parser returns.

Then go through and verify they all are parsing correctly.

https://spec.commonmark.org/0.30/spec.json

In exporting Markdown I wonder if it makes sense to keep the Markdown in Symbolic form. Converting the Markdown expressions to Styled expressions obfuscates the markdown expressions, or adds fluff to it? I'm considering including a stylesheet, that way on the export mechanism just needs to interface with the symbolic markdown, rather than the 'rendered' expressions.

For the sake of lean files and organization, I think I should move the lexer code to a separate file.

The 3 backtick sequence indicating the start and end of a code block is not parsing correctly. Each pair of backticks is parsing as an InlineCode MarkdownElement.

Support YAML frontmatter. In what form though I'm not sure, as objects, or just styled text?