greghendershott / markdown Goto Github PK

-> (parse-markdown "1. [A][a]\n\n[a]: google.com\n\n2. item2")
'((ol () (li () (a ((href "google.com")) "A"))) (ol () (li () "item2")))

Same example on babelmark

Looks like babelmark results are mixed. I expected it to parse as one list, but not sure which is correct.

[Reported by @stchang. Moving to its own issue.]

Getting a parse error for two headers separated by more than 3 lines.

> (parse-markdown "# abc\n\n\n\n## def")
parsack.rkt:345:0: parse ERROR: at <string>:5:1:10
unexpected: "non-empty input"
  expected: "end-of-file"
> (parse-markdown "# \n\n\n\n## ")
parsack.rkt:345:0: parse ERROR: at <string>:5:1:7
unexpected: "non-empty input"
  expected: "end-of-file"
> (parse-markdown "# \n\n\n\n\n## ")
parsack.rkt:345:0: parse ERROR: at <string>:6:1:8
unexpected: "non-empty input"
  expected: "end-of-file"
> (parse-markdown "# \n\n\n## ")
'((h1 ((id ""))) (p () " ##"))

Images links are not parsed correctly if the path contains space

![hello](hel lo.jpg)

will not produce an image sexpr.

The commonmark spec says that the path to the image should conform to the same rules as a regular link. If you want a space in a regular link you should encapsulate it with <>, which would make the above example

![hello](<hel lo.jpg>)

which looks stupid, and also does not work with racket markdown.

Conforming to some kind of spec is of course nice, but wouldn't doing the expected thing (rendering the image anyway) also be nice? GitHub markdown does what I would expect (both examples work), pandoc as well.

Best regards
Linus

Example of what should work:

![caption][url]

[url]: http://path/to/some/image

I thought I was handling this correctly but it doesn't seem to be working. For now, just quickly logging this to look at the code later.

## Heading **with** _formatting_

should be formatted:

Heading with formatting

but instead is

Heading *_with_* formatting

Example:

Here is a footnote[^1].

...

[^1]: Here is the footnote definition.

Note: If this is in Gruber's spec, I don't see it. But I noticed it in the markdown mode for Emacs.

One spec: http://rephrase.net/box/word/footnotes/syntax/
Another: http://pythonhosted.org/Markdown/extensions/footnotes.html

I've been trying to improve the performance of the markdown parser through parsack improvements but haven't had much luck so far. I think it's just the bracketed-style of the grammar leads to too much backtracking, which my parser doesn't handle well.

That did give me an idea to profile some examples (ie your tests) to see if I could improve the ordering on any of the <or> options, to cut down on the backtracking.

For example the following changes produces a decent speedup of perf-test.rkt:

1. Flip the choices in $normal-char.
2. Remove the singleton <or> in source-url.
3. In $inline, drop $smart-punctuation down to above $code.
4. In $inline, drop the 4+ ... down to above $footnote-ref.

Obvious caveats:

1. I don't know how representative perf-tests.rkt is.
2. I know sometimes the order of the <or> choices matters, when the choices are not disjoint, so I don't know if I've actually messed something up. But all the tests do pass.
3. I do occasionally get a perf warning from random-test.rkt. I guess this is unsurprising since random-test emphasizes special chars?

Some numbers:

Before: non-strict: 3200, strict: 3340
After: non-strict: 2598, strict: 2547

Have you played around with the <or> ordering any?

Parse:

<p>X <a>Y</a> Z</p>

Expected:

(p () "X " (a () "Y") " Z")

Actual:

Note lack of spaces:

(p () "X" (a () "Y") "Z")

which as HTML is:

<p>X<a>Y</a>Z</p>

footnote has been surported, then how about toc ?

<pre>...</pre> blocks need to be detected early on -- as blocks, and before any line-break consolidation that is acceptable or desirable for intra-block elements. (In other words, the fix is not to detect all HTML elements earlier on.)

[^def]: Definition of footnote def.


aaaaa

Fail to parse because $footnode-def don't eat the two #\newline (don't know if it's responsability).

diff --git a/markdown/parse.rkt b/markdown/parse.rkt
index 30f2b85..60d29ee 100644
--- a/markdown/parse.rkt
+++ b/markdown/parse.rkt
@@ -805,7 +805,7 @@
             (optional $indent)
             (xs <- (sepBy $raw-lines
                           (try (>> $blank-line $indent))))
-            (optional $blank-line)
+            (many $blank-line)
             (return
              (begin
                (on-footnote-def! label (string-join xs "\n"))

Solve the problem.

I can't nest code in bold, italic in bold, bold in italic, etc. It causes the outer style to fail to be interpreted.

See greghendershott/frog#59

For example it should work as here on GitHub:

```
yo
```
ABC

is:

yo

ABC

I had a code block using triple-backticks followed by a <!-- more --> marker, but unless I had three blank lines between them it failed to be processed correctly.

It would be truly awesome if this could generate scribble data types, and maybe it wouldn't be so hard, given the xexpr you're already generating.

A link with an empty URI like this:

[label][]

Is a reference link. It must be defined elsewhere in the Markdown:

[label]: www.example.com/

Now that we allow arbitrary markdown in link labels (see #5), if a reference link label is something that's not a simple string, for example the following as a result of adding smart quotes (see #13 ):

[some 'quotes' here][]
...
[some 'quotes' here]: www.example.com/

Then the reference is unresolved, because the reference link definition is still trying to match on the original string "some 'quotes' here", as opposed to what it is after running it through intra-block, namely '("some " lsquo "quotes" rsquo " here").

See: https://github.com/trentm/python-markdown2/wiki/Testing-Notes

Discovered via randomized testing:

#lang racket

(require markdown)

(define input @~a{___*  '   "[<br />

<br />        ipsum'<div>"(<    ")_<]<div>[**<br /> **lorem*>***__```]```***"ipsum
<br />'*)***ipsum___    <br />***    *lorem___*ipsum**<br /></div><div> _</div>    <ipsumipsum &__
ipsum_[   <div>__<div>&ipsum(


(***<div>lorem`]  ___ </div>&lorem("**[ipsum__&   *

loremlorem*<br />lorem]``')___lorem&___''   ``">



```lorem<div>**    </div>[ lorem`]ipsum``<br />**&


__lorem   <br />lorem`*lorem  </div>lorem_ 'lorem```
})

(parse-markdown input)

Doesn't terminate (for at least 30 seconds).

I was trying to do:

[\[\[Stuff\]\]](stuff.html)

but since (as Issue #8 says) I can't escape brackets, I tried using the literal HTML entities [ and ] instead, but this caused the link to fail to be correctly interpreted. I worked around it with literal HTML, as always.

http://daringfireball.net/projects/markdown/syntax#link

According to that, clearly in the [label][ref] case, with an explicit id ref.

Is an implicit ref ID from the label, e.g. [some _arbitrary_ markdown **here**][] supposed to match case-insensitive, too? I guess so.

IIRC the issue here was a line that starts with -- which is supposed to be an en dash. This got ignored and passed through as two dashes instead.

I'm trying to set up random testing of markdown inputs as well. I still have some kinks to work out but it found one case that I thought I would bring up.

A trailing space at the end of the input gets parsed as br but as " " when it's not at the end. Is this expected?

> (parse-markdown "![UoFEK](pFivY) ")
'((img ((src "pFivY") (alt "UoFEK"))) (br ()))
> (parse-markdown "![UoFEK](pFivY) [ToFEwaE](VYiYppJd)")
'((img ((src "pFivY") (alt "UoFEK"))) " " (a ((href "VYiYppJd")) "ToFEwaE"))

The second example has an extra space at the end and therefore doesn't parse properly. Is this the intended behavior?

$ racket
Welcome to Racket v6.0.0.3.
-> (require markdown)
-> (parse-markdown "```racket\n(define x 10)\n```")
'((pre ((class "brush: racket")) (code () "(define x 10)")))
-> (parse-markdown "```racket\n(define x 10)\n``` ")
'((p () (code () "racket\n(define x 10)")))

I wrote a long blog post using Frog and left an unclosed HTML tag towards the front of the file (<TODO>). When trying to convert the file to HTML the parser hung. It sent my CPU to 100% and didn't finish after 5 min. The memory usage was oscillating around 200MB so it appears to be doing a lot of backtracking.

This sounds related to issue #43 but wanted to provide an test case I came across.

A list of the form

    * blah blah
    * blah blah
    * blah blah

crashes the parser, whereas if the list items have no spaces before them it works fine.

This:

<img src="img/yunocoros.jpg"/> The topic of coroutines (or
fibers, or continuations) for JavaScript comes up from time to time

does not preserve the <img src="/img/yunocoros.jpg"/> HTML.

The problem seems to be with "self-closing" (if that's the correct terminology) tags.

<img src="foo" />

does not work, whereas

<img src="foo"></img>

does work

cf greghendershott/frog#20

Example:

Jay McCarthy posted about a macro to do a [`case` with `break` in Racket](http://jeapostrophe.github.io/2013-06-24-cas-cad--post.html).

Creates an invalid xexpr:

'(p () "Jay McCarthy posted about a macro to do a " (a () ((href "http://jeapostrophe.github.io/2013-06-24-cas-cad" (span mdash) "post.html")) (code () "case") " with " (code () "break") " in Racket") ".")

Note the (span mdaash) in the href attribute, from the -- in the URI.

I believe the parser does not handle Windows-style line breaks, ie "\r\n", properly. For example, all the tests that rely on test.md fail in Windows.

Here is another example:

> (parse-markdown "[test][1]\n\n[1]:http://test.com \"test-title\"")
'((p () (a ((href "http://test.com") (title "test-title")) "test")))
> (parse-markdown "[test][1]\r\n\r\n[1]:http://test.com \"test-title\"")
Reference link not defined: (linkref "1")
'((a ((href "")) "test") "\r \r [1]:http://test.com " ldquo "test-title" rdquo)

When the markdown parser parses the text below, it takes a long time to do it:

% time .../m2h try-md.md 
Doing try-md.md
'#(#<void>)
.../m2h try-md.md  6.68s user 0.17s system 100% cpu 6.836 total
%

(the m2h script does little more than parse the markdown and then dump it to HTML).

Now, the text here is obviously not very good markdown – it's just a list of filenames – but it is just text, and I would have expected the parser to broadly cope with it, even if the result isn't pretty. The case this is a reduction of consisted of about double this number of lines, and I thought the parser had crashed. Is there perhaps something O(n^2) happening here?

Now, one response to ‘the parser is slow when I give it rubbish markdown’ is ‘well, don't do that, then’. That is perfectly fair. But if there's some way of getting the parser to give up gracefully, that would be nice.

font-util-1.3.0_1   Create an index of X font files in a directory
fontconfig-2.11.0_1,1 XML-based font configuration API for X Windows
fontsproto-2.1.2    Fonts extension headers
freetype2-2.5.2     Free and portable TrueType font rendering engine
gdk-pixbuf2-2.28.2  Graphic library for GTK+
gettext-0.18.3.1    GNU gettext package
git-1.8.5.2         Distributed source code management tool
glib-2.36.3_1       Some useful routines of C programming (current stable versi
gmake-3.82_1        GNU version of 'make' utility
gnome_subr-1.0      Common startup and shutdown subroutines used by GNOME scrip
gnomehier-3.0       A utility port that creates the GNOME directory tree
gobject-introspection-1.36.0_2 Generate interface introspection data for GObject libraries
graphite2-1.2.4     Rendering capabilities for complex non-Roman writing system
gtk-update-icon-cache-2.24.22 Gtk-update-icon-cache utility from the Gtk+ toolkit
gtk3-3.8.8          Gimp Toolkit for X11 GUI (current stable version)
harfbuzz-0.9.25     OpenType text shaping engine
hicolor-icon-theme-0.12 A high-color icon theme shell from the FreeDesktop project
icu-50.1.2          International Components for Unicode (from IBM)
inputproto-2.3      Input extension headers
intltool-0.50.2     Tools to internationalize various kinds of data files
jasper-1.900.1_12   An implementation of the codec specified in the JPEG-2000 s
jbigkit-1.6         Lossless compression for bi-level images such as scanned pa
jpeg-8_4            IJG's jpeg compression utilities
kbproto-1.0.6       KB extension headers
lcms2-2.5           Accurate, fast, and small-footprint color management engine
libICE-1.0.8,1      Inter Client Exchange library for X11
libSM-1.2.2,1       Session Management library for X11
libX11-1.6.2,1      X11 library
libXau-1.0.8        Authentication Protocol library for X11
libXcomposite-0.4.4,1 X Composite extension library
libXcursor-1.1.14   X client-side cursor loading library
libXdamage-1.1.4    X Damage extension library
libXdmcp-1.1.1      X Display Manager Control Protocol library
libXext-1.3.2,1     X11 Extension library
libXfixes-5.0.1     X Fixes extension library
libXfont-1.4.7,1    X font library
libXft-2.3.1        Client-sided font API for X applications
libXi-1.7.2,1       X Input extension library
libXinerama-1.1.3,1 X11 Xinerama library
libXrandr-1.4.2     X Resize and Rotate extension library
libXrender-0.9.8    X Render extension library
libXt-1.1.4,1       X Toolkit library
libXtst-1.2.2       X Test extension
libcheck-0.9.11     Unit test framework for C
libgcrypt-1.5.3     General purpose crypto library based on code used in GnuPG
libgpg-error-1.12   Common error values for all GnuPG components
libiconv-1.14_1     A character set conversion library
libpthread-stubs-0.3_4 This library provides weak aliases for pthread functions
libtool-2.4.2_2     Generic shared library support script
libxcb-1.9.3        The X protocol C-language Binding (XCB) library
libxml2-2.8.0_3     XML parser library for GNOME
libxslt-1.1.28_1    The XSLT C library for GNOME

Haskell lambda function does not generate <code> correctly, it works on Mou markdown editor and Github.

This is Haskell lambda `(\_ -> ...)` code.

Result:
This is Haskell lambda (_ -> ...) code.

Github result:
This is Haskell lambda (\_ -> ...) code.

I read the code, when I put code before escape in function intra-block, it worked fine :).

See what happens to the async attribute.

➜  markdown git:(master) racket markdown/main.rkt
<script async class="speakerdeck-embed" data-id="f0b571b0759a0131f0bd026a5a2b7ed1" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script>
<!DOCTYPE html>^D
<html>
 <head>
  <meta charset="utf-8" /></head>
 <body>
  <script async="async" class="speakerdeck-embed" data-id="f0b571b0759a0131f0bd026a5a2b7ed1" data-ratio="1.33333333333333" src="//speakerdeck.com/assets/embed.js"></script></body></html>%

If you type something like **fr**ozen bl**og** then I think you should get frozen blog: you currently get something with embedded asterisks instead (this is using markdown in frog of course). I don't know what the markdown standard, as far as there is one, says about this.

I think this used to work (at least, when I wrote that markup originally I assume I checked the output!).

This is a non-major issue.

See original report and discussion: greghendershott/frog#106

cf greghendershott/frog#41

Unlicensed software is nonfree software... could we get a COPYING? :)

cf greghendershott/frog#25

cf greghendershott/frog#33

\n**Bold** blah blah.\n results in bullet list instead of bold.

Likewise, \n1.23 blah blah.\n results in number list instead of number.

In both cases, the issue is that a #\space should be required after *, -, +, or 1. otherwise it is not a list.

In the sense that text between \[ and \] or between \( and \) is used literally, not parsed as markdown.

See greghendershott/frog#129 (comment)

Because \ already has a meaning in markdown -- for example \[ means a literal [, not e.g. part of markdown syntax for a link -- this may need to be \\[ and so on.

Originally submitted by @chipotle as mbutterick/pollen#17, but seems to also belong here. This Markdown snippet:

"Reductive. Yes."

Is correctly parsed with its smart quotes like this:

<p>&ldquo;Reductive. Yes.&rdquo;</p>

But when inner single quotes are added:

"'Reductive.' Yes."

Then the parser ignores the outer double quotes:

<p>"&lsquo;Reductive.&rsquo; Yes."</p>

I believe markdown allows escaping square brackets with a backslash but this doesn't work.

Footnote definitions should appear in numerical order, but currently they are appearing in the order they are defined in the Markdown text.

Another way to put it: Footnotes definitions should appear in the order of their usage not of their definition.

Example:

A usage[^foo] and another[^bar].

[^bar]: Bar note.

[^foo]: Foo note.

The foo note will be number 1 and bar will be number 2. The footnotes should appear in that order.

Obviously someone can work around this by reordering the footnote definitions, but it should work correctly especially when using the label variant.

I've tried nesting images in links either with:

[![foo](foo.jpg)](foo.html)

or

[<img src="foo.jpg"/>](foo.html)

and both cause the outer brackets to fail to be treated as a link. I can work around with literal HTML.

A code block that isn't preceded by a blank line fails to be interpreted correctly.

Three dashes are treated as an em dash in other markdown parsers, but they're ignored.

(parse-markdown "```racket\n'(foo)\n```")
;; => '((pre ((class "brush: racket")) (code () "'(foo)"))) 
;; good

(parse-markdown "```racket\n`(foo)\n```")
;; => '((p () (code () "racket\n`(foo)")))
;; bad

First glance, I don't understand why. $verbatim/fenced is using $any-line which should not be affected by the ```. Somehow it is getting parsed as an inline $code, instead.

Specific case first:

The Markdown library currently generates (h1 () ...) for Markdown ==== headers, and (h2 () ...) for ---- headers. For my application I'd prefer these to be (h2 () ...) and (h3 () ...) respectively. That's easy to fix, since I can just walk the tree adjusting them post-parse. However it would be neater if I could ask the parser to do that for me.

I see that this overlaps with the discussion of tree-walking, and its costs, in pull request #48, so might relate to that.

Looking at $setext-heading/para/plain in parse.rkt, I see that this h1/h2 interpretation is implemented by a (match ...) expression, so that looks like a natural location for a parameter, but you'll have a better idea than me of how expensive that would be.

There's a potential more general point, in that there are other places where it looks natural for a user to impose a different interpretation on the parse – functions $_emph and $_strong – but those cases are more marginal, and it might be that the sort of configurability I'm suggesting above is not worth generalising.

The xexpr-drop-empty-attributes parameter in xml/xexpr controls whether empty attribute lists are omitted from X-expressions.

parse-markdown makes X-expressions, but does not observe this parameter. Can it? Should it?

(parameterize ([xexpr-drop-empty-attributes #t])
  (parse-markdown "I am _emph_ and I am **strong**."))

> '((p () "I am " (em () "emph") " and I am " (strong () "strong") "."))

;; vs. '((p "I am " (em "emph") " and I am " (strong "strong") "."))