wikiotics / creoleparser Goto Github PK

Currently, when blog_style_endings are enabled, an explicit line-break (//)
at the end of a line produces 2 <br \> elements. There should only be one.

Hello,

I started using creoleparser some days ago, and I needed to use a
distinct base URL for wiki page links than for images. For instance:
   <a href="/mysite/pages/apage">a page</a>
   <img src="/mysite/files/image.jpg"/>

This is often needed, because wiki pages and images require a completely
different processing:
- wiki pages are generated using the creoleparser;
- images are served statically or with binary modifications such as
 resizing.
Thus, one may want to separate their URLs in different “namespaces”, to
allow the wiki engine to run the appropriate code.

So here is a patch that implements such a distinction, by adding two
arguments wiki_files_base_url and wiki_files_path_func to
creole10_base(). For compatibility, when they are not provided they
default to wiki_links_base_url and wiki_links_path_func.

I hope this patch will be useful.

Regards,

--
Tanguy Ortolo

It would be nice to include all links in the callback
function (not just WikiLinks). This is useful so that
for any given link you can perform some tests and
assign different types of links different classes (CSS stylse)

eg: external links vs. internal wiki links.

--JamesMills (prologic)

The current creole unit tests are a little cumbersome to integrate into
standard testing frameworks/test runners (e.g., for Zope-based projects). 

There are a few options to consider:
 1. rewrite the tests as unittest.TestCase unit tests
 2. rewrite the tests as complete doctest unit tests
 3. a combination of the two, where sensible

At Canonical (on the Landscape team), we tend towards the latter, with a
pretty heavy bias towards unittest tests. We use doctest when we need to
document API usage, for example.

fix version number in __init__.py

[deleted issue]

Creoleparser currently doesn't expose the parse tree in a usable way. This
function would return the output of the fragmentize() function, which is
effectively a parse tree.

Creoleparser, in its default configuration, allows images hosted on external 
servers to be included inline.  There should be a way to disable this behavior, 
and the documentation should be updated accordingly.

Wikipedia has a page that lists a few reasons one may want to disable external 
images at
<http://en.wikipedia.org/wiki/Wikipedia:IMAGE#Inline_linking>

See also the thread on the creoleparser list at 
<http://groups.google.com/group/creoleparser/browse_thread/thread/6bd8593be57ef7
13>

The HTML generated for an image declaration has the form :

<img src="my_image.png" alt="My image" />

But, Firefox does not use the alt attribute and uses the title attribute
instead. For compatibility with both IE and Firefox, the generated HTML
should be :

<img src="my_image.png" alt="My image" title="My image" />

Fix:
In elements.py, line 909 should be (in class: Image, method: _build)
        return bldr.tag.__getattr__(self.tag)(src=link ,alt=alias, title=alias)

Sometimes, users may need to have access to the raw text of a wiki without
all the markup included. The situation that comes to mind is preparing wiki
text for full-text search parsing. 

We should provide a utility function that strips all legal wiki markup from
a give text, for just such use cases.

Probably the easiest way to do this is to define the complete set of tokens
recognized by a dialect and then write some utility functions/methods for
easy "querying" of tokens. Likely, dialect subclasses will take a tokens
parameter and use it to set a tokens attribute. The assigned tokens
object/dictionary/whatever would then be queried using base class methods
such as get_all_tokens, get_all_token_names, get_token, get_token_name, etc.

The basis for this work has been outline as part of ticket #12 (the tokens
part).

What steps will reproduce the problem?
1. Write a macro_func that returns a unicode object.

What is the expected output? What do you see instead?
I expected the unicode object to have creole expanded in it, like it would
if it were a str.  Instead, the unicode object is unexpanded as if it were
in a nowiki.

I'm using Creoleparser 0.4.0.

Replacing each isinstance(foo,str) in elements.py with
isinstance(foo,basestring) seems to fix this.

wiki2html.py contains the code, IRC Meeting 2.wiki is the file that causes
the issue:

Traceback (most recent call last):
  File "./wiki2html.py", line 36, in <module>
    sys.exit(main(sys.argv[1:]))
  File "./wiki2html.py", line 31, in main
    print tmpl.generate(body = creole_parser.generate(file2string(filename)),
  File "/usr/lib/python2.5/site-packages/creoleparser/core.py", line 117,
in generate
    return
bldr.tag(fragmentize(text,self.dialect.parse_order,element_store)).generate()
  File "/usr/lib/python2.5/site-packages/creoleparser/core.py", line 84, in
fragmentize
    frags = fragmentize(text,wiki_elements[1:],element_store)
  File "/usr/lib/python2.5/site-packages/creoleparser/core.py", line 84, in
fragmentize
    frags = fragmentize(text,wiki_elements[1:],element_store)
  File "/usr/lib/python2.5/site-packages/creoleparser/core.py", line 84, in
fragmentize
    frags = fragmentize(text,wiki_elements[1:],element_store)
  File "/usr/lib/python2.5/site-packages/creoleparser/core.py", line 82, in
fragmentize
    frags = wiki_element._process(mo, text, wiki_elements, element_store)
....
  File "/usr/lib/python2.5/site-packages/creoleparser/core.py", line 84, in
fragmentize
    frags = fragmentize(text,wiki_elements[1:],element_store)
  File "/usr/lib/python2.5/site-packages/creoleparser/core.py", line 84, in
fragmentize
    frags = fragmentize(text,wiki_elements[1:],element_store)
  File "/usr/lib/python2.5/site-packages/creoleparser/core.py", line 84, in
fragmentize
    frags = fragmentize(text,wiki_elements[1:],element_store)
  File "/usr/lib/python2.5/site-packages/creoleparser/core.py", line 84, in
fragmentize
    frags = fragmentize(text,wiki_elements[1:],element_store)
RuntimeError: maximum recursion depth exceeded
scons: *** [build/IRC Meeting 2.html] Error 1

After calling creole10_base(), certain arguments that were passed by reference 
may be modified.  Specifically, if any value of interwiki_links_base_urls, 
interwiki_links_path_funcs, or interwiki_links_space_chars is a tuple (denoting 
a different function for interwiki links), then the argument as passed to the 
function will be modified.  If the caller to creole10_base() then uses this 
dictionaries for another purpose (e.g. another, later call to creole10_base()), 
it will behave unexpectedly.

The attached patch fixes this by simply copying these dictionaries before 
performing any operations that may modify them.

Creoleparser should be smart enough to take a multi line url and stitch it
together. The current behavior forces a url to be on a single line.

Ticket for branch: #11

Purpose of code changes on this branch:
1) make the tests unittest-compatible
2) allow standard test-runners to run them
3) provide the same usability as the previous tests

After the review, I'll merge this branch into:
/trunk

Note: I took a look at the review-this-revision feature, and it's pretty
limited. I didn't see a way to compare more than one revision ago. Ideally,
you'd be able to look at (and comment on) the diff from copy-to-branch
(i.e., trunk) and the latest branch revision. But who knows, maybe you can
do that and I just didn't see how :-)

Create parser via:

creoleparser.Creole10(wiki_links_space_char=' ')

And when processing a Interwiki Link the result will still use '_' instead
of ' '. Patch is attached.

Currently, create_dialect accepts as argument wiki_links_class_func, which 
allows the developer to provide a callback function which returns what should 
be the class attribute for the link.  This function works only for internal 
wiki links.  It would be nice to be able to provide another argument, 
interwiki_links_class_funcs, which contains a dictionary of such functions 
which will be used when preparing interwiki links for display.

A macro like this:

{{{
<<mymacro>>
body line 1
body line 2
<</macro>>
}}}

If unhandled (i.e., there is no macro_func supplied or macro_func returns
None), you get this output:

<<mymacro>>body line 1 body line 2 <</mymacro>>

Creoleparser should preserve the linebreaks somehow. 

Following issue #36, headers should have an automatically generated id so users 
can link to section of a page. This would also ease the creation of table of 
contents.

Here's how I would generate ids : take the id of the preceding header and 
concatenate it with the name of the current header. A number could be added at 
the end if this combination was already generated.

I've attached a python file that implement this. I'm pretty new to python so 
this might not be the "python-way" of doing it. Also, I'm not sure if my 
implementation will fit the current design at all. It needs to be called as you 
pass through each header in order.

This is just a place holder to attach my patch:

   * Allow path functions per interwiki link type
   * Base urls are left intact, and path func is called in addition
   * Docstrings modified
   * Test cases included

Use the following source:

'http://www.google.com\\foo'

It will render as:

'<p><a
href="http://www.google.com&lt;&lt;&lt;171154380&gt;&gt;&gt;test">http://www.goo
gle.com&lt;&lt;&lt;171154380&gt;&gt;&gt;foo</a></p>\n'

( http://www.google.com<<<171154380>>> )

This is because the text is transformed using placeholders like this:

1. 'http://www.google.com\\\\test'
2. 'http://www.google.com<<<171154380>>>test'
3. <<<171154412>>>
Note that the raw link element grabbed all the text at step 2, because of
the regexp for raw links that looks basically like this: "http://\S+?"

4. Then at placeholder-expansion time it will only look for '<<<171154412>>>'
5. Expanded to http://www.google.com<<<171154380>>>

I can see two solutions here:
* Either have the placeholder contain a space, eg. ' <<<xxxx>>> ' and then
adapt the regexp to get that whitespace as well, and the process code to
output those spaces (elements.py:146, elements.py:174, core.py:18, core.py:30)

* Modify the RawLink regexp so that elements.py:374 has:
look_ahead = r'(?=([,.?!:;"\']|\*\*|//|<<<(-?\d+?)>>>.*)?(\s|$))' 

This allows to stop the URL content if seeing a placeholder followed by any
text. I'm not sure if i like this solution because it makes the placeholder
foo appears in the element definition. However i don't think other inline
elements are affected because they are generally enclosed in a delimiter
string like ** ** or // // or [[ ]] or {{ }}

I'm going to monkey-patch the code to use method 2. for my project since
it's easy enough to do, but i think maybe method 1 should be implemented in
the lib itself..

Currently, inline no-wiki cannot extend multiple lines. This seems to be an 
unnecessary limitation.

Creoleparser can output unicode by setting encoding=None

What steps will reproduce the problem?
1. Try to parse this markup: [[ɤ]]

What is the expected output? 

<a href="http://www.wikicreole.org/wiki/%C9%A4">ɤ</a>

What do you see instead?
...
  File "C:\Users\steve\Desktop\creoleparser repository 
checkout\trunk\creoleparser\elements.py", line 570, in href
    the_path = urllib.quote(self.page_name(mo))
  File "C:\Python25\lib\urllib.py", line 1205, in quote
    res = map(safe_map.__getitem__, s)
KeyError: u'\u0264'

It's not that hard to write my own convenience function for each project 
but I think the creoletohtml function name can be misleading and it would 
be nice to have an HTML method available for import from the root module.

What steps will reproduce the problem?
1. make a really long bullet list. Mine had about 400 lines total.
2. render

What is the expected output? What do you see instead?
It should generate the proper ul list.

A RuntimeError with the message "maximum recursion depth exceeded"

What version of the product are you using? On what operating system?
Creoleparser-0.6-py2.5.egg on Linux and Mac OS X. Using the creole11_base 
dialect.

Please provide any additional information below.
It appears to be bouncing between core.py line 152 [frags = 
wiki_element._process(mo, text, wiki_elements, element_store, environ)] 
and elements.py lines 124-125 [frags.extend(fragmentize(text[mo.end
():],wiki_elements,
element_store, environ))]

What steps will reproduce the problem?

These links do not work as expected; the "#" gets encoded.

[[#some_id|My Alias]]

[[My Page#some_id|My Alias]]


What is the expected output? 

Expect that the "#" is not encoded.

add "indent_class" and "indent_style" arguments to create_dialect() 

r203 added support for custom base urls, functions, and space chars for images. 
 However, these are currently only supported for images within the wiki, not 
images that are included using an interwiki link.  It would be nice if the user 
of creoleparser could supply tuples as the dictionary values passed to 
interwiki_links_base_urls, interwiki_links_path_funcs, and 
interwiki_links_space_chars as well.

For unknown inline macros, use style="white-space:pre-wrap;" instead of
inserting <br />'s

Hi,

I'm not sure if creoleparser has support for this at all
but I need to be able to have macros preprocess as well as
postprocess.

--JamesMills

What steps will reproduce the problem?
1. Macros with capital letters in their name will not be matched by the
macro regexps, and thus will not be passed to the macro_func.

What is the expected output? What do you see instead?
When rendering a wiki page that has a macro like <<Foo>> with a capital
letter, I expect macro_func to be called; instead, the text <<Foo>> is
rendered in the HTML unchanged.

I'm using 0.4.0.

A simple change to the regexps seems to fix this just fine.  I'm currently
using ([A-Za-z]([A-Za-z-]*[A-Za-z])?) as the expression for the macro name,
though using something like (\w([\w-]*\w)?) would also allow underscores
and digits (and be simpler).

When creating a dialectical variant of dialects.Creole10, one has to
subclass Creale10, super the __init__ and then redefine potentially lots
attributes other than just exactly what one wants to override.

In an effort to make the creation of dialects as painless as possible, I
offer the following API-compatible suggestion:

 1. Create a tokens dictionary where all basic (non-composite) markup will
be defined.
 2. For each attribute that is currently set in Creole10.__init__, create a
separate method for easy overriding.
 3. Ensure that each method that sets up a basic piece of markup uses the
token dictionary.
 4. Rewrite Creole10.__init__ to set attributes by calling each of these
methods.
 5. Ensure that __init__takes a tokens dictionary as a parameter, using the
new creaol10 tokens dictionary as the default.

This will allow the following:

 * When a user wants to simply redefine a single markup token, all they
need to do is import the tokens dictionary, update the key for the token
they want redefined, and instantiate Creole10 with their new dictionary.
 * When a user wants to keep the token but change the way that Creole10
treats that token, all they need to do is subclass Creole10 and override
the method responsible for that treatment.
 * When a user wants easy access to the all tokens defined by a given
dialect, all they need to do is inspect the token dictionary.

This last option is important, as it could provide the means by which wiki
text is stripped of markup.

Preprocessing is only done if context='block'. This could be problematic. 

Change behavior so preprocessing always happens unless explicitly turned off.

inline hash marks (##) currently produce <tt>, which is depreciated in
xhtml 1.1

Change to <code> instead.

What steps will reproduce the problem?
1. List creoleparser in a requirements.txt
2. Run pip install -r requirements.txt
3. Watch the fireworks

setup.py currently imports __version__ from creoleparser/__init__.py, which
imports core, which in turn imports genshi.

This means that creoleparser can't be installed using pip requirements
files, because pip runs egg_info prior to installing anything, which
triggers the above, meaning it bails with genshi not being available.

Instead, __version__ should be moved elsewhere, free of external dependencies.

Using Creoleparser 0.5.1 on Ubuntu 8.10. When using a pre formated block in
a page the generated output does not include empty lines, i.e. when using:

{{{
Hello

World
}}}

The generated output is:

<pre>Hello
World</pre>

Instead of the expected:

<pre>Hello

World</pre>

Add support in the parser and dialect config
to support icons/emoticons.

What steps will reproduce the problem?
1. Make a piece of text with either a macro <<mymacro>>blah<</mymacro>> 
where the innards have two repeated '\n's, e.g.

<<code>>
Hey
there

big
guy
<</code>>
2. Try to parse this through creoleparser
3. Realize that it doesn't see the macro as having a body.

What is the expected output? What do you see instead?
I expect that the entire middle between the macro beginning and end is fed 
in as the body, but only the beginning tag is and it acts as if it has no 
body.

What version of the product are you using? On what operating system?
CreoleParser 0.6.0, using creole11_base. Python 2.5 on Mac OS X 10.5 and 
Ubuntu Linux.

Please provide any additional information below.
This also occurs for pre statements, e.g.
{{{
Hey
there

big
guy
}}}

The following is not possible:
<<fontsize size=5>> some text <</fontsize>>

The example is not very useful but there are cases where attributes are
desirable for block macros.

In my case I have made a math plugin, and would like to be able to chose
between display style(bigger, centered) and inline style without making two
separate macros.

Use style="white-space:pre-wrap"

For my wiki, I want the creole macros can return to be able to contain
other macros.  For example, say the macro '<<Footer>>' returned the
contents of a page that contained '<<center>>This is a footer.<</center>>'.
Another example would be a construct like 'Hello<<ifloggedin>>,
<<username>><</ifloggedin>>!'

One way of doing this, used in the example at
http://creoleparser.googlepages.com/documentation in the include macro, is
to have each macro that allows nesting generate tags from its contents with
a separate parser.  However, this results in stuff in a p tag, and thus
doesn't work for macros used in an inline context, such as the second
example above.

Returning macros in a string (as you would normally do for creole you want
to be expanded) doesn't work because the macros aren't preprocessed (i.e.,
marked with ~s by the core.preprocess function).

My initial attempt at fixing this was to return preprocess(text,dialect)
instead of just returning text.  This mostly worked, except in the case
where a bodyless macro evaluates to creole containing a bodied macro.  For
example, if '<<Footer>>' evaluated to '<<center>>This is a centered
footer.<</center>>', instead of evaluating center with the body text, it
would try to evaluate the open tag by itself with no body text.  This is
because the parser evaluates bodied macros first in the parse order,
followed by bodyless macros.  Thus, by the time the bodyless macro is
parsed, no more parsing of bodied macros will happen.

To fix this, I changed dialects.py at line 146 by changing:
  if use_additions:
            self.parse_order = [self.bodiedmacro,self.macro,...
to:
  if use_additions:
            self.parse_order = [(self.bodiedmacro,self.macro),...
This causes bodied macros and bodyless macros to be parsed "at the same time".

I also needed to change the following section at line 66 in core.py:
    if isinstance(wiki_elements[0],(list,tuple)):
        found_elements = []
        for wiki_element in wiki_elements[0]:
            mo = wiki_element.regexp.search(text)
        if mo:
                found_elements.append((mo.start(),wiki_element,mo))
        if found_elements:
            x,wiki_element,mo = min(found_elements)
        else:
        mo = None
to:
    if isinstance(wiki_elements[0],(list,tuple)):
        found_elements = []
        for wiki_element in wiki_elements[0]:
            mo = wiki_element.regexp.search(text)
        if mo:
                found_elements.append((mo.start(),
                                       wiki_elements[0].index(wiki_element),
                                       wiki_element,mo))
        if found_elements:
            x,q,wiki_element,mo = min(found_elements)
        else:
        mo = None
to make sure that the first element in the tuple was the one used if both
elements matched at the same place (since the unbodied macro regexp would
match wherever a bodied macro matched).

These two changes seem to solve my problem so far, and I haven't noticed
them causing any other changes (though I haven't done any formal testing).

I think macros that evaluate to other macros is a useful feature (Wikipedia
uses it all the time), and this seems like the simplest way to implement
it.  Whether macros should be evaluated in the creole returned by macros by
default is another question, but if not I think there should at least be a
supported way of explicitly causing them to be evaluated.

Another way of handling this would be to make some way of having the macro
parse the creole itself that would work in an inline context, but that
seems like it would be more complicated to implement.

I'm using 0.4.0.

It would be nice if the parser would accept an option to use blog-like
newline breaks (a simple \n instead of \\).

A first attempt at doing this is the following

class MyDialect(Creole10)
    def __init__(self, *args, **kwargs):
        super(MyDialect, self).__init__(self, *args, **kwargs)

        # Auto-convert newlines to <br> tags
        br = self.br
        br.regexp = re.compile(esc_neg_look + "(\n|\\\\\\\\)",
re.DOTALL) 


See also google group thread:
http://groups.google.com/group/creoleparser/browse_thread/thread/de502bb7068966a
3

Hi,

Python2.6 does not allow to rise string exceptions, but there are three of them 
in the creoleparser source.

I'm using
http://svn.debian.org/wsvn/python-modules/packages/creoleparser/trunk/debian/pat
ches/raise_string_fix.dpatch
to fix this issue in Debian.

Cheers,

Bernd

It should be easy to add WikiWord or other linking functionality (e.g., issue 
32, or revision 234). This could be implemented using ideas from:

http://bryanalsdorf.blogspot.com/2010/07/automatically-process-text-in.html

Currently, in order to suppress a regular (non-bodied) macro from being
enclosed in paragraph marks, it must be on a line by itself, with no
leading whitespace, and be surrounded by empty lines. With the new behavior
the "surrounded empty lines" will no longer be needed.

Many creoleparser markup blocks need terminal newlines to work. In order to 
compensate for this, a newline have to be added to input text.

What steps will reproduce the problem?
1. make a really long bullet list. Mine had about 400 lines total.
2. render

What is the expected output? What do you see instead?
It should generate the proper ul list.

A RuntimeError with the message "maximum recursion depth exceeded"

What version of the product are you using? On what operating system?
Creoleparser-0.6-py2.5.egg on Linux and Mac OS X. Using the creole11_base 
dialect.

Please provide any additional information below.
It appears to be bouncing between core.py line 152 [frags = 
wiki_element._process(mo, text, wiki_elements, element_store, environ)] 
and elements.py lines 124-125 [frags.extend(fragmentize(text[mo.end
():],wiki_elements,
element_store, environ))]

Currently, there is a bug in creoleparser that prevents the use of the
"text" genshi rendering method.

To demonstrate this, do the following:

from creoleparser.core import Parser
from creoleparser.dialects import Creole10
parser = Parser(dialect=Creole10(), method="text")
parser("= Heading\nAnd some **bold** text")

What should be displayed on stdout is the following:

'Heading\nAnd some bold text\n'

However, the following traceback occurs instead:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/home/oubiwann/lab/Canonical/Landscape/team/oubiwann/landscape-server/293686-kb
-wiki/src/canonical/lib/creoleparser/core.py",
line 184, in __call__
    return self.render(text,element_store,context)
  File
"/home/oubiwann/lab/Canonical/Landscape/team/oubiwann/landscape-server/293686-kb
-wiki/src/canonical/lib/creoleparser/core.py",
line 160, in render
    encoding=self.encoding,**kwargs)
  File "/var/lib/python-support/python2.5/genshi/core.py", line 153, in render
    generator = self.serialize(method=method, **kwargs)
  File "/var/lib/python-support/python2.5/genshi/core.py", line 189, in
serialize
    return get_serializer(method, **kwargs)(_ensure(self))
  File "/var/lib/python-support/python2.5/genshi/output.py", line 70, in
get_serializer
    return method(**kwargs)
TypeError: default __new__ takes no parameters

What steps will reproduce the problem?

This markup:

*# item 1

Should render a list with a # as the first character, however, the # is
being dropped.