jg-rp / liquid Goto Github PK

Python Liquid currently raises a NoSuchFilterFunc exception upon rendering an output statement that uses an unknown filter.

By default, the reference implementation seems to silently ignore unknown filters, passing over them if there are more filters later in the chain. Ruby Liquid also offers a strict_filters mode, which captures filter errors for the caller to explicitly check. In strict_filters mode, a single undefined filter causes the whole output statement to return nil.

Python Liquid should probably follow suit. Although without the need to explicitly check for errors. Let the exceptions raise.

Add HTML auto-escape functionality to Python Liquid.

In keeping with the reference implementation, auto-escape would be disabled by default. We don't want to assume that everyone is using Liquid for generating HTML or XML, or that all Liquid generated HTML contains user edited content, but common use cases like Django and Flask web apps really do need an auto-escaping option.

If markupsafe is installed and the autoescape argument to Environment or Template is True, context variables will be escaped before output, unless explicitly marked as "safe".

Provisionally, it looks like we'll need:

• an escaped version of liquid.builtin.statement.StatementNode
• a "safe" version of liquid.expression.StringLiteral
• a "safe" version of liquid.builtin.tags.capture_tag.CaptureNode
• a method of flagging filters as "safe" or "needs autoescape"

Unknowns:

• If we have a chain of filters, do we escape after every filter, or just the last one if an unsafe filter exists anywhere in the chain?
• Is a non-standard safe filter or block tag required?

The sort filter (and probably sort_natural too) is inconsistent with the reference implementation.

Trying to sort an array of hashes without providing a key/property should raise an error. Python Liquid doesn't have an issue comparing hashes.

puts Liquid::Template.parse("{{ a | sort }}").render(
  { "a" => [{"title": "foo"}, {"title": "bar"}, {"heading": "apple"}] })
# Liquid error: comparison of Hash with Hash failed

Also, when a sort key/property is given and items in the target array support subscripts but don't have the key property, the sort comparison block will always return 1. This has the effect of reversing the target array if all items don't contain the key property. Python Liquid currently raises a liquid.exceptions.FilterArgumentError.

puts Liquid::Template.parse("{{ a | sort: 'title' | join: ',' }}").render(
  { "a" => ["Z","b", "a", "B", "C", "A"] })
# A,C,B,a,b,Z

The template object built-in to Python Liquid is not part of "core" Liquid. It is Shopify specific.

It's tempting to disable the template object by default, and allow it to be enabled via an argument to the Environment constructor. This, however, risks setting a bad precedent.

Maybe some pre/post render hooks/signals are in order. Then we can move this kind of functionality to liquid-extra, along side non standard tags and filters.

The reference implementation allows for else and elsif blocks to appear in unless blocks. Like this.

{% unless true %}
  foo
{% else %}
  bar
{% endunless %}

Where the expected output is bar. Python Liquid currently raises a LiquidSyntaxError: unexpected tag 'else', on line 3

From the recently updated reference documentation..

"To start a loop from where the last loop using the same iterator left off, pass the special word continue."

Template

<!-- if array = [1,2,3,4,5,6] -->
{% for item in array limit: 3 %}
  {{ item }}
{% endfor %}
{% for item in array limit: 3 offset: continue %}
  {{ item }}
{% endfor %}

Output

1 2 3
4 5 6

Python Liquid does not currently support the special continue keyword as a for loop offset.

Extend the outward facing API to allow for instantiating a Template directly, without the need for an Environment.

A suitable Environment would be created automatically and used for subsequent templates created without an explicit environment.

When comparing nil, null, blank and empty, some of the corner cases are not quite what I had expected.

In the reference implementation, where foo is the empty string, the following expressions all evaluate to true.

"" == ""
"" == foo
"" != nil
"" != null
"" == empty
"" == blank
blank == ""
blank == foo
blank == nil
blank == null
blank != empty
blank != blank
empty == ""
empty == foo
empty != nil
empty != null
empty != empty
empty != blank
nil != ""
nil != foo
nil == nil
nil == null
nil != empty
nil == blank
null != ""
null != foo
null == nil
null == null
null != empty
null == blank
foo == ""
foo == foo
foo != nil
foo != null
foo == empty
foo == blank

Note, for example, that blank != blank but blank == nil.

Python Liquid does not stay true to all these "rules".

In Ruby Liquid, forloop.parentloop is available inside nested {% for .. %} tags, Giving access to the parent forloop object.

Python Liquid's forloop object does not currently have a parentloop property.

Python Liquid follows the reference implementation's default behaviour of returning nil for any variables that are undefined. No warnings are given and no exceptions are raised.

Template

Hello, {{ noshuchthing }}.

Output

Hello, .

Whereas the reference implementation offers a strict_variables mode and the render! method, Python Liquid does not currently offer an alternative to the default mode.

Note that, in the reference implementation, strict_variables is not affected by the error_mode.

I'm inclined to follow Jinja2's example, by offering a choice of Undefined type.

Remove the @abstractmethod decorator from liquid.loaders.BaseLoader.get_source so that custom loaders can implement get_source_async without having to implement get_source too.

I'm thinking of a database loader, where most async database drivers/packages don't expose a synchronous API. Meaning we'd need to either raise a NotImplementedError inside an otherwise empty get_source, or use multiple database drivers/packages.

Offer an Undefined type that returns debug information when rendered.

The Django Liquid template backend will use this to provide contextual line information.

I want to extend Python Liquid's template loader API to optionally include template meta data. Said meta data will be added to a template's globals mapping, making it available to each render context.

This gives custom loaders the chance to emulate "front matter" style templates, similar to that found in Jekyll, or include extra data from a database of templates, for example.

By default, meta data loaded with a template will take priority over variables from existing Environment or Template globals, but not keyword arguments passed to .render(). Foreseeing situations where that order of priority does not make sense, we'll implement this in BoundTemplate.make_globals, making it simple enough to subclass BoundTemplate to change this behaviour.

When using offset: continue and looping over the same sequence three or more times, a for loop's limit can be calculated incorrectly.

Example template

{% for item in (1..6) limit: 2 %}a{{ item }} {% endfor -%}
{% for item in (1..6) limit: 2 offset: continue %}b{{ item }} {% endfor -%}
{% for item in (1..6) offset: continue %}c{{ item }} {% endfor %}

Expected output

a1 a2 b3 b4 c5 c6 

Actual output

a1 a2 b3 b4 c3 c4 c5 c6 

The built-in first and last properties should follow the semantics of the first and last filters. That is they should not work on strings.

This should render as an empty string or raise an exception in strict mode. Python Liquid currently renders f.

{% assign x = "foo" %}{{ x.first }}

Hey there,

I am trying to use your package to parse some liquid template strings without a "real" environment. When creating a new Environemnt like

from liquid import Environment
env = Environment()

I get the error message from the title:
ModuleNotFoundError: No module named 'liquid.utils'

I installed the library using pipenv: pipenv install python-liquid. When checking the virtualenv created by pipenv, the utils module indeed is not installed:

site-pacakges $ ls liquid
ast.py  context.py  environment.py  exceptions.py  expression.py  filter.py  __init__.py  lex.py  loaders.py  mode.py  parse.py  __pycache__  stream.py  tag.py  template.py  token.py

I assume the setup.py does not explicitly include the submodules needed in its packages (i.e. it does not include liquid.utils).

Is my assumption correct or did I install the package wrong?

When the built-in slice filter is given a negative start index and a length, Python liquid can calculate the wrong stop index, potentially returning an empty sequence when it shouldn't.

{{ "Liquid" | slice: -3, 3 }}

Expected output

uid

Actual output is an empty string.

Unlike liquid.context.extend, liquid.context.copy does not guard against recursive use.

extend, as used by the built-in include tag, will raise a ContextDepthError when MAX_CONTEXT_DEPTH is reached. copy, as used by the built-in render tag, should do the same.

Example include:

>>> from liquid import Environment
>>> from liquid.loaders import DictLoader
>>> loader = DictLoader({"some": r"{% include 'some' %}"})
>>> env = Environment(loader=loader)
>>> template = env.get_template("some")
>>> template.render()
.
.
ContextDepthError: maximum context depth reached, possible recursive include, on line 1

If we swap include for render in the example above, we get a RecursionError: maximum recursion depth exceeded while calling a Python object.

Liquid supports accessing array items from the end using negative indexes. For example.

{%- assign some_numbers = "1,2,3,4,5" | split: "," -%}
{{ some_numbers[-1] -}}

Which is expected to render 5. Python Liquid currently raises a LiquidSyntaxError: invalid identifier, found negative.

The reference implementation allows range literals to be assigned to variables for later use.

Template

{% assign nums = (1..5) -%}
{% for x in nums limit: 2 %}{{ x }}{% endfor -%}
{% for x in nums %}{{ x }}{% endfor -%}

Output

1212345

Python Liquid currently raises a LiquidSyntaxError: unexpected '(', on line 1

The special blank keyword has not been implemented.

Currently, when blank appears in an expression, it will be treated as any other identifier. If that identifier is not in scope, it will default to nil.

The intended behaviour is that blank be equivalent to an empty string. Making it possible to write conditions like this.

{% unless settings.heading == blank %}
    <h1>{{ settings.heading }}</h1>
{% endunless %}

Both template and expression lexing functions are currently defined in liquid.lex, and parsers for template tags and tag expressions are bundled into liquid.parse. Moreover, all tag expressions are parsed through liquid.parse.ExpressionParser.parse_expression(), which handles liquid identifiers, loops and boolean expressions.

For reasons of easier maintenance and potential improvements in performance, I intend to move and refactor each of the expression lexers into their own package, along with a specialised parser and independent TokenStream (independent from the top-level token stream).

Built-in tags will transition to use these new parsers now, via liquid.Environment.parse_*_expression_value functions. Existing tokenize* functions and the ExpressionParser will be maintained until at least Python Liquid version 2.0, which is quite some time away, for those who use them in custom tags.

Some possible optimisations that can be realised include:

• Lexers that yield tuples rather than NamedTuples. Benchmarks show the former to be faster.
• Lexers that recognise identifiers with bracketed indexes and string literals. Doing this with regular expressions in the lexer will be much faster than stepping through a token stream in the parser.
• Don't do unnecessary infix operator parsing when handling loop or output expressions. They don't have any infix operators.
• Don't do unnecessary token precedence look-ups when handling expression that don't have any precedence rules. Only boolean expression use precedence rules.
• Remove unnecessary prefix parsing. No built-in tag expression uses prefix operators. Negative numbers can be handled during tokenization.

The reference implementation has added support for "drops" that can resolve to primitive values when used in some Liquid expressions.

For example, if a class defines a to_liquid_value method, like this

class SomeDrop:
    def __init__(self, val):
        self.val = val

    def to_liquid_value():
        return self.val

It could be used to access a Liquid array like this

from liquid import Template

source = "{{ greetings[foo] }}, World."
template = Template(source)
print(template.render(foo=SomeDrop(1), greetings=["Hello", "Goodbye"]))
# Goodbye, World.

There might be some other subtleties that require investigation. Like having a drop that resolves to an integer work well with maths filters.

When using Liquid's special size property on a collection, you get the number of items in that collection, equivalent to calling len() on the object in Python.

data:

{
  "some_list": [1,2,3],
  "some_dict": {"a": 1, "b": 2},
}

template:

{{ some_list.size }}
{{ some_dict.size }}

output

3
2

If an object already has a size property or key, the reference implementation will return its value rather than the object's length. Python Liquid always returns the object's length.

data:

{
  "some_list": [1,2,3],
  "some_dict": {"a": 1, "b": 2, "size": 42},
}

template:

{{ some_list.size }}
{{ some_dict.size }}

expected output

3
42

The same goes for .first and .last, although .last should not work on a dictionary (or Mapping).

Ruby Liquid's implementation of the strip_html filter has some specific rules for handling HTML comments, <script> blocks and <style> blocks. Python Liquid's built-in strip_html filter does not implement these rules.

For example, {{ "<style type='text/css'>foo bar</style>" | strip_html }} will render an empty string with Ruby Liquid, but foo bar with Python Liquid.

I am wondering if this library has the same security guarantees as the Ruby version. If I run an untrusted template from a client, is it dangerous? Is security a design goal in addition to the API compatibility?

The date filter should accept 'today' as well as 'now' as its left value.

Currently Python Liquid raises an exception if 'today' is used.

Error: filter 'date': unexpected error: Unknown string format: today, on line 1

Python liquid will successfully parse a chained identifier containing a bracketed index or identifier, followed immediately by another identifier, with no separating dot.

{{ products[0]title }}

Ruby Liquid will raise a Liquid::SyntaxError when parsing the above template, instead expecting {{ products[0].title }}. Python Liquid will accept either without error.

Similarly, Python Liquid will handle {{ products.[0].title }}, while Ruby Liquid will again raise a Liquid::SyntaxError.

The default template cache created with every environment has a "least recently used" policy with a capacity of 300 templates.

Add a cache_size argument to the liquid.Environment constructor for controlling the default template cache capacity.

The built-in truncatewords filter should have a default word count of 15. The word count is currently a required argument in Python Liquid.

The built-in truncate filter should have a default length of 50. Length is currently a required argument in Python Liquid.

When exposing templating to end users, which is one of the primary use cases for Liquid, it is vital for the stability of existing deployments that minor and patch updates to the template engine don't change template rendering behaviour.

At the same time we want to offer bug fixes, including behavioural changes, to new deployments or to developers that don't rely on built-in tags and filters.

This dilemma should be familiar to anyone who's glanced at Shopify's Liquid issue tracker. Here's an example of one bug that the Shopify developers can't fix because it could break existing templates maintained by their vast user base.

To solve this problem I propose we add tag and filter version pinning to Python Liquid. This means:

• Python Liquid version 1.2.0 pins the current behaviour of all tags and filters. This behaviour will not change until version 2.
• Any bug fixes involving filter and/or tag behavioural changes will be opt-in at Environment creation time.
• Developers can choose which version of any built-in tag or filter to pin.

Come Python Liquid version 2, we'll reverse this, making new Environments default to the latest version of each tag and filter. Pinning older version will be done at the developer's discretion.

The only unknown at this stage is how we might go about testing multiple tag and filter implementations, and maintain a test suite for each.

Use markupsafe.soft_str instead of, for instance ..

if not isinstance(arg, str):
    arg = str(arg)

Note that soft_str was renamed from soft_unicode at some point and speed ups were added at the same time.

When the cycle tag is given a name, Python Liquid will use that name and all other arguments to distinguish one cycle from another. Ruby Liquid will disregard all other arguments when given a name. For example.

{% cycle a: 1, 2, 3 %}
{% cycle a: "x", "y", "z" %}
{% cycle a: 1, 2, 3 %}

Ruby Liquid Output:

1
y
3

Python Liquid Output:

1
x
2

when expressions should accept one or more literals or identifiers, separated by commas.

Template

{% assign a = "foo" %}
{% case a %}
{% when "" %}
no a
{% when "foo", "bar" %}
a is foo or bar
{% endcase %}

Expected output

a is foo or bar

Python Liquid currently raises a LiquidSyntaxError as it expects exactly one literal or identifier.

Also, if multiple when expressions match the case, including multiple matches from a comma separated list, the when block will be rendered multiple times. One for each match.

Template

{% assign a = "foo" %}
{% assign b = "foo" %}
{% case a %}
{% when b %}
when b
{% when "foo", b %}
when "foo" or b
{% endcase %}

Expected output

when b
when "foo" or b
when "foo" or b

Python Liquid currently breaks after the first match.

The page on creating custom filters at https://liquid.readthedocs.io/en/latest/user/filters.html is out of date.

It shows examples of inheriting from the depreciated Filter class instead of using the preferable filter decorators.

Update the built-in FileSystemLoader to accept either a single search path string or a sequence of paths to search for templates.

Provide an example loader and include (or render) tag subclass that uses a context variable to give the loader a prefix or scope to search. Much like the "section" tag found in Shopify's Liquid.

The following tasks need to be completed before releasing Python Liquid version 1.0.

• Complete the documentation (issue #9).
• Complete type annotations for Mypy in strict mode.
• Fix pylint/flake8 warnings or update pylint/flake8 config.
• Add a Mypy GitHub workflow/job.
• Add a coverage GitHub workflow/job.
• Add a linting GitHub workflow/job

The built-in captialize filter currently uses Python's str.captialize internally. Which makes "the first character have upper case and the rest lower case".

The reference implementation's capitalize filter is roughly equivalent to this..

def captitalize(val):
    if val:
        return val.replace(val[0], val[0].upper(), 1)
    return val

Where characters after the first are not forced to lower case.

I'm proposing adding asynchronous support to Python Liquid (using async/await syntax) by way of the following additional methods.

• Template.render_async
• Template.render_with_context_async
• Environment.get_template_async

Drops (classes that mimic Liquid primitive values) can implement __getitem_async__, which is assumed to be a coroutine function. When defined, Liquid will use (and await) __getitem_async__ instead of __getitem__ when doing Liquid attribute and array access, and hash lookups. This will allow drops to perform asynchronous lazy loading of objects.

Initially async support will be targeted at only some Python Liquid use cases. Specifically those that ..

1. Load templates frequently. Either because templates change frequently or because there are more templates than can reasonably held in a cache.
2. Don't do blocking IO or time consuming processing in custom tags or filters.
3. Want to do asynchronous lazy object loading with custom drops.

Liquid will suppress blocks that only contain whitespace characters. Python Liquid currently checks for whitespace only blocks after rendering the block. Whereas the reference implementation will output whitespace only string literals if they are in an output statement or echo tag.

Template

{% assign array = '1,2,3' | split: ',' -%}
{% for value in array -%}
{{ value -}}
{% if forloop.first %} {{ '   ' }} {% endif -%}
{% endfor %}

Expected output

1     23

Actual output

123

In addition to to_liquid_value (allows drops to behave like a Liquid primitive value inside some expressions), Ruby Liquid uses to_number to allow drops to work with filters that expect numbers as arguments.

The same effect can be achieved in Python Liquid using an __int__ and/or __float__ method.

Python Liquid's documentation for drop objects needs to be expanded to cover this use case.

Add filters for encoding and decoding strings to and from Base64. As per the reference implementation.

New filters are:

• base64_encode
• base64_decode
• base64_url_safe_encode
• base64_url_safe_decode

Write some documentation.

TODO:

• Introduction
• Rendering
• Loading templates
• Context
• Strictness
• Auto-escape
• Tags
• Filters
• Drops
  • __liquid__
  • __html__
• API reference
  • High level
  • Low level
• Hosting

The reference implementation includes a tag called ifchanged that has not been implemented in Python Liquid.

Usage appears to be something like this.

Template

{% assign list = "1,3,2,1,3,1,2" | split: "," | sort %}
{% for item in list -%}
    {%- ifchanged %} {{ item }}{% endifchanged -%}
{%- endfor %}

Output

1 2 3

When given an additional argument, the built-in uniq filter will use the value of that argument as a key into each item in the sequence to dedupe. Python Liquid currently raises an exception if any of a sequence's items don't have that property/key. The reference implementation does not.

data:

{
  "things": [
    {"title": "foo", "name": "a"},
    {"title": "foo", "name": "b"},
    {"title": "bar", "name": "c"},
    {"heading": "bar", "name": "c"},
    {"heading": "baz", "name": "d"},
   ]
},

template:

{% assign uniq_things = things | uniq: 'title' %}
{% for obj in uniq_things %}
  {% for x in obj %}
    x[0]: x[1]
  {% endfor %}

{% endfor %}

expected output

title: foo
name: a

title: bar
name: c

heading: bar
name: c

Many filters built in to Liquid will automatically convert a string representation of a number to an integer or float as needed.

When converting integers, Ruby Liquid uses Ruby's String.to_i method, which will disregard trailing non-digit characters. In the following example, '7,42' is converted to 7

template:

{{ 3.14 | plus: '7,42' }}
{{ '123abcdef45' | plus: '1,,,,..!@qwerty' }}

output

10.14
124

Python Liquid currently falls back to 0 for any string that can't be converted to an integer in its entirety. As is the case in Ruby Liquid for strings without leading digits.

This does not apply to parsing of integer literals, only converting strings to integers (not floats) inside filters.

Python Liquid currently maintains an isolated namespace for named counters. This namespace is not accessible from output statements ({{ some_name }}), and {% increment %} and {% decrement %} do not touch context global or local variables.

template:

{% increment foo %} {% increment foo %} {% increment foo %} {{ foo }}!

current output:

0 1 2 !

With the same template, the reference implementation outputs..

0 1 2 3!

Not only is foo in scope outside of increment and decrement, but it increments/decrements the value after rendering it.

If foo is assigned with {% assign %} or {% capture %}, foo will exist in the local scope independently of any named counter.

template:

{% assign foo = 5 %}{% increment foo %} {% increment foo %} {% increment foo %} {{ foo }}!

output

0 1 2 5!

If foo is defined as a global variable, increment and decrement use it. Where foo is equal to 5..

template:

{% increment foo %} {% increment foo %} {% increment foo %} {{ foo }}!

output:

5 6 7 8!

We need an async version of liquid.template.BoundTemplate.is_up_to_date to use when checking the template cache from liquid.Environment.get_template_async.

An async database loader will probably want to make an async query as part of is_up_to_date.