That is, tests written in actual .hebi files.
We can test the parser just by the Hissp output, but we should compile the macros and run the helper functions.

We do need some minimal bootstrap machinery working just to write the tests though. I can maybe get away with using the basic Lissp macros, since they're included.

There's no __package__ at compile time. This might require a Hissp-level fix. But runtime-only relative imports also need to work.

How should these look in Hegibo?
Given the Python examples,

if a < b:
    print('less')
    return a  # Assume we're in a function.
elif a > b:
    print('more')
    return b
elif a == b:
    print('equal')
    return a
else:
    print 'nan'

and

if case == x:
    print(x, a)
else:
    print(b)

Naiively, one might think

if: (case == x)
    print: x, a
else:
    print: b

This is gramatically valid Hebigo, but then the implied brackets would look like this

{if: (case == x)
    {print: x, a}}
{else:
    {print: b}}

The if and else are completely separate forms. Not going to work. They need a surrounding form.

I'm renaming :keywords to "control words". "Keyword" means something else in Python. In the Hissp documentation I was calling them "key symbols", I think I'll change that as well for consistency.

:controlwords are used by some of the little DSLs created by macros, sometimes to label what the next argument is for (e.g. :as in from:/import: and try:.)

But there are cases when I want blocks to have these labels attached. I think it's more natural to just allow the control word to take arguments like symbols do than to use :foo pass: or pass: :foo.

Obviously this makes no sense at the top level. Python identifiers can't have :. (That doesn't stop you from putting a non-identifier string in a __dict__ though.) So I don't think Hissp would compile it into anything sensible. But inside a macro, the code would get rewritten anyway, and this can make sense, for example

try:
    !begin:
        print: "It's dangerous!"
        something_risky: thing
    :except: LikelyProblemError
        print: "Oops!"
        fix_it:
    :except: Exception :as ex
        do_something: ex
    :else:
        print: "Hooray!"
        thing
    :finally:
        .close: thing

or

if: (a<b)
    :then:
        print: "less"
    :elif: (a>b)
        print: "more"
    :elif: (a==b)
        print: "equal"
    :else:
        print: "nan"

Because of automatic qualification of reserved words and of normal unqualified symbols in masks, using symbols here is kind of more difficult than it should be. Control word wouldn't have this issue. And because control words can have special characters, the !mask: macro can look a little more lispy and use :,:/:,@: instead of _:/__: like before.

On the downside, we're using even more colons than before. Unfortunately, Hissp had to special case :keywords at the compiler level to make them work, and Python used colon to start blocks for completely unreleated reasons. Maybe I could change the :keyword symbol to $keyword or something. The code examples in docstrings also look funny if the IDE thinks it's supposed to be reStructuredText, because having a word beginning and ending with a colon (like :foo:) means something in that language.

Lissp has template quotes, but Hebigo doesn't. I think Scheme's quasiquote reader macros just expand to normal macros, so I think this should be possible.

Hebigo so far has no munging like Lissp does, so unary hotwords must be valid Python identifiers. `: wouldn't work. quasiquote: is maybe too long. Even template: is not much better. I'm thinking mask:.

Devs who already know Python are Hebigo's primary audience. Should be in the docs. We'll need docs first #28.

Hissp already has this capability for Lissp.

I've already done most of the work in Drython. I think I'll just copy the relevant bits over, but this may become a separate package later.

These will be special cased in the Hebigo reader because they're Python reserved words, e.g. the symbol def would munge as hebi.basic.._macro_.def_.

def is an especially important one. I'm thinking roughly Scheme semantics:

So in Hebigo,

def: key value

assigns a simple global.

But

def: name: arg1 arg2 : kw1 default1
    "docstring"
    body...

defines a function. It should compile to Hissp like:

('hebi.basic.._macro_.def_', ('name', 'arg1', 'arg2', ':', 'kw1', 'default1'),
    ('quote', 'docstring', {'str': True}),
    body...

Hebigo will automatically expand Python reserved words to qualified macros. But we'll want some more "reserved" words than this. We also don't want them to conflict with normal Python identifiers. Thus far, Hebigo does not allow special characters in its symbols. So we can special case symbols that start with ! for this purpose.

At minimum, I think we'll need

• !let for locals
• !mask for quasiquotes
• !require for macro imports

Perhaps we should think of these as Hebigo's macro "builtins", in which case, we might want a lot more of them than this.

3.10 got a new statement type for pattern matching. Too bad it wasn't an expression or we could just put it in brackets. But Lisp macros are easily powerful enough to do this. Like the other statement macros, this should follow Python's form and semantics as closely as reasonably possible.

Even after it updates the __name__ and __qualname__, functions made with the def macro are still getting called "lambda" in the tracebacks. It seems to be part of the code object itself.

With Python 3.8's new types.CodeType.replace(), the code object itself might be easy to update with a new name.

But, if we use 3.8 features, that means that Hebigo's def macro would only work on 3.8 or later.

While parsing a bracketed expression, we could recursively switch back to the base Hebigo parser when encountering a macro character that Python doesn't use, like !, until we finish the next Hebigo expression, then place the result of compiling that back into the Python string we're building. This would effectively be a builtin reader macro. For example,

def: fibonacci: n
  :@ functools..lru_cache: None  # Qualified identifier in decorator.
  if: (n <= 1)
    :then: n
    :else: (!fibonacci:(n - 1) + !fibonacci:(n - 2))

Maybe a bad example, since Python's expression syntax can handle this part fine.

    :else: (fibonacci(n - 1) + fibonacci(n - 2))

But suppose we needed a macro.

(!macro:spam + !macro:eggs)

We'd currently have to do something like

!let: :,: s e
  :be hebi.bootstrap..entuple: macro:spam macro:eggs
  (s + e)

or

operator..add: macro:spam macro:eggs

On the other hand, we might want to encourage using hotword expressions instead of Python expressions, because it's much easier to write macros to work with those.

Hebigo's lexer relies on Python to determine when a bracketed expression is complete. It reads up to the first matching ending bracket, and asks Python to parse it. Brackets can be nested and escaped in things like string literals, meaning the first one encountered might not be the right one, so on certain SyntaxErrors it reads up to the next matching bracket and tries again. Other SyntaxErrors are unrecoverable and should propagate out. Unfortunately, the only way to distinguish these cases is by the error message, and those changed in 3.10.

I could include the new messages, but this seems just as brittle, and I feel like I could easily miss one. Another option would be to assume all SyntaxErrors are recoverable. This is probably good enough in the REPL, but it will result in poorer feedback in case of unrecoverable errors, as the kernel will simply keep asking for more lines, rather than pointing at the problem. In a .hebi file, an unrecoverable error would result in the remainder of the file lexing as one bracketed expression.

Some notes from a private chat with @brandonwillard who seems to think this would help with tooling.

Hebigo uses Python's expression syntax (but not statements), so that would have to be copied for a complete grammar. Note that Python expressions in Hebigo must be simple literals or "bracketed" somehow. Parentheses work for anything, but if it already has {}, [], or valid Python quotation marks, that's also enough. (Including things like f"", b"", etc.)

Hebigo translates very directly to Hissp tuples.

In Lisp, the head of the list is the function/macro, and the rest are arguments. The head is almost always a symbol.

In Hebigo, you make a "list" (tuple) with a hotword: a symbol ending in a colon, like foo:.
The hotword is the "head" of the "list". The expressions following that are the arguments. The layout determines where the "list" ends, much like Python.

I designed the layout rules to make Hebigo easy to edit with a minimal editor. Hotwords can be either unary or polyary. Unary has no whitespace after the colon and attaches directly to its single argument, like a tag. The hotword becomes the "head" of the tuple. A polyary tuple extends to the end of the line, or in the case of the first polyary hotword of the line, it also includes the indented block below it. (As a matter of style, simple function invocations would use either arguments on the same line as the hotword or in the block beneath it, but not both. Macros, on the other hand, may use both.)

There's also an "empty" hotword ::. It uses its first argument as its "head". You can use this to make an empty tuple, or a tuple that has a complex expression as its head (instead of a simple symbol.) You can also use :: stylistically in cases when the tuple is just grouping things and isn't an invocation where the head is special.

Unary hotwords fill in for tag macros and such. (There are no reader macros in Hebigo.) A common use would be quote:

There are a lot of examples in the tests. Any grammar would, at minimum, have to pass those tests. But I wasn't really testing the Python expression grammar there. Any grammar would pretty much have to be machine-checked before I'd trust it to be accurate. Lark is an option. But indented blocks aren't even context-free without some preprocessing. This doesn't look easy.

If you forget a colon, but indent a block more you should get an error.

Recent example from tests:

test_default_strs lambda self:
    self.assertEqual:
        ['ab', 22, 33]
        !let
            :=: :strs: a b c
                :default: a ('a'+'b')
            {'b':22,'c':33}
            [a, b, c]

Note that lambda and the !let were intended as hotwords and should have colons. The former is valid, since self: gets the block. (Maybe a good reason to spell it as pass:self instead.) The latter should be an error (but isn't), because it indents the block of self.assertEqual: more.

How should the class: macro look/work? I want Hebigo to look and feel familiar to Pythonistas. But at the same time, Hissp is targeting a functional subset of Python to reduce incidental complexity. It can't be simpler if it works exactly like Python.

I installed hebigo this way without any errors:
pipx install --verbose git+https://github.com/gilch/hebigo.git
but when running hebi I get this error message:

; hebi                                                                     
/usr/local/bin/python: Error while finding module specification for 'hebi.kernel' (ModuleNotFoundError: No module named 'hebi')
[ZMQTerminalIPythonApp] CRITICAL | Could not find existing kernel connection file kernel-6944.json

pipx is the most comfortable way to experiment with commandline tools, so that's why it would be great, if hebigo would support it. Thanks

See #2.

Jupyter makes this weirdly awkward in a venv. It wants global kernelspecs. Install that if you want, but that seems like overkill when trying out a REPL in a venv. You can start a kernel without installing it, and connect to the most recently started one. But getting these to connect up reliably and automatically in one command is surprisingly difficult. The current solution works, but not reliably. Sometimes it starts the kernel but fails to connect. Sometimes the first error crashes the console. Can't seem to reproduce that one.

Quitting is also awkward. I'm worried about accidentally leaving kernels running in the background, using up resources, and possibly stealing connections later. It also seems to be possible to exit the kernel without quitting the console, which leaves you with a broken console.

No showstoppers, but no easy fixes, and it just doesn't work as well as it should.

We might need a REPL first. #2. Lissp has doctests.

The empty hotword is currently a double colon, like ::. A triple colon is not that rare, like quote::: ... (for '(...)), which gets harder to read, and a quintuple colon, like quote::::: ... (for '((...))) is probably possible. Triple is maybe OK, but quintuple is nuts.

The natural empty hotword would have been :, but that's already a :keyword due to an unfortunate historical collision. And Hissp already uses it to separate single from paired parameters in lambdas, and single from paired arguments in normal calls, so we can't just make it a special case.

Now that we have the !macros, !: would be an option. quote:!:!: ... looks much less bad. I could maybe even implement it as a macro defined in readerless mode, which would simplify the parser.

Hissp's target Python subset only has expressions. Python's comprehension syntax (being expressions) already works in theory.

Loop-equivalents (for/while) are fairly easy to implement using higher-order functions, though while gets awkward without assignment statements. (It may be less of a problem when we get the walrus in Python 3.8.)

Technically, yield is considered an expression in Python, but it does something weird to its surrounding function: It makes it return a generator instead, so they compose differently. Hebigo control-statement macros expand to thunks to be able to delay, repeat, or avoid evaluation, so any yield expression in an if or loop (the normal case) would transform the wrong function's return: the inner thunk, instead of the surrounding definition. Higher-order functions would now require a yield from to propagate this behavior. Even if we could add that everywhere it's needed, how do you tell if the thunk has a yield in it?

One possible solution is to have a parallel version of each control statement that has the yield from, and then the user has to pick the right one (or uses a macro that does it for them). I think this could get awkward fast. And then what about new user macros?

This lack of natural composability makes me think that yield expressions are the wrong approach.

Drython's solution was to create a background thread to save the execution state. This works, but has greater overhead than a native yield would. Perhaps that is acceptable, since using higher-order functions everywhere has a higher overhead to begin with. Performance was never really the point of Hissp. I'm also not sure if I ever got them garbage collected properly. Threaded code is notoriously hard to test due to nondeterministic behavior (race conditions).

Scheme's call/cc or Haskell's IO monad also look promising. They seem a lot more naturally composable than Python's yield. Unfortunately, I don't think I understand them well enough to tell, much less implement yield this way.

Another option might be to not support yield at all. Given itertools, genexprs, and statement-equivalent macros that work in lambdas, you'll probably rarely need them. But not having them means adding friction to the use of Python libraries. There are cases even in the standard library that require yield, like @contextlib.contextmanager.

A class with the right dunder methods can be a generator, though implementing them seems more difficult generally. This goes for awaitables as well.

Python allows a semicolon for joining statements on one line, so there's precedent. It's mostly considered bad style in production code, but is useful for shell commands, although the inability to join certain statement types limits its usefulness.

for i in range(3): print(i); print(i*i)  # Python allows this.
print('hi'); for i in range(3): print(i)  # SyntaxError
for i in range(3): for j in range(3): print(i, j)  # SyntaxError
class Foo(object): def: foo(self, x): print(x)  # SyntaxError

Hebigo can do some of these already.

for: i :in range:3 print:i print: (i*i)
print:'hi' for: i :in range:3 print:i
for: i :in range:3 for j :in range:3 print: i j

But the class doesn't quite work:

class: Foo:object def: foo: self x print:x  # print in params
class: Foo:object def: foo: self x
  print:x  # print in class

But with a semicolon acting like a Lisp closing parenthesis,

class: Foo:object def: foo: self x; print:x

It's almost never used at the end of a line in Python (although this is technically allowed) and it would tend to be a long train in a Lisp, so I'd rather not allow it in Hebigo. But even internal-only, you can get doubled semicolons for reasonable use cases:

class: Foo:object
 def: foo: self x y
  print: x y
 def: bar: self a b
  print: a b

# one-liner version
class: Foo: object; def: foo: self x y; print: x y;; def: bar: self a b; print: a b

Maybe this is OK for shell commands, but it does require more careful thinking than the usual indentation-based notation.
If you think of x:/; as parentheses, the Lispiness becomes more apparent. Both forms at once to show where the double came from:

class:
  Foo:
    object;
  def:
    foo:
      self x y;
    print:
      x y;;
  def:
   bar:
     self a b;
   print:
     a b;;;

You can see a train of three at the end. This wouldn't be allowed of course, but it does show the structure.

It could also make normal code more compact, but I find this less readable:

class: Foo:object
 def: foo: self x y; print: x y
 def: bar: self a b; print: a b

So, as in Python, I think they should be considered bad style in source code. I can't think of a case where it reads better, but it would make any expression possible as a one-liner, a capability that Python lacks but has some compromise support for. You can theoretically do all of your one-liners in Lissp anyway, so I'm not sure if this is worth it.

https://www.reddit.com/r/whitespaceLisp/comments/n68qj2/gilchhebigo_an_indentationbased_skin_for_hissp/

How should it look in Hebigo? In Scheme it's like

(let ((foo 1)
      (bar 2))
  (frobnicate foo bar))

The same structure wrritten in Hebigo would be

!let:
    pass:
        foo: 1
        bar: 2
    frobnicate: foo bar

That doesn't look too bad, but I'd prefer to avoid using pass:. We could use a meaningful control word with arguments instead.

!let:
    :def:
        foo: 1
        bar: 2
    frobnicate: foo bar

Arc Lisp's let only binds one name.

!let: foo 1
    frobnicate: foo 2

This looks a lot cleaner, until you have to start nesting them.

!let: foo 1
    !let: bar 2
        frobnicate: foo bar

Still not that bad when there's only two, but indents can add up fast. I'd like to flatten them for the same reason we have elif in Python. Arc has with for this.

With multiple variables, there's also let* and letrec variants to consider. letrec looks hard without mutation. Clojure doesn't have it, but has letfn instead. But at that point you might just want a class.

Clojure has a powerful recursive destructuring syntax built into its let. Even with one binding pair, you could still bind multiple variables this way. Python has sequence destructuring only, but requires statements to do it. An example like

(x1, y1), (x2, y2) = rect
frobnicate(x1, x2, y1, y2)

Might look like

!let:
    pass:
        pass: x1 y1
        pass: x2 y2
    rect
    frobnicate: x1 x2 y1 y2

It's not so pretty without the brackets. With more meaningful control words,

!let: rect :as
    :,: :,: x1 y1
        :,: x2 y2
    frobnicate: x1 x2 y1 y2

More Clojurelike options

!let:
    foo :as
    :,: :,: x1 y1 :& rest
        :,: x2 y2
        :as all
    bar :as
    :=: "spam" food1  # associative destructuring
        "eggs" food2
    frobnicate: ...

Basing one on IPython might be a good idea. We'd also get a Jupyter kernel out of it.