tmontes / ppytty Goto Github PK

Objective:

• Avoid direct yields from Task.run.
• Use yield from / await API wrapping functions / methods.

Benefits:

• More natural and readable Task code.
• Easier to implement "user space" argument validation, defaults, etc.

Possibilities:

• Create an kernel.api module with one function per trap.
• Add methods to Task corresponding to kernel traps.

General idea:

• Tasks can ask the kernel to spawn an operating system process.
• Kernel will require that process to be associated to a window.
• Such window must be previously created by the task and passed to the "spawn process" trap.
• Kernel will create a PTY and automatically handle all process I/O via the PTY and the Window.
• Keyboard input, from the kernel Terminal, will go to the process if the Window itself "has focus".

More:

• On Task termination (reaching end of run, or stopped by a parent), the kernel will (try hard to) stop the sub-process, destroy the PTY and Window (already accounted for). Decided otherwise: unneeded complexity here + risks of stopping unknown processes. Logging a warning message instead.
• Task resets need to be considered (the simplest case is: do nothing, sorry!) Makes no sense.
• There should be a kernel trap for the task to "wait" for the spawned process termination (pretty much mimicking the UNIX wait syscall).

More thoughts (copied from later comment to be accounted in issue progress):

• There could be a way of having a task feed data to a process' input, via the PTY. (see #100)
• There could be a way of having a task monitor a process' output, again via the PTY. (see #101)
• Should there be a trap to "destroy" a spawned process? Or should we go for a more low-level (but more controllable) "send signal" trap?

More thoughts (copied from later comment to be accounted in issue progress):

• There could be a way of supporting out of band bidirectional task <-> process communication. (see #102)
  • Needs to be other than stdin/out/err based (given those are handled by the associated terminal).
  • Maybe pipes could be used.
  • Will we need new traps for that or can existing ones be reused, like message-send / message-wait?

IMPORTANT:

• Factor out all real I/O calls (like os.write/read, fcntl.fcntl/ioctl) to the kernel hw.py module. (see #103).
  • Needs thinking, otherwise, tests running under the "no i/o" test cases fail to "wake" the select via the "child terminated" pipe write.

Identified in #77:

• Would allow a given keyboard reader to know whether some previously unread data came back or not.

Possibility:

• Reading keyboard returns (<tag>, <data>): I/O obtained data is tagged with None.
• Unreading allows tagging which will be returned on the associated read.

Rough Idea:

• With relative window positions/sizes in #10, maybe the kernel could update all window sizes (if relative) and then somehow notify Window owning Tasks that their sizes/positions have changed.
• More: Maybe Window objects can fire an optional callback in those cases: Tasks wanting to support Window resizes would supply one.

(as a continuation of #41)

Ideas:

• Bar, Area, Line and X-Y charts.
• Should work with both static and dynamic data (sourced from messages).

Status quo:

• Windows are rendered on top of other windows in a "fully opaque mode".

Idea:

• Window.render could take an optional alpha_bg defaulting to 1.0.
• If alpha_bg is less than that, the window background would be rendered as an interpolation of its color and the color of whatever is beneath it (the terminal background and/or other windows).

Challenges:

• Same as #114 regarding color interpolation.
• Rendering will need access to the "destination render buffer" to read the background from there.

Notes:

• Window movements leave whatever is/was under the window now visible: maybe just a bunch of nothing.
• This will require some re-rendering.

Ideas:

• Simple approach: re-render everything.
  • Needs either the window-render trap to be told that or the kernel to somehow remember rendered window position, detecting such changes.
• An optimised approach will only re-render the changed bits.
  • Probably needs to be line oriented.
  • Only cleaning up a sub-set of the the kernel Terminal lines and fully rendering Windows overlapping them.

Hand tested scenario:

• Two process + their windows.
• /bin/bash and sys.executable for a Python REPL.

Tested systems:

• macOS Sierra's Terminal, Python 3.6.6, bash 3.2.57.
• raspbian stretch, via ssh from macOS Sierra's terminal, Python 3.6.5, bash 4.4.12.
• Ubuntu 18.04, Gnome Terminal, Python 3.6.5, bash 4.4.19.

Test procedure:

• Try CTRL-C. (fixed in PR #111)
  • Expected behavior: interrupts current line, maybe prints something, new blank prompt below.
• Type spaces and then backspace. (fixed in PR #112)
  • Expected: Cursor moves right as spaces are types and then moves left as they're backspaced.
• Type word word word word the edit the line with cursor movement (emacs and vi keybindings). . (fixed in PR #112)
  • Cursor moves as expected.
• Send a SIGINT from the bash window to the Python REPL. (did nothing, suspect fixed from PR #112)
  • Expected: KeyboardInterrupt should be displayed. Focus and visible should still be in the bash shell!

Test results matrix:

[1] Cursor position is actually tracked and the next "visible" input is be properly positioned as well as the cursor, which is also correctly updated.

Interesting... Needs more investigation.

Identified while working on #5.

Real calls: os.read, os.write, os.close, os.pipe, signal.signal, fcntl.*, and maybe more.

Note:

• Needs thinking, otherwise, the tests running under the "no i/o" test cases fail to wake up the select system call via the "child terminated" pipe write.

Original idea in #5.

Sub-issue of #36.

Objective:

• Currently blocking traps -- sleep, read-key and task-wait -- should no longer block.
• They will return a request id opaque thing.
• Once completed, the kernel will message-send the task a message tuple (request id, trap result).
• The kernel sent messages will have a task argument of None.

With this, instead of:

def task():
    yield ('sleep', 5)    # Blocks for 5 seconds

...the following code should be used:

def task():
    request_id = yield ('sleep', 5)            # Does not block
    sender, message = yield ('message-wait',)  # This blocks until we get a message
    assert sender is None
    msg_request_id, sleep_result = message
    assert msg_request_id == request_id
    # discard sleep_result, not useful

Pros:

• A task can now "block" on multiple things simultaneously.
  Cons:
• This is a lot of code.

Possible message-wait Improvement:

• It could accept some kind of optional argument(s) such that instead of waiting/returning the first message in the task's INBOX, it would return the first matching message instead.
• Matching should support:
  • Waiting for a given trap's request id, as return by the kernel.
  • Being generic enough (but simple) to be used by general inter-task messaging.

• Processes attached to Windows that resize (somehow), should be delivered a SIGWINCH signal.
• More, the PTY may need to have a termios.TIOCSWINSZupdate to properly reflect the new terminal geometry.

It should have two sub-packages:

• kernel
• lib

The last one may, in turn, have other sub-packages -- we'll see about that once we get there.

Summary:

• Traps like task-wait and message-wait return, among other things, a reference to a task.
• The returned value is, however, not necessarily the same that was used in task-spawn.

Motive:

• The kernel will call the passed in task object if it is callable (assuming it is a generator function) which is then used in those traps results.

Failing example:

def child():
    yield ('sleep',)

def parent():
    yield ('task-spawn', child)
    completed_task, success, result = yield ('task-wait',)
    # This fails because completed_task is actually the generator object obtained by calling 
    assert completed_task is childchild

Possible solutions:

• Have the kernel require passed in tasks (to top-level run/task-spawn) to be a generator object or alike.
• Have the kernel track what it was passed vs. what it uses internally.

Let's sleep on this for a while.

From #5 and #104.

Current limitation:

• Cursor mostly not visible.

How it should be:

• When no process windows are focused, cursor should not be visible.
• If a process window is focused, its cursor should be visible (unless that process has hidden it!).

Random thoughts:

• On running, the kernel would create and initialise a default terminal, associated to stdin/stdout.
• A new (set of?) kernel traps/apis would allow terminal management:
  • terminal-create would create and initialise a new terminal.
  • With no arguments, the TTY associated to stdin/stdout is assumed.
  • If passed an argument, probably a TTY dev file, it would identify the new terminal.
  • If a matching terminal exists, no creation/initialisation is done; the existing terminal object is returned.
  • Not sure we would need/want a terminal-destroy... But implementation should be simple.
• The direct-print and direct-clear traps would become methods on the terminal object.
• Windows would be associated to specific terminals (defaulting to the default terminal).
• Maybe keyboard input traps like read-key could also be associated to a terminal.
• Additional terminals could be passed to the ppytty entry point script such that they would be automatically created and initialised, being ready to to use by driven tasks/presentations.

More notes:

• We probably won't be getting SIGWINCH signals from "secondary" terminals.
• Maybe a different, cleaner approach will be needed.

Possible uses:

• Selecting between dark/light background for a "presentation task" to use.
• Passing in information about external terminals (see #76).
• Asking the "presentation task" to dump itself as a PDF.

Related to #7, somehow.

Notes:

• The underlying pyte.Screen needs to be resized.
• Re-rendering is trivial when the Window grows in both dimensions, not so much otherwise.

• Nice.
• Supports multiple, sequential runs, even if a run does not leave the state clean.

This scheduling limitation should now be easy to address, after the keyboard input revamp in #79.

Current status:

• Highly chaotic, more pseudo-manual-test oriented that useful, guiding examples.

Wishlist:

• Have a nice set of lib using examples.
  • Crazy idea: single slide with two widgets: editor on top, Python REPL on bottom. REPL runs the code in the editor (pretty much mimicking the running behaviour of the Mu editor).
• Have a comprehensive set of kernel api using examples.

Status quo:

• Windows are rendered into their parent with a 1:1 row/col mapping, offset by their left/top position (in parent).

Crazy idea:

• Add support for "render mapping".
• One "render map" would map a window region (row-range/col-range) to a particular parent left/top offset.
• By default windows will have a single "render map" behaving like the status quo, above.
• Task callable API (Window method?) would allow changing such "render maps": adding, removing, changing their "source region" and their "target left/top".

What this would allow:

• Crazy effects like visually slitting and animating a fully working PTY process.
• Other animations for common Windows (like bringing together two halves, from left/right).

Thoughts on implementation:

• The Window.render method should not be very difficult to update.
• The window_render trap will probably be much more complex.

Sub-issue of #36.

Notes:

• If the top task sends a message to its parent, a warning/error will be logged by the kernel.
• message-wait does not have a timeout argument; it will be supported via a different mechanism (see #36 and #46).

Status quo:

• Windows render their contents "as-is". Let's call it "fully opaque".

Idea:

• Window.render could take an alpha_fg argument, defaulting to 1.0, meaning behave as status quo.
• With smaller alpha_fg, the foreground colours would be "interpolated" to the background.

Challenges:

• Existing code assumes output TTY supports 256 colors (not more, not less).
• Interpolating grays is easy as they sit between 0 (black), 232 (nearly black) and 255 (white).
• Interpolating non-grays is not as easy/smooth; AFAICT:
  • Colours 0-15 are palette based: no way to ensure they match a given RGB triplet.
  • Colours 16-231 may have "standard" RGB values, but resolution is very low: 6 levels/channel.

Possible compromises:

• Just alpha blend foreground colors when both foreground and background are in the "gray scale range".
• May need to be told what the background is, if window has an unspecified background.

Notes on having state as a module:

• Simple way to get global mutable state (do we want that?).
• Otherwise we would probably need to:
  • Either pass it around...
  • ...or build the scheduler (+ trap handlers + ...) under a common class with state as an attribute.

Wishlist:

• We'll need to be able to "reset" it such that multiple scheduler runs can be tested independently.
• The name state is not that great...

Original idea in #5.

Notes from there:

• Needs to be other than stdin/out/err based.
• Maybe pipes could be used.
• Will we need new traps for that or can existing ones be reused, like message-send / message-wait?

Original idea in #5.

...or the exception object, if the top task raises one.

It currently goes completely wild!

The implementation of #7 in PR #90 has the following limitation:

• Once a window is moved, it must be rendered before any other.
• Otherwise, the rendering strategy may result in wrong visual results.

Motive:

• The strategy does not consider that other windows might have moved since their last render!

One possibility:

• Make use of the dump-state trap.
• If the kernel is running under some kind of test mode, that trap could return a data structure representing the full kernel state.

Notes:

• We may need a special test terminal to drive kernel I/O tests.
• We'll get to that! :)

(that's it, nothing else to say)

From #5 and #104.

Other than cursor visibility (once #106 is fixed), the user has no way of knowing where terminal keyboard input will be directed to.

This needs a solution: UI and technical.

...leave that to main, as a default.

Thoughts:

• Priorities are unintuitive and lead to bad design (they were initially put in place to handle concurrent input at different hierarchical levels; now that we have inter-task communication they're no longer necessary).
• Input buffering should allow for:
  • Cleaner separation of loop code.
  • Better separation of loop vs trap handling code (see put-key trap implementation).
  • (guess) ...improving the loop such that always runnable tasks don't starve waiting/reading tasks.
• The buffer itself could be stored in the terminal (will work even if we ever handle multiple terminals - see #76).

Notes on concurrent reads:

• Input will be delivered to concurrent readers on a round-robin basis.
• If a reader puts back a read result, it will be delivered to the "next in line" reader.
• Should there be a way for a given reader to know if its getting a read it has previously put back? (for example, by tagging it when putting it back?)

Steps to reproduce:

• Bottom window running a text editor process on a PTY.
• Top window, partially overlapping it.
• Focus the bottom window.
• Move the text editor cursor to the area covered by the top window.
• Cursor is visible! :)

Example:

    +-----------------------------------+
    | text editor process with focus    |
    |                                   |
    |                  +-----------------------+
    |    move the      | top window            |
    |  text editor     |                       |
    |      here - - - -|- - -> []              |
    |                  |                       |
    +------------------|                       |
                       |                       |
                       +-----------------------+

Context #5 and #104.

Current limitation:

• q, D and CTRL-F are handled by the loop and can't be sent to process PTYs.

Options:

• __init__ to have a new cursor argument, defaulting to False.
• New method (or property?): cursor_visible with a visible argument, defaulting to True.
• Maybe others...

Idea:

• User hits a given key.
• Kernel toggles a task:
  • Spawns it if the task associated to that key isn't running.
  • Stops it otherwise.

Driving idea:

• Hitting a key and getting a Python REPL at any time.

Extra craziness:

• Maybe such tasks could consult some "global shared state" to determine their behaviour:
  • Where to position their windows (left? right? top?).
  • Which process to launch (Python REPL on a Python slide, Ruby REPL on a Ruby slide, etc..)

...no instance is actually necessary.

Current Status

Existing traps:

Future library design seems to indicate the need for one or more kernel traps to support inter-task communication. This will lead to, at least, one more trap that blocks the calling task (assuming a simple message-send / message-receive like trap pair).

This type of design, while somewhat intuitive, leads to an unnecessary limitation. Waiting/blocking on multiple things:

• What if one task wants to block, waiting for either a child task termination or 5 seconds?
• What about waiting for either keyboard input or a child task termination or a message from some task?

Possible Solution

• All Tasks will have an INBOX, where they receive messages.

• There will be two new kernel traps to support inter-task communication:

  • message-send: sends a message (any object) to another Task, does not block.
  • message-wait: returns the 1st message in the INBOX, or blocks until there is one.
    • Actually returns a (<sender>, <value>) tuple where <sender> will be the sender Task or None if the message is originated from the kernel.

• Inter-task communication:

  • Only supported between parents and their children.
  • message-send takes two arguments: the destination Task and the message.
    • Parents can easily send messages to their children.
    • Children can send messages to their parents by using None as the destination Task (or some TBD constant).

• About waiting for multiple things simultaneously:

  • Currently blocking traps, like sleep, read-key and task-wait will no longer block: they will return some kind of "request id" the caller must track.
  • The calling task must message-wait to confirm their completion: the kernel will send a message to the task once any of the pending requests is completed. Such message will itself be a (<request-id>, <result>) so that the waiting task knows what completed.

• Simpler, blocking variations of the traps can be implemented at a library level.

Will certainly need adjustments, but I like the idea of "async traps" and a single "blocking trap" a lot.

Broken down this issue into:

• #45 - Implement inter task communication.
• #46 - Make all traps except message-wait non-blocking.

Given the initially chaotic exploration, many (variable, function, class) names are either not very good or completely misleading now. Let's improve that:

• Rename the _PlayerStop exception, it no longer makes sense.
• state.running should be state.runnable.
• Find more...

Not sure if it should be handled by the kernel of the lib... (maybe the kernel as it may simplify handling automatic output terminal resizes).

Ideas:

• Position
  • Defined by two args: x and y.
  • Positive int values correspond to zero-based, absolute terminal coordinates.
  • Negative int values correspond to "from-right/bottom" absolute terminal coordinates, where -1 corresponds to the last "column/row".
  • Positive float values correspond to zero-based, relative terminal coordinates where 1.0 is the first "column/row" after the visible ones.
  • Negative float values correspond to "from-right/bottom" relative terminal coordinates, where -1.0 is the first "column/row" before the visible ones.
• Size:
  • Defined by two arguments: w and h.
  • int values correspond to absolute terminal "rows/columns".
  • float values correspond to sizes relative to the terminal geometry with 1.0 represents the full width/height of the terminal.

May be of use:

• Some kind of dx, dy, dw and dh to somehow manage margins.

Examples:

• A full-screen window: w = Window(x=0, y=0, w=1.0, h=1.0).
• A full-screen window with a 1 row/column margin: w = Window(x=0, y=0, w=1.0, h=1.0, dx=1, dy=1, dw=-2, dh=-2)
• Two side by side windows: w1, w2 = Window(x=0, y=0, w=0.5, h=1.0), Window(x=0.5, y=0, w=0.5, h=1.0) (may need dx/dw adjustments to ensure there is no overlap?)

That's the general idea.

Fact:

• Currently the kernel scheduler uses task.throw(trap_result) to throw exceptions into running tasks.
• trap_result is set to an exception class via common.trap_will_throw.

Why:

• help(generator.throw) says first argument is exception type, optional second is exception args.

Given this:

def f1():
    yield 1

Both of these approaches similarly:

r1 = f1()
r1.throw(RuntimeError, 'argument')

r2 = f1()
r2.throw(RuntimeError('argument'))

Given the non 100% clear docs, not even here I went checking CPython's source code where I found:

 481     if (PyExceptionClass_Check(typ))
 482         PyErr_NormalizeException(&typ, &val, &tb);
 483 
 484     else if (PyExceptionInstance_Check(typ)) {
 485         /* Raising an instance.  The value should be a dummy. */
 486         if (val && val != Py_None) {
 487             PyErr_SetString(PyExc_TypeError,
 488               "instance exception may not have a separate value");
 489             goto failed_throw;
 490         }

...this confirms that exception instances can indeed be raised -- the docs need fixing (I should file a bug)

With this, calls to common.trap_will_throw can be given an exception instance with useful arguments such as, for example, the trap name for TrapDoesNotExist.