Has anyone tried something similar yet?

Nah, and most of the maintainers are on holiday 🏄

I'm kinda holding my breath on the python side of the house until pyodide 0.18 lands, as it's going to give us:

• more flexiblity in the filesystem side of the house
  • it will have the various FS modules compiled into the main pyodide runtime... hopefully one of them will give us enough flexibility to mount jupyter contents as FUSE-like "normal" files (as far as pyodide will know)
• new challenges and opportunities in the whl and micropip handling
  • right now, the "special" pyodide-aware wheels get hard-coded 😿

cloning packages pyolite-kernel and pyolite-kernel-extensions

Yeah, something like that.

In addition, we probably want a robolite python package (which would terminate in a wheel-what-won't-go-to-PyPI) which does the various patches we'll need before importing robotkernel... probably importing pyolite and patching its patches 🤯

I think doing it in-tree (off a fork of jupyterlite) is really the only way to get to a working interactive development environment right now, due to the layers and layers of cross-language nonsense... we've been mainly focused on getting something usable out there (and supporting the JupyterLab 3.1 release, in which we found a number of upstream bugs), rather than the full decoupling which would culminate in:

• a deduplicated webpack to cut the total build down to size (retro and lab have mad duplication)
  • so that we can start to think about more frontends
• a (no-webpack) build for adding new "server" extensions

from robotkernel.

copy-paste fork

Right, so that's the first part, but there's a bit of care-and-feeding around making that typescript into a more-or-less self-contained "real" PyPI package.

So one play could be:

• we start a second PyPI package in this repo
  • inside of that is a labextension by normal means (data_files)
• as part of the build of the docs for this repo (which we should have anyway, #93, 😊 )
  • get the per-PR builds
    • use the as-deployed-to-PyPI pip install --pre robotlite to build this docs site

from robotkernel.

@bollwyvl Thanks for all the above! I am sure I am busy enough learning pyolite-kernel and -extension for some time :)

from robotkernel.

Agreed that iteration is slow. My personal X220 seems no longer suitable for real work in typescript world... and there is no upgrade with matching keyboard available :(

But the approach above seems to work. I'm importing pyolite and using it publishing hooks. It seems that once I learn the quirks in pyolite display publishing API, I'll might have a POC.

@bollwyvl What would be the right way to add marketplace/jupyterlab_robotmode into jupyterlite?

from robotkernel.

@bollwyvl Thanks. Locally I had issues with all possible download URL I could find, but by adding the Conda tar.bz2 package into repository worked. So, highlighting works now at https://robolite.readthedocs.io/en/latest/try/lab/ and now I focus on broken messages.

from robotkernel.

Highly recommend getting kernelspy in there for message debugging:

https://github.com/jupyterlite/jupyterlite/blob/main/examples/jupyter_lite_config.json#L12

(ah, see you already do 😊 )

from robotkernel.

Sterling work, sir.

once JupyterLite is “ready”.

My rough order of getting ready for an 0.1.0a7 release:

• presently seeing how the newest pyodide is working out on jupyterlite/jupyterlite#274.
• then jupyterlite/jupyterlite#299 (which may well be delayed due to probably needing a RetroLab release which uses the 3.1 upstream

I am not sure if this will constitute readiness, as I think we really need to have a way to add out-of-tree, user-specified kernels, and server extensions in general, without the WebPack build... preferably reusing the existing @jupyterlab/builder mechanism. This would mean at the very least getting proper npm packages out for the stack up to @jupyterlite/pyolite, but very possibly with dedicated support packages (e.g. @jupyterlite/builder, and if need be, a cookiecutter).

In the meantime, please don't hesitate to do a draft PR to lite itself, so other can be made aware of this effort, which is probably the first wrapper kernel anyone has been willing to share.

from robotkernel.

I've noticed some similar weirdness with @ipywidgets.interact, which also does tricky things to the output area, but hadn't gotten around to investigating it further, as other folk have been throwing a lot of weight at it.

Keeping the parent_header straight is indeed a minefield in the face of successive execution.... will take a look.

from robotkernel.

Yeah, we're generally going to run into a bunch of things around asynchrony.

If you can get XMLHttpRequest working, that's great... but for other stuff (e.g. the input builtin) we're going to need some more clever techniques.

from robotkernel.

from robotkernel.

Oh, I definitely want to get off the CDN!

The conda-inspired packages.json format changed in 0.18, and so I was waiting to have that to work with. Eventually, jupyter_lite_config.json will grow a way to point to a bunch of local whl files (and maybe custom pyodide js/data packges 😱 ) and generate warehouse-compatible /api/ endpoints and packages.json responses which will be overlaid. We need this first-party, anyway, for our custom wheels, and this will also reduce the webpack build time some 🎉.

Good luck building pyodide locally! I got it working long enough locally to do the FS PRs, but it's a beast. Now that we can cross-compile, I'm really hoping conda-forge can start doing a wasm-32 target to bootstrap the ecosystem... but there's politics there, as browser-based WASM isn't the only game in town!

from robotkernel.

Keyword widgets (buttons) work now. Next I wait for pyolite rename, then make robotkernel release with JupyterLite support and make pull request to JupyterLite for kernel-extensions.

from robotkernel.

It's been a spell, sorry about that! I had a couple things go down outside of FOSS, and haven't been slinging much code. Hopefully we can start warming things back up, as it cools off more up here!

Of interest, jupyterlite landed and released jupyterlite/jupyterlite#352 which allows for adding new "server" extensions (such as kernels). So far there's:

• a simple, but "real," kernel split off during that PR
• an even simpler one
• and a cookiecutter for getting up to "packaged for pypi"

Python is all still in-tree: I don't know whether it makes more sense to do a name-only release of the renamed kernel jupyterlite/jupyterlite#116, or pursue the flexibility improvements on I've started jupyterlite/jupyterlite#310 which would normalize the management of custom python packages (and help keep them in sync with labextensions)... probably the latter, so that we can start anticipating what a "complex" modification to the build environment will entail (hence all the entry_points and schemas and stuff).

from robotkernel.

Yep, works for me!

from robotkernel.

@bollwyvl Minor update. Last weekend was not like I expected, but I am back on this starting from today. I have still things to clean up, but I expect to the the first version out (much) later today. I seem to be super slow nowadays :)

from robotkernel.

weekend was not like I expected

when are they ever!

first version out

obviously rope me in if you hit any snags...

meanwhile, stuff marches on over at lite: the next version of pyodide will be based on python 3.10, and does everything with wheels (no more .data files). with any luck i'll get some runway to work on the embedding case bit more intently soon, but can do quick hits if needed.

from robotkernel.

extension that implemented the three API endpoints

no dice: it doesn't use the right fetch code path to be affected by the lite server... will see what else i can do, short....

from robotkernel.

Actually did some looking: addons are not getting initialized with the parent configuration... PR incoming...

from robotkernel.

Another question: Should jupyter-videochat already work in lite?

Does now (just dropped 0.6.0). The room argument is different JVC-PUBLIC so a full URL might be, after updating:

https://pidgy.readthedocs.io/en/latest/_/lab/index.html?path=tangle_examples.ipynb&room=FOO&JVC-PUBLIC=FOO

from robotkernel.

@bollwyvl What's your current recommendation on this? I spent some time looking around in Jupyter Lite and, I must admit, I am not completely sure is this possible yet.

If I would try this now, I start by cloning packages pyolite-kernel and pyolite-kernel-extensions and then try to figure out how to refactor robotkernel so that the service worker could patch robotkernel comms... but there could still be blockers that I am unaware.

Has anyone tried something similar yet?

from robotkernel.

oy, pyodide 0.18 is up: https://github.com/pyodide/pyodide/releases/tag/0.18.0

no idea on when i'll be able to hit it, but no external blockers is a good place to be...

from robotkernel.

from robotkernel.

https://robolite.readthedocs.io/en/latest/try/lab/

I believe that robotmode is bundled, but does not yet activate. For now that fork includes prebuilt robotkernel wheel while I'm locally resolving issues with different message types.

Update: Neither .robot -files are highlighted, nor is Robot Framework available as mime type manually for text files.

Update: I'm also puzzled that "doit watch" keeps two of my cores 70-100% utilized even when there is no updates. Actual build on update is pretty fast though.

from robotkernel.

@bollwyvl What could push robotmode into disabled plugins?

docs/_build/_static/lab/build/build_index_js.HASH.js:

// custom list of disabled plugins
const disabled = [
  ...JSON.parse(_jupyterlab_coreutils__WEBPACK_IMPORTED_MODULE_2__.PageConfig.getOption('disabledExtensions') || '[]'),
  '@jupyterlab/apputils-extension:workspaces',
  '@jupyterlab/application-extension:logo',
  '@jupyterlab/application-extension:main',
  '@jupyterlab/application-extension:tree-resolver',
  '@jupyterlab/apputils-extension:resolver',
  '@jupyterlab/docmanager-extension:download',
  '@jupyterlab/filebrowser-extension:download',
  '@jupyterlab/help-extension:about',
  '@marketsquare/jupyterlab_robotmode'
];

Update: So it is disabled, because it is not bundled after all.

from robotkernel.

"regular" extensions should be provided via the federated_extensions mechanism during the "full" build, a la

https://github.com/jupyterlite/jupyterlite/blob/main/examples/jupyter_lite_config.json#L3

from robotkernel.

So, basic usage seems to work. Execution count is lost on successful executions but not on failures. Maybe I didn’t understand all the shortcuts I took. Keyword execution widgets don’t work yet. Anyway, it should be safe to say that this could be done once JupyterLite is “ready”. And I should continue earliest after Pyolite has been renamed 🤔

from robotkernel.

Thanks again. I'll see if I can clean things up enough for a pull request. Don't hold your breath.

I try to avoid writing new code, so my clones of pyolite-kernel and pyolite-kernel-extension have just the very minimal changes from their original versions, and then I've been refactoring robotkernel to be compatible with pyolite. Maybe at some point I could simply depend on pyolite-kernel and inherit its base classes to avoid maintaining copy & paste.

Also, I finally tried this out on a desktop and that has enough juice to run the development environment in reasonable way.

Keyword widgets mostly worked out of the box after all, with one issue:

With custom output widget the output is not rendered:

If I remove with out and render to the default output, it works.

Update: Got trace of this. I need to fix kernel._parent_header.

from robotkernel.

Widgets mostly work. Once readthedocs has updated. Widget execution clears the button, which is probably because the message helper from pyolite did not pass display id forward...

Anyway. Added a smoketest notebook: https://robolite.readthedocs.io/en/latest/_static/retro/notebooks/?path=robolite%20-%20smoketest.ipynb

from robotkernel.

pyolite did not pass display id forward...

Oooh, sounds like a real bug.

Maybe at some point I could simply depend on pyolite-kernel and inherit its base classes to avoid maintaining copy & paste.

Yeah, we should enable this upstream to the extent possible. Presumably the sticky wicket is the huge case statement... we likely want a pattern where an implementation need only implement handlers for the message types it supports, and the rest are silent no-ops, by default. I think one only really needs to implement kernel_info, execute_request, and probably shutdown_request to be "a kernel," but so much else will be dependent on the implementation.

Another thing i'd like to explore is normalizing the worker implementation with comlink, as pyodide supports interacting with that a little more natively... this is probably how we'll end up being able to support DOM manipulation from inside the worker, which would no doubt become importance for providing an in-browser facade for WebDriver.

from robotkernel.

pyolite did not pass display id forward...

Oooh, sounds like a real bug.

Well, that was too hesitated guess from me. The issue is still with

For keyword execution widget execution, RobotKernel creates ipywidgets' output widget and relies on its context manager to make that the output default display output. Unfortunately, that does not work, but display commands are sent to the last active cell's default display output. (And sorry for the possibly inaccurate terms.)

Even more unfortunate, I don't understand well enough how messages are targeted to understand, where things go wrong. I wonder, would you be able to understand Kenel spy output for the keyword example (what happens when button for the keyword cell is clicked) at "robolite - smoketest" at https://robolite.readthedocs.io/en/latest/_static/lab/ ?

The expected behavior is to render keyword output or stderr below the button, before the log | report -links. What happens is the output replaces button and possible stderr is not displayed (Kernel spy does show the message, but it is not displayed.)

That output widget context manager seems to work on just Pyolite, so this is a robolite issue.

from robotkernel.

Yeah, i can't track much what's going on. A demonstration didn't get added with jupyterlite/jupyterlite#154 so it might not actually be fixed...

from robotkernel.

@bollwyvl I'm able to reproduce this with pyolite

import micropip
await micropip.install("ipywidgets")
import ipywidgets

out = ipywidgets.widgets.Output()
display(out)
with out:
    display("1")
def click(*args):
    with out:
        display("2")
button = ipywidgets.widgets.Button(description="Display")
button.on_click(click)
display(button)

Expected behavior is tsimilart that this does on normal Jupyter

import ipywidgets

out = ipywidgets.widgets.Output()
display(out)
with out:
    display("1")
def click(*args):
    with out:
        display("2")
button = ipywidgets.widgets.Button(description="Display")
button.on_click(click)
display(button)

from robotkernel.

Awesome, that's a great reproducer, please issue!

from robotkernel.

jupyterlite/jupyterlite#303

from robotkernel.

@bollwyvl I was about to start learning js.fetch in pyodide, but with Robot Framework that turned philosophical sooner than I expected.

js.fetch is async for obvious reasons, but Robot Framework is not, and probably won't be any time soon. On CPython the workaround is to turn asynchronous code synchronous with loop.run_until_complete but that is not supported by Pyodide WebLoop to not block the browser thread. That said, similarly to pyolite-kernel also robolite-kernel runs in web worker, which should prevent it from blocking the main thread?

Would you have an idea, how to make calling asynchronous js.fetch possible in synchronous Robot Framework Python keyword?

Update: My best guess so far is that I might be able to use Atomics.wait to sleep in loop until fetch has been completed.

Update: I was naive about Atomics.wait. It blocks also the worker executing fetch, so that approach would probably require creating a new worker from fly, executing fetch on that worker and then relying on Atomics.wait on the main worker to wait for the async fetch to complete.

Update: Also worker started from the worker is only executed after the code that created it no longer blocks... 🙈

from robotkernel.

@bollwyvl After learning too much about web workers, I learned that XMLHttpRequest has synchronous mode 🙈

from robotkernel.

I had not tried yet, but I wonder, if the approach I tried would work better the other way around: if the main thread would provide postMessage based i/o -services for the worker thread (exposed through js module for Python) and the worker would call the main thread and then do Atomics.wait until the main thread has the answer. Although, I am not completely sure can SharedArrayBuffers really be passed with postMessage for this kind of signaling 🤔

from robotkernel.

Hm... postMessage doesn't even connote that a reply is expected... this would put you back in a polling/queue kind of situation... but I am also not familiar with the Atomics API.

arrayBuffers are available, usually, in a performant manner: micropip uses this approach to whls, so is likely good up to some bounds, and probably greater than the 256mb string limit.

from robotkernel.

@bollwyvl Atomics API allows to do a blocking wait for a triggered change in Int32Array (Atomics.wait is only allowed in service worker). In theory: child would send buffer with postMessage and start Atomics.wait for a change in the buffer in a blocking manner. But this is just lipservice until tried out what happens if parent first does postMessage and then Atomics.trigger or in the opposite order.

I did try this so that kernel worker starts another worker, but as I mentioned the another worker started only when the kernel worker no longer blocked (Atomic.wait configurable timeout occurred).

from robotkernel.

@bollwyvl Btw, have you already thought about CLI execution of Jupyter Lite notebooks? I assume that it is not trivial due to differences APIs provded by NodeJS and browsers. So, 1:1 execution would require execution in browser and that is probably a mess. Possibly with puppeteer?

from robotkernel.

Slightly unrelated, but I did embed Jupyter Lite into Camunda Platform where it could be used to both try analyze its historical data and learn to build Robot Framework bots for process automation tasks:

from robotkernel.

@bollwyvl That's well prioritized. It was a bit confusing what could be done with just jupyterlite-command and what requires forking the project, but I was happy to find my way eventually. I am happy now and maybe give more feedback when the more current issues have been solved. Eventually I would like to have "static builds" that would not need "Internet" to get started. Because pyodide could be server locally and micropip could install from any URL, that is probably "just work".

Right now, I have been trying to do my own pyodide builds with hoodmane's http.client-implementation to see how far would it be to make requests to work. But could not get a working build due to Module._sqlite3_libversion is undefined runtime error with pyolite. I will probably try to do a build later without their Docker image to see if it as any difference.

from robotkernel.

About Pyodide, I got it compiling properly, but http-client-patches ended up being too complex for me.

from robotkernel.

@bollwyvl Thanks. I cannot say when I can dive into learning all that, but I've been thinking about making a submission to the next RoboCon about that...

from robotkernel.

dive into learning all that

we have to dig the lake before you can go swimming!

submission to the next RoboCon

Welp, you can put me down... i'm fairly confident, between the two of us, we've got something entertaining to say.

Of course, I don't know if i'd be able/allowed to make it in person, but who knows! It's been too long, for sure. I think we've proposed to Pydata global for core, but I think there's a broader, not-exactly-python-or-data-science story!

from robotkernel.

RoboCon CFP is open until the end of November, so there is plenty of time left to think about it https://cfp.robocon.io/robocon-2022/cfp

They'll make it "hybrid" with pre-recorded presentations being allowed.

How far ahead, would you think, is being able to do CDN free builds or embed jupyter lite notebooks into Sphinx docs?

from robotkernel.

CDN free builds

Assuming what is meant is "end users won't contact CDN"... the buck has to stop somewhere, and the build will require getting packages from something that is probably a CDN (e.g. conda-forge, pypi).

At present, the biggest self-hosting/reproducibility "holes" that come to mind:

• mathjax
  • jupyter-lite.json already has mathjaxUrl
  • could use a build-time flag to download/cache/locate an existing one, and then copy it
  • if it is known it is not needed, might be able to be disabled already
• pyodide runtime and core WASM packages
  • same as mathjax, pyodideUrl is a thing in jupyter-lite.json
  • same as mathjax, needs a build-time flag to locate/download/cache, then copy a full pyodide
• pypi JSON API, wheels
  • 80% covered by jupyterlite/jupyterlite#310
  • needs a build time option for find wheels here and here
    • working on this right now
  • falls back to pypi if a package is not available on import
  • might be able to allow disabling this

embed jupyter lite notebooks into Sphinx docs

To get this, today, I'd iframe in a /retro/tree URL and get freaky with the CSS to hide all the stuff i didn't want (menu bars, etc). With that, one could get all the way down to jupyterlite/jupyterlite#148, or somewhere in-between.

from robotkernel.

@bollwyvl Thanks. All this sounds good enough. I'll draft something for RoboCon CFP in coming weeks and will then ask for comments. I might prefer pre-recorded presentation even myself, because it is easier to attend conferences virtually while on paternity leave 🙈

from robotkernel.

@bollwyvl I posted a RoboCon CFP draft link to your GMail from Goole Docs. If you are still interested, please, proof read and make corrections & additions as you like. I tried to phrase it so that a working demo is enough and we are not promising to have ready-to-use releases by the conference yet. I have not updated my JupyterLite + Robot for some time, so it might need some love. I am not sure when the pre-recodrings must be ready, but I'd expect that two week before is enough (so we could do that right after holidays).

from robotkernel.

proof read and make corrections & additions as you like

Great, will do so today!

not updated my JupyterLite + Robot for some time

I think we've almost gotten down to where the piplite, son-of-micropip stuff is squared away, and was actually hoping to be able to demo with entirely self-sufficient assets at the next jupyter community call.

when the pre-recodrings must be ready

sounds good!

from robotkernel.

Actually, if you had any time to look over the delta to the changes to kernel startup itself and give any pointers how it might better suit your needs to get closer to reuse-by-inheritance rather than -fork, that would be excellent.

from robotkernel.

So that PR got merged, along with a couple others that at least make self-hosting of everything i can observe work, and have been released as 0.1.0a18.

With pyolite still in the main js build, it's not very nice to reuse the assets directly: this pr shows one path to getting a custom kernel by mostly copy-pasta. There are a lot of pointy edges, still, but hopefully getting more fingers/eyes on this stuff will help us smooth things out.

from robotkernel.

@bollwyvl Now that Robocon got moved to May, is it still ok for you? 19-20 May 2022. I recall we submitted for remote talk, so we would need to record it beforehand, but then be available for remote-live-discussion after the scheduled presentation.

from robotkernel.

@bollwyvl Cool! I’ll accept our acception and will ping you via Google sheets via near future.

from robotkernel.

@bollwyvl I finally managed to update my JupyterLite https://robolite.readthedocs.io/en/latest/_static/lab/?path=robolite%20-%20smoketest.ipynb (1.0b3) https://github.com/datakurre/jupyterlite/pull/1/files

I guess that next I should use the kernel extension scaffold to re-create robolite-kernel and robolite-kernel-extension as separate packages to make them generally available. There is already robotkernel 1.6a2 that supports pyolite. Will do during the weekend.

AFAIK the last thing missing is somehow command jupyter lite CLI to generated pipliteUrls configuration also for robolite when jupyter lite build. Currently it is hard coded to only do that for pyolite.

from robotkernel.

Extension-by-subclass, which was originally intended, didn't quite work last i checked. For pidgylite i ended up doing a copy-paste fork of pyolite-kernel and pyolite-kernel-extension. That'll be able to force installing it.

There is an entry_point which can change how jupyter lite build works, with a couple lifecycle stages to hook.

If you think the 1.6a2 whl is ready to rock, I can probably take a look at adapting the pidgylite approach relatively quickly, if you know what form we'd like to document it having, e.g.

• what do you pip install
• what do you run to get a site with the kernel

from robotkernel.

Doing a copy-paste fork of pyolite-kernel and pyolite-kernel-extension sounds like exactly what I have done with robolite-kernel and robolite-kernel-extension. I'll try to follow pidgylite path during the weekend...

from robotkernel.

@bollwyvl Almost there: https://robotkernel.readthedocs.io/en/latest/_/lab/index.html

pypi/packages from pyolite are probably being replaced with real ones and that causes issues.

I tried to stole everything I could from pidgylite.

And probably a lot of things to polish. This repository is getting a bit messy, while I am dancing between Nix, Conda and Pip.

Continuing tomorrow...

from robotkernel.

Couldn't find a pure Python 3 wheel for: 'psutil', 'tornado>=4.1', 'pyzmq>=13', 'tornado<7.0,>=4.2'

Ah, that's from (failing to) use the ipykernel shim, as the real one does all kinds of stuff that the browser... can't. So if, before you run jupyter lite build, you ensure that ipykernel, etc are not included.

Getting the pyolite wheels right is still error-prone... but it's hard to make everybody happy. for example, if you try to micropip.install (or piplite.install) anything that gets shipped with pyodide, it will use that, and there's nothing much we can do about it. We use micropip to bootstrap up to piplite, and then use that to move further.

For piplite, we left the ability to overload any of the dependencies of the kernel, and ship those few wheels to handle common cases... unless you really mean it, though, overloading them is a bad idea, as they'll pull in all kind of deps.

stole everything I could from pidgylite.

Excellent.

dancing between Nix, Conda and Pip.

yeah, well we do what we can. at least no docker or windows... unfortunately, i'm unlikely to be of much help if nix is required to get going on it, as i still haven't had the time to dig into it.

Anyhow: when you want to take a breath, it's worth trying to scope down the fork into an actual pip-installable package. I can slo

from robotkernel.

@bollwyvl It works now https://robotkernel.readthedocs.io/en/latest/_/lab/

I added extra steps to cleanup the conflicting wheels.

Next I will reorganize docs and link lab with collaboration features, set those pull request versions and then merge.

A question: Pidgy and now also RobotKernel uses dynamic doit-tasks to build JupyterLite with the latest development version. But what is the official end-user way? Have jupyter_lite_config.json with absolute URLs for all the required extensions and PYPI packages? Is there way to generate example jupytger_lite_config.json?

Nix is just my personal passion, headache and mostly aesthethic issue because of the current bloat in this repo. I know Conda is more practical and works with all the services like RTD and even Windows. In my projects, I have preferred to have Nix setup for myself and then "graceful degradation" to pip for the others. In this case, I should forget bare pip and only support Nix and Conda 😄

from robotkernel.

from robotkernel.

@bollwyvl Thanks again.

So, following jupyter lite build approach, it should now be possible to

pip install jupyterlite-robotkernel
jupyter lite serve

https://pypi.org/project/jupyterlite-robotkernel/

Thanks to robotkernel wheels being fetched by overridden entrypoint:

This seems to work, but would you prefer to do it otherwise?

Another question: Should jupyter-videochat already work in lite? Eg. at https://pidgy.readthedocs.io/en/latest/_/lab/index.html?path=tangle_examples.ipynb&room=

from robotkernel.

Figured out that jupyter-videochat is just missing a release (and the supported version is at develop branch).

from robotkernel.

🤯

overridden entrypoint:

that's awesome that it works, but might have unwanted longer-term side-effects.

in this case, the labextension can bring its own wheels to the party, and lite build should find them... pidgylite is probably the only place this has actually been done, so...

• in package.json
  • claim some wheels
    • the prefix is relevant
      • apparently, for webpack's sake, it can't be pypi, and has to be at pypi/something because the root takes pypi
  • include in tarball
  • use a custom webpack
• in webpack.config.json
  • set up webpack
• get the wheel dir right
  • ensure at least the wheel for the kernel itself and any hacked dependencies ends up in there
    • pyolite, piplite, etc. will get shipped from core... but if we need to hack those, too, we can
  • generate the all.json
• run jupyter lite
  • it looks for that package.json metadata and wheels, and updates the runtime pipliteWheelUrls

obviously this should work better.

jupyter-videochat is just missing a release

yeah... i merged it, and tried it out for a demo: i introduced some nasty react focus bugs from something i did in the last refactor, and i started getting fed up with how i'd been adding features to react (just use more props!) instead of proper components, and got somewhat frustrated, and then moved across the country, and never came back.

if we don't mind some react hiccups, we could check in the the CI-built wheel in the demo repo, and configure rooms with settings

to use the current real release, could make a short lifespan lite "server" extension that implemented the three API endpoints... i'll see if i can make that real.

from robotkernel.

@bollwyvl Thanks for explaining all the steps in pidgy for bundled Python packages step by step. I kind of followed it through earlier, to but could not believe that I should bundle robotkernel and all its dependencies into jupyterlite-robotkernel.

That would make a 23M wheel. Sure it would be robust (without side-effects and breakages due to updating packages) and also offline-runnable then. So, maybe I'll do that anyway after all.

from robotkernel.

robotkernel and all its dependencies into jupyterlite-robotkernel.

you'd only need to bundled hacked dependencies in the labextension that would be bad or impossible to put on PyPI (because you don't own them, or they only work in the browser).

pidgy is still super experiemental, but robotkernel has been through a bunch of a releases. If the off-the-shelf kernel and all dependencies work as-downloaded, then no bundled wheels are required... but does put the onus of specific wheel version management on the site owner... but that's probably just fine as they are going to want to configure a bunch of stuff.

if anything, a new entry_point would augment the existing behavior, rather than replace it, and ensure that an appropriately-versioned robotkernel wheel was not already specified by a site owner, that it would get downloaded and injected... but then it gets back to the problem of "from where?" The complexity immediately goes crazy: what about proxy servers? what if i'm offline? what if i want to pull from multiple sources? zzzz.

And indeed, that's what we do upstream on pyolite:

• in core, just ship ipykernel and widgetnbextension
• in the demo site, download and ship a managed , known good IPython, traitlets, etc.

23M

right, that's a huge wheel, and feels bad to push to npm and or pypi if all the stuff already exists on pypi....

offline-runnable

...but for an actual site to be deployed that could stay running for years without any maintainence, it would be totally worth it to either check those 23M all in, or at least the whole "big list of wheel URLs", and have a plan (outside of dumb-ol-jupyterlite) to get those wheels to the right place.

from robotkernel.

in this case, the labextension can bring its own wheels to the party, and lite build should find them... pidgylite is probably the only place this has actually been done, so...

It seems that there might be still some discovery issues when labextension has been installed into virtualenv. I think, the discovery is only triggered for piplite_urls from jupyter_lite_config.json.

Anyway, I seem to have three options:

1. ship with no wheels and use pip to resolve wheels in built-time

2. ship with no wheels and provide jupyter_lite_config.json with KGS

3. ship with all wheels

I think, I start with 1), but with generating jupyter_lite_config.json KGS while running the first build. It seems that at least now there are deterministic URLs for the required wheels.

I did try 3) and was able to resolve wheels (./pypi/robot and ./src/_pypi.ts) with pip in doit lite and bake them into labextension wheel. I'll probably return to this approach later, when everything else is more stable and mature.

from robotkernel.

discovery issues when labextension has been installed into virtualenv

the labextension stuff is older, and.... differently complicated than the piplite stuff. But yeah, it's not super clever, but has been working well enough on e.g. RTD. Unless to to ignore_sys_prefix, jupyter labextension list and the resulting federated_extensions should line up.

three options:

Maybe a fourth, of actually adding (instead of overridng) another entry_point, e.g.

• make an entrypont in jupyterlite.addon.v0 for robotkernel in pyproject.toml/setup.cfg/setup.py/poetry.toml
• have it implement a build that downloads the needed wheels into output_dir / 'pypi'

bake them into labextension wheel

yeah, this is a tough one, as a balance between upgradability and stability. perhaps a middle ground of shipping a known set of pypi urls to be cached and copied to the output_dir, but then allowing them to be overridden? I actually don't know how such configuration would be added today, though... but maybe the point is if they aren't downloaded as part of pyolite.

I guess let me know how you'd like to proceed, and i can try to PR something that moves towards it, wherever it should go... we're doing some fairly novel stuff here.

from robotkernel.

Maybe a fourth, of actually adding (instead of overridng) another entry_point

Well, I could think if "hook entrypoints", which could have multiple registration and the default implementation would call all of them in some random registration order.

For example, right now I would need something like:

• piplite_post_init_hook
• piplite_patch_jupyter_lite_json_hook

But those would be quite specific indeed. Therefore, because the current approach already works (until someone wants to compose jupyterlite-robotkernel with other similar packages), I'd wait for more use-cases until tailor something for robotkernel. After all, the issue of adding populating piplite's pypi for a specific kernel is a more generic requirement.

from robotkernel.

oy, hadn't actually looked at the code, but... a lot of that isn't going to work cross-platform.

So anyhow:

• each addon is a traitlets thing
• each addon can implement hooks that are strictly ordered, e.g. (pre||post)init, then build, for gross ordering
  • each hook emits tasks
    • each task can determine its ordering either with file_dep (i need this file to exist) or task_dep (I need this task to have run)

In the case of wheels, they just need to get into output_folder/pypi/*.whl, before PipliteAddon.post_build.

import json
from jupyterlite.addons.base import BaseAddon
import traitlets

# or importlib/pkginfo or wathever
DEFAULT_WHEELS = json.loads(
    (Path(__file__).parent / "wheels.json").read_text(encoding="utf-8")
)

class RobotKernelAddon(BaseAddon):
   """Ensures the unique dependencies of robotkernel are available."""
    __all__ = ["pre_build"]
   wheel_urls = traitlets.List(DEFAULT_WHEELS).tag(config=True)
   
   def pre_build(self, manager):
        """Downloads wheels.""""
        for wheel in self.wheel_urls():
            dest = self.output_dir / "pypi" / Path(wheel).name
            if dest.exists():
                continue
            yield dict(
                actions=[(self.fetch_one, [wheel, dest])], 
                targets=[dest]
            )

and then overridden in jupyter_lite_config.json:

{
  "LiteBuildConfig": {},
  "RobotKernelAddon": {
    "wheel_urls": [
       "..."
    ]
  }
}

from robotkernel.

Thanks again!

oy, hadn't actually looked at the code, but... a lot of that isn't going to work cross-platform.

Something else that rm calls, which could be replaced with stdlib calls?

How is that RobotKernelAddon discovered? Is the discovery part of traitlets (and just because it inherits `BaseAddon``)?

from robotkernel.

from robotkernel.

If we get some useful pattern, we could work it up as a docs PR... e.g.
"writing a wrapper kernel with pyolite"

Oh, so I can just invent my own, like

[project.entry-points."jupyterlite.addon.v0"]
robotkernel = "jupyterlite_robotkernel.piplite:RobotKernelAddon"

that was the thing, I somehow missed from the beginning. I though, I could only override some of the predefined entry-points.

Now all the above makes much more sense! Thanks for your patience!

Iterations, iterations...

RoboCon is expecting recordings around Mid-April, so there is still some time. I suppose, I can finally make this jupyterlite-robotkernel proper and cross-platform. Next I need to update my Camunda-integration with everything I have learned to have a better demo. Hopefully be able to start outlining around weekend...

from robotkernel.

Could not get the overriding part to work

{
  "LiteBuildConfig": {},
  "RobotKernelAddon": {
    "wheel_urls": [
       "..."
    ]
  }
}

Any tips to debug, what could be wrong?

from robotkernel.

from robotkernel.

@bollwyvl Would you be available to complete our RoboCon recording next week? I'll be working on plan and slides little every day and then the next weekend, and would be available from Mon to Thu (18–21th). Unfortunately, I am not available during the weekend after that, but then again 25th–. Feel free to contact me by email. I'll try to adapt to your available slots.

from robotkernel.