Consider adding live demo with binder? about robotkernel HOT 36 CLOSED

@bollwyvl Thank you for the nice words! I'm sorry that I forgot to follow this repository and your work fell under my radar.

I have some opinions, which prevent me to merge your work as such, but please keep your branch alive so that I can use it as a reference when I get chance to work on this again.

I definitely want to make this usable with Jupyterlab.

from robotkernel.

from robotkernel.

@bollwyvl Thanks for asking for the license. It's now licensed under BSD 3-Clause.

from robotkernel.

from robotkernel.

@bollwyvl Thanks for your help. I have now migrated your recommendations for kernel installation and binder support.

README also has now Binder links for Notebook and Lab

Jupyter Notebook: https://mybinder.org/v2/gh/datakurre/robotkernel/master?urlpath=tree/example.ipynb

Jupyter Lab: https://mybinder.org/v2/gh/datakurre/robotkernel/master?urlpath=lab/tree/example.ipynb

I've not managed to create working Jupyter lab extension for CodeMirror mode yet. The code is here https://github.com/datakurre/robotkernel/tree/master/src/jupyterlab_robotmode and it is installed with Binder.

Also those fancy log links don't seem to work for me with Jupyter Lab.

from robotkernel.

I opened issues for the lab issues #5

from robotkernel.

from robotkernel.

@bollwyvl Thanks for the update. That's even more exciting! Sounds like you have made JupyterLab a fully featured robot IDE... Please, keep me up-to-date how the open sourcing progresses. If you kept the name, I should probably pass the PyPI reservation for you when that happens :)

from robotkernel.

@bollwyvl Any chance that you are planning to submit a talk to http://robocon.io/ (next January in Finland, CFP opens within the next weeks) about your robotkernel work and use-cases? If not, I'll submit mine.

from robotkernel.

Thanks for the update

Shame on me for not updating sooner! It's too bad we're duplicating work, but there is definitely value in multiple implementations and perspectives: I'm carrying a lot of conda and Jupyter baggage, and am relatively new to robot... though I tell you, writing a faithful syntax highlighter and completer with tests will get you up to speed REAL FAST.

If you kept the name, I should probably pass the PyPI reservation for you when that happens :)

I was actually thinking of ipyrobotframework, to more concretely reflect that it still has all the IPython goodies under the hood...

Sounds like you have made JupyterLab a fully featured robot IDE...

That's the goal! I desperately want to get my code out there, but since The Man is paying for the work, I have to follow procedures.

Any chance that you are planning to submit a talk

Actually I was! However, the cost might be prohibitive for me to attend in person, so it would make sense to collaborate. Indeed, it would be a sporty 30 minutes plus questions, as I have a whole boatload of in-house work around Robot going on. Maybe also submit a training session?

Not on company time, but would like to present some how, I'm also doing some exploratory work (also in JupyterLab) for rich input types including blockly.

The idea here would be to have a libdoc -> blockly JSON -> blockly blocks -> Robot workflow to allow visual authoring. The first cut will be to get it to work, but the long con is having additional output types that can register new blocks and code generators from kernels... like ipyrobotframework.

Exciting times!

from robotkernel.

from robotkernel.

Yeah, so turns out you get not only clientId, but also deletedCells.

I put it in, and no longer get Setting was already set, etc. when interactively building things.

Basically,

• self.robot_history = OrderedDict(), and after overloading execute_request

from robotkernel.

(whoops, sent too early)

... to pass in the parent['metadata'] around here

• pull the cellId and the deletedCells
  • defaulting to a uuid.uuid4() and []
• robot_history.pop each of the deletedCells
• pre-populate with history except the current cellId (i made this a function, since it's used in complete and inspect)
• clear the test cases
• populate the current cellId
• if successful, robot_history.set(cellId, code)

And away you go!

Anyhow, I got some buy-in from The Man on submitting the talk (with a requisite open-sourcing prior), so who knows! I'm sure Helsinki in January is lovely!

from robotkernel.

@bollwyvl I recall I was really missing cell id's and deleted cells. That's great news! But those are only in Lab, so the kernel needs to fallback to previous behavior without them?

Blocky / visual editing sounds interesting also from "RPA" perspective (there is some push currently for Robot Framework into "RPA" markets). Do you have anything public on that yet?

I'm sure it is enough to send one talk submission about Jupyter. Ping me if you need a co-author :-)

Helsinki sure is lovely in January if it happens to get snow in time. Otherwise, not that much (dark, wet, chilly and windy)...

from robotkernel.

fallback to previous behavior without them?

Right: the fallback (random cellid) is first in, first replayed... The ordereddict takes care of the rest, preserving the original behavior. I suppose you could also use the hash of the content, to avoid trivial repetition.

Do you have anything public on that yet?

Just this branch... Doesn't even work on binder, yet, because I'm still figuring out how to do it, and don't want to check in 50k of generated xml and json:

Initially, I am building it offline and bundling into the front end along with a hacked python generator. The idea, however, is to have the kernel emit content types for:

• blocks
• toolbox
• code generator recipes
• natural language translations (!)

Which would update a custom generator (e.g Blockly.RobotFramework). This could be taught how to make python and javascript for keywords that spoke those languages.

Then there are a number of ways to model a domain with blocks

For example, a suite "hat" block could have two "shelves" which accept settings and variables, while each test or keyword "hat" block could have a "shelf" for bracket settings, and another for keywords and variable assignments. As usual, templates mess everything up, so you'd need a whole separate approach for them.

Anyhow, tons to do there, but not on anybody's critical path, so it will be slow going.... But would definitely add some validation of the robotkernel model.

Another rich output straight out of RIDE would be the tables, a la

https://github.com/quantopian/qgrid

Ping me if you need a co-author :-)

Right! Will do!

from robotkernel.

Thanks for the insight into your blocky work.

So, blocky editor itself could be a JupyterLab extension, but it requires support from the kernel to parse robot libraries and keywords into blocky models?

Should the blocky extension parse robot syntax or should robot have parser for blocky json?

from robotkernel.

Thanks for the insight into your blocky work.

Thanks for listening!

So, blocky editor itself could be a JupyterLab extension, but it requires support from the kernel to parse robot libraries and keywords into blocky models?

Requires is a strong word: heck, i'm pretty sure you could run robot right in the browser. But ignoring that for a moment, I'm trying to reuse existing machinery to the extent possible.

In Lab, you have a few choices for getting new ui in:

• build it in at webpack time (bleah)
  • for my in-house builds, I'm considering packaging up a robotlab pre-built set of extensions, because node and windows. probably stuff a few choice extensions like @jupyter-widgets/jupyterlab-manager (did i mention widgets work?), @jupyterlab/toc
• output MIME-typed data from the kernel
  • the plan, for sure, for output.xml
• server extension serving a new REST endpoint, or even HTML if you need it
  • considering for libdoc output to avoid having to re-implement that special syntax

Should the blocky extension parse robot syntax or should robot have parser for blocky json?

from robotkernel.

(oops, sent too early, again)

Aside from BuiltIn (which maybe could be, well, built in), everything else is so dynamic, and might not be readily accessible to the notebook contents server. So having the js side do too much of that work is probably a lot to ask, but could step in for certain cases, e.g. multiple open notebooks.

Should the blocky extension parse robot syntax or should robot have parser for blocky json?

Aye, there's the rub. Blockly blocks are defined in JSON, but a configuration of a blockly workspace is defined in XML, and I feel like there are enough unschema-ed XML documents flying around at this point. Blockly desperately wants to not be a language, and instead a code generator, though folks have gone pretty far down that road, as well.

Anyhow, I feel like keeping the kernel (and robot) parsing the almost-human-readable content as a priority, with things like blockly, or RIDE-style spreadsheets, as purely authoring tools.

from robotkernel.

@bollwyvl Thanks again. I’ll return to this in a couple of weeks. I’ll probably expriment with these topics myself already with robotkernel, because we started to use it for RPA development by ourselves and all this would fit perfectly also with our use case.

from robotkernel.

from robotkernel.

@bollwyvl 25 minute talk and workshop sound good. Too bad that the workshops are before the talk. Would you also be able to stay for the sprints on Friday?

I'be been playing catchup on my legacy kernel: simple auto-completion, keyword / section documentation as inspections, and use of cellId and deletedCells.

A more interesting one was "sticky webdriver sessions" https://github.com/datakurre/robotkernel/blob/master/src/robotkernel/listeners.py#L149 so that when browsers are not closed in teardown, subsequent cell executions may re-use the created webdrivers until kernel is shutdown.

Maybe variable autocompletion could use those sticky webdriver sessions look the available selectros directly from the browser?

I was not aware of importnb. For parsing robot notebooks, why didn't the same TestCaseString parser work that robotkernel uses? https://github.com/datakurre/robotkernel/blob/master/src/robotkernel/model.py

from robotkernel.

@bollwyvl Let me elaborate that "sticky webdriver sessions" option for element autocompletion:

When cell execution with test cases is completed, I save those SeleniumLibrary webdriver sessions that remain open, including the info of the current session. That sessions remains open as long as kernel is running. Kernel shutdown closes the sessions. When new cell with test cases is executed, I populate SeleniumLibrary webdriver cache (and current webdriver) from the kernel.

So, user could start with a dummy test with just Open browser or Go to and execute it. The opened browser remains open after shell execution. Kernel knows the current window and has access to it. And the completion call could make synchronous webdriver calls to get a list of available selectors (eg. list all ids to autocomplete id:-selector) from the open browser.

from robotkernel.

Would you also be able to stay for the sprints on Friday?

If I'm crossing an ocean, I bet I can swing the full week.

why didn't the same TestCaseString parser work that robotkernel uses

Oh, that's one of the things that I haven't changed much, though I ended up adding a couple more things, and split them out as a set of in-memory string-based things that might eventually be worth proposing for inclusion in RF core.

The bigger issue becomes robot notebooks importing robot notebooks importing... with importnb, this is handled by patching the actual FileFinder in python itself, so it works without much innovation on my part, as everything ends up in a module. In the robot case, as additional magics become available, the kernel needs to be able to evaluate each cell as things get parsed.

"sticky webdriver sessions"

That's a pretty exciting case. Will check it out!

For library-by-library features (RESTInstance, SeleniumLibrary, etc.), I would see those as hooks into lifecycle events of the in-memory robot and the kernel... similar to the %reload_ext magic in IPython, and the various mechanims it can use.

There are advantages and disadvantages to a monorepo with a contrib folder: it's great if you can test them all together, but I wouldn't want to force someone doing AutoIt to install Selenium stuff, for example. Lots to think about!

from robotkernel.

The bigger issue becomes robot notebooks importing robot notebooks importing...

Ok. For me this sounds like that robot notebooks could be handled as Resources in Robot Framework and they might work with implementing ipynb parsing support into Robot Framework itself. By following the implementation of ReST-parser it should not actually be too much of work (it is a bit similar in that it parses ReST-files with Docutils and then just exracts contents of .. code:: robotframework-directives. Here we would extract contents of robot cells.

from robotkernel.

Auto-completion for simple selectors works quite ok

https://www.youtube.com/watch?v=YFIUP_Eno_U

I guess, you already have auto-completion for variables.

from robotkernel.

from robotkernel.

Logical next step for selector auto-completion is probably to let user pick the element directly from the browser, though, UX is not trivial: https://www.youtube.com/watch?v=-gU4srzslpc

from robotkernel.

from robotkernel.

from robotkernel.

Already using simmerjs in that picker :)

I also looked that it might be possible to hook into JupyterLab completion menu in labextension and highlight elements while going through the options, but I left that for later... (picker was hanging lower...)

from robotkernel.

from robotkernel.

@bollwyvl With this robotframework branch it is possible to run "robot something.ipynb" or use "Resource something.ipynb" https://github.com/datakurre/robotframework/tree/feature/nbsupport

Probably it needs support for multikernels, but I'm not sure do they have standard way to define the language of the current cell.

Getting that upstream would need proper testing and documentation, but probably possible.

from robotkernel.

from robotkernel.

from robotkernel.

from robotkernel.

@bollwyvl I realised that I needed this to run robot suites from notebooks without exporting them at first, so I added the simplest version into kernel for now efbf41e with "nbrobot" and "nblibdoc" -entrypoints that enable the nbsupport for parsing.

Using markdown cells for documentation is interesting.

For more generic documentation I'm usually using Sphinx, where I do have https://pypi.org/project/sphinxcontrib-robotframework/ for generating screenshots from inline robot scripts. Although, I've got mixed response from inline robot screenshots (some would prefer them to be separated to keep documentation sources easier to read and maintain).

from robotkernel.