Tasks

run & test requirements ?

Tasks/Steps/Elements

This is the task to make a (bash or maybe better python) script that installs the spyder-remote-server daemon on the OS.

We are at this point in time only interested in Linux, and specifically Ubuntu 20.04. (other Linuxes that might be a bit different have to be added in community work)

(SCT) me@mybox:~$ conda install spyder-remote-server
... 
(SCT) me@mybox:~$ sudo spyder-remote-server --install
...
(SCT) me@mybox:~$ sudo spyder-remote-server --uninstall

Versions checklist

• Spyder version: >= 5
• Python version: 3.7, 3.8
• Qt version: all
• PyQt version: all
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows/Linux/macOS

Tasks/Steps/Elements

Versions checklist

• Spyder version: >= 5
• Python version: 3.7, 3.8
• Qt version: all
• PyQt version: all
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows/Linux/macOS

Reproducing the failure:

When launching the spyder-remote-server, and on my local machine opening spyder (dependencies see below), I do
Console/new remote console.
The popup nicely detects my remote host (Tom's RPi) and the guest account

but if I now click connect I get the below message :

Note : we install python no longer with anaconda, but with maxiconda as per www.maxiconda.org

• I added the GUEST user, with password GUEST (all in capital, had to add the --force-badname flag to do so)
• I installed maxiconda for the GUEST user
• I installed maxiconda for the root user, and in the base environment I also installed spyder-remote-server
• I did a sudo su root, install the deamon with a spyder-remote-server install GUEST
• I started the deamon from a root prompt with spyder-remote-server start

Spyder dependencies

Mandatory:

atomicwrites >=1.2.0 : 1.4.0 (OK)
chardet >=2.0.0 : 4.0.0 (OK)
cloudpickle >=0.5.0 : 2.0.0 (OK)
cookiecutter >=1.6.0 : 1.7.3 (OK)
diff_match_patch >=20181111 : 20200713 (OK)
intervaltree >=3.0.2 : 3.0.2 (OK)
IPython >=7.6.0 : 7.27.0 (OK)
jedi >=0.17.2;<0.19.0 : 0.18.0 (OK)
jsonschema >=3.2.0 : 3.2.0 (OK)
keyring >=17.0.0 : 23.2.1 (OK)
nbconvert >=4.0 : 6.1.0 (OK)
numpydoc >=0.6.0 : 1.1.0 (OK)
paramiko >=2.4.0 : 2.7.2 (OK)
parso >=0.7.0;<0.9.0 : 0.8.2 (OK)
pexpect >=4.4.0 : 4.8.0 (OK)
pickleshare >=0.4 : 0.7.5 (OK)
psutil >=5.3 : 5.8.0 (OK)
pygments >=2.0 : 2.10.0 (OK)
pylint >=2.5.0;<2.10.0 : 2.9.6 (OK)
pyls_spyder >=0.4.0 : 0.4.0 (OK)
pylsp >=1.2.2;<1.3.0 : 1.2.2 (OK)
pylsp_black >=1.0.0 : None (OK)
qdarkstyle =3.0.2 : 3.0.2 (OK)
qstylizer >=0.1.10 : 0.2.1 (OK)
qtawesome >=1.0.2 : 1.0.3 (OK)
qtconsole >=5.1.0 : 5.1.1 (OK)
qtpy >=1.5.0 : 1.11.1 (OK)
rtree >=0.9.7 : 0.9.7 (OK)
setuptools >=49.6.0 : 58.0.4 (OK)
sphinx >=0.6.6 : 4.2.0 (OK)
spyder_kernels >=2.1.1;<2.2.0 : 2.1.1 (OK)
textdistance >=4.2.0 : 4.2.1 (OK)
three_merge >=0.1.1 : 0.1.1 (OK)
watchdog >=0.10.3 : 2.1.5 (OK)
zmq >=17 : 22.3.0 (OK)

Optional:

cython >=0.21 : 0.29.24 (OK)
matplotlib >=2.0.0 : 3.4.3 (OK)
numpy >=1.7 : 1.21.2 (OK)
pandas >=1.1.1 : 1.3.3 (OK)
scipy >=0.17.0 : 1.7.1 (OK)
sympy >=0.7.3 : 1.8 (OK)

Spyder plugins:

ATE.spyder.plugin 0.1.27 : 0.1.27 (OK)
spyder_remote_client.spyder.plugin 0.0.24 : 0.0.24 (OK)
spyder_screencast.plugin 0.0.17 : 0.0.17 (OK)
spyder_terminal.terminalplugin 1.1.0 : 1.1.0 (OK)

Tasks/Steps/Elements

Versions checklist

• Spyder version: >= 5
• Python version: 3.7, 3.8
• Qt version: all
• PyQt version: all
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows/Linux/macOS

Tasks

We need a conda-forge feedstock from PyPi

Task

The configuration file for spyder-remote (server) should be in /etc.
... maybe the correct name should be spyder-remote-server.ini ?

Tasks/Steps/Elements

https://docs.python.org/3/library/uuid.html

Versions checklist

• Spyder version: >= 5
• Python version: 3.7, 3.8
• Qt version: all
• PyQt version: all
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows/Linux/macOS

Tasks

This Repo holds 2 conda packages:

• spyder-remote-server
• spyder-remote-client

spyder-remote-client is a straight forward package implementing a spyder plugin.

spyder-remote-server is a bit different, in that it provides a script (in /bin or something like that)
so that one installs and enables the package as follows:

(base) me@mybox:~$ conda install spyder-remote-server
(base) me@mybox:~$ sudo spyder-remote-server --install

Prerequisites

• Issue#1 : zeroconf needs to be finished
• issue#2 : spyder-remote needs to be finshed
• Issue#4 : spyder-remote-server daemon needs to be finished
• Issue#5 : spyder-remote-server script needs to be finished

Anyway this task is to build the 2 packages.

Versions checklist

• Spyder version: >= 5
• Python version: 3.7, 3.8
• Qt version: all
• PyQt version: all
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows/Linux/macOS

Tasks

Let's try to stick to only one entry in the IPython Console hamburger, and fit everything in the New Remote Console entry point.

Once the (only) entry point is selected, we are presented with the following dialog:

At first, the spyderHosts QComboBox current item is empty, but the drop-down list is filled with all systems we find on zeroconf.
Credentials and Conda QGroupBox-es are disabled.

The user has no other option then to select a remote spyder host. Once he did do so, the Credentials QGroupBox enables.

The user QCombobox is filled with:

• a list of all users we logged into this remote host with in the past
• if the remote host has guest account enabled, this one is also in the list.
• the last used user is the current item.

The password QLineEdit initially will read 'from keyring' in the placeholder text. (as the keyring will know the last password we used). If keyring is disabled (keyring QCheckBox is unchecked) the placeholder text is empty (indicating we need a password)
If keyring is enabled, but the user has nothing in the keyring, again the placeholder text is empty.

As soon as user and password are supplied, a login attempt will be made to see if login/password is valid. If not, the feedback line notifies the user that login/password is invalid. If it is valid, the Conda QGroupBox enables.

The condaEnvironments QComboBox will be filled with the available conda environments for the logged in user. (note that conda is guaranteed to exist, as the whole system is installed over conda, however we can in principle log in with a username that is not added to the conda group on the remote system ... in such case the Conda QGroupBox is deactivated, and the feedback line states that the user isn't 'conda-enabled'.

If the user is the gues-user, the condaEnvironments QComboBox is filled with the environments made available over zeroconf (so some enviroments can be excluded!).

By default the condaEnvironments QComboBox-es current item is the project name (if a project is open), if no project is open, the current item is an empty string. It is possible that the user can define another environment name.

If a project is open, the plugin will look for a requirements folder in the root of the project. If it is found, the requirements QComboBox is filled with the entries found there. if a requirements file is present that starts with run, then that one is activated (made the current item). In case of multiple files that start with run, we are again in the case where none is made the current item.

The findRequirements QToolButton (text is '...') presents us with a default -search-for-a-file- dialog, with which one can search the file system for a requirements file.

if the 'Connect' Button is clicked, the filled-in info is transmitted to the remote host, and he will:

1. if the given conda environment doesn't exist:

• activate the base envirionment
• execute : conda create -n + the python version if available in the REQUIREMENTS_FILE

2. execute:

• execute : `conda install -n --file <REQUIREMENTS_FILE>
• execute : `conda activate
• execute : jupyter --runtime-dir and recuperate the RUNTIME_DIR from the stdout.
• execute : python -m spyder_kernels.console and recuperate the name from the .json file mentioned on the stdout.
• send the .json file from the RUNTIME_DIR to the client

3. the client will do the necessary with the .json to connect transparently to the remote spyder console.

any errors from the remote side are also transmitted to the client side to be displayed. If no errors, the dialog closes and the remote console becomes available.

Notes:

• All login's need saving in the user's spyder config (~/.config/...) for future use.
• PLEASE 🙏 use the .ui and in the code the uic to compile & load the dialog dynamically (iow: don't construct the dialog by hand)

References:

• https://docs.spyder-ide.org/ipythonconsole.html

Task:

• client side (UI that communicates with the server side and starts the remote IPython Console, as a plugin)
• server side (as a process)

Versions checklist

• Spyder version: >= 5
• Python version: 3.7, 3.8
• Qt version: all
• PyQt version: all
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows/Linux/macOS

Tasks

all github actions stuff that is not included in issue#3 : package building

• commit & PR test stuff
• release stuff (pypi & feedstock)

TBD: What PyPi account will we use ?!?

Prerequisite

Versions checklist

• Spyder version: >= 5
• Python version: 3.7, 3.8
• Qt version: all
• PyQt version: all
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows/Linux/macOS

Tasks

Whenever a system starts, spyder-remote-server should be started by system.d

so I presume there must be a spyder-remote-server.service file in /etc/systemd/system/

not sure how you make it so that the used environment is base ... something like

conda run -n base spyder-remote-server

... or setting the PATH variable ? (smells like a hack)

be sure to test that we can do:

sudo systemctl stop spyder-remote-server
sudo systemctl start spyder-remote-server
sudo systemctl restart spyder-remote-server

by hand

Anyway, this means that spyder-remote-server is running as root (so it has access to /var/log) spyder-remote-server#20

Tasks

Take the time to familiarize yourself with the zeroconf library.

I would like to use this library because:

• it is compatible with both Bonjour and Avahi but doesn't depend on either of them. (~portability)
• the eventloop is build in (~ease of use)
• is pure python (noarch ~ portability)

Do so by playing with -, and improving on discoverer.py (client side) and publish.py (server side).

Points of attention:

• re-publish. A publish is done with a (default) TTL (Time To Live), before this time expires we need to re-publish automatically.
• if our server side is going down, an un-publish HAS to be done, so the clients can pick up on it. (not sure if __del__ suffices, maybe we need also atexit?)
• by default zeroconf publishes on all local interfaces ... not sure if that is a good idea ... maybe we should only publish on the active external interfaces (and maybe the local loopback) ... needs some more thinking ... in any case, if you stumble over netifaces, that is what that is all about 😉

In order not to have surprises afterwards, use 2 physical boxes in your network segment, one being a Raspberry Pi 4B.

This Raspberry Pi holds an Ubuntu 20.04 Linux (install see References below)

References:

• Download page for Ubuntu 20.04/Raspberry Pi ➜ use the 64 Bit version as the tester will also have the 64 Bit version!
• Best active cooling case for Raspberry Pi 4b (tried many, this is the best one, especially if you overclock)
• Ubuntu 20.04 Install on RPi 4b video
• our conda install procedure for RPi and later the tester.

Versions checklist

• Spyder version: >= 5
• Python version: 3.7, 3.8
• Qt version: all
• PyQt version: all
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows(1)/Linux(2)/macOS(3)

Tasks/Steps/Elements

I've just installed the latest version of Spyder (5.0.0a3) with the spyder-remote-client plugin.
However, I don't see the two new entries in the IPython Console hamburger.

Could you please help me to resole this ?

Versions checklist

conda5_env.txt

• Spyder version: 5.0.0a3
• Python version: 3.9.1
• Qt version: 5.12.9
• PyQt version: 5.12.3
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows

(SCT8) sct@MiniSCT-1:~$ conda deactivate
(base) sct@MiniSCT-1:~$ spyder-remote-server start
(base) sct@MiniSCT-1:~$ Before daemonization
Unable to acquire PID file.

(base) sct@MiniSCT-1:~$ 

also as root this doesn't work ...

(base) root@MiniSCT-1:/home/sct# spyder-remote-server start
(base) root@MiniSCT-1:/home/sct# Before daemonization
Unable to acquire PID file.

(base) root@MiniSCT-1:/home/sct# 

Yesterday we installed and ran it for the first time, so there was no "old" PID file, now there apparently IS, and he is choking on it ... maybe a permission problem ?
In any case, I presume that spyder-remote-server start should be run as root (systemd is as root after all) right ?

Tasks

loggings should go in /var/log, the name is perfect, the owner:group should be root:root and the permissions should be 644, this means that spyder-remote-server itself runs as root, but that should be fixed by spyder-remote-server#21

Tasks

Jupyter must have a mechanism to push a file (or project directory) toward the Jupyter 'server', and afterwards pull the results back, this functionality needs building in.

We didn't foresee any time in the budget for this, so if this is a major job, we need to extend the quotation ...

We need to talk about this.

Tasks/Steps/Elements

The server-side is first implemented as a normal process, and then turned into a daemon.

Our first concern is that it runs on Linux, so one could use the stevens.py template, but for future compatibility to macOS and Windows, maybe there are better ways (read: libraries).

Task:

• wrap the server-side process (from issue#2) into a daemon process

Versions checklist

• Spyder version: >= 5
• Python version: 3.7, 3.8
• Qt version: all
• PyQt version: all
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows/Linux/macOS

Time Estimation & Quotation

1. zeroconf : 1d + 3d
2. spyder-remote : 1d + 3d
3. spyder-remote-server daemon : 1d + 1d
4. spyder-remote-server script : 1d
5. package building : 1d
6. further github actions : 1d
7. documentation : 0d †

@rgommers : 13 days @ 8h = 104h

† The only documentation that is needed is the README.md in the project root, that outlines installation. The documentation how to use it will be made in video format (=test of the QScreenCast plugin)

@rgommers ,

Please remember the new process :

• QuanSight emails me ([email protected]) the Quote. The Quote holds the following info:
  • a reference number (=your reference)
  • the email address where to send the PO (Purchase Order)
  • the individual points (like above #1..#6)with each time the amount for each point ➜ in this case it is not so important, but for future projects you can do partial invoices, just stating for which points the invoice is 😄
• I put your Quote it in our system, and about 10 days later you will receive our PO (from our purchasing department) in the stated email address of the Quote. The PO holds the following info:
  • The PO number (=our reference)
  • The Quote number (=your reference)
  • The ordered items (like above #1..#6)
  • The email address where the Invoice should go to.
• Upon reception of the PO QuanSignt sends a Confirmation to the email address where the Invoice should go to.
• When (partial) work is done, you just send your Invoice to the email address stated in the PO. (so the same address that also received the PO Confirmation!) The Invoice holds the following info:
  • The PO number.
  • The ordered items to be settled.
• The Invoice will pop-up in my SAP account for approval, I press the button, and the payment is released (after some other people also pressed their buttons, but as long as I stay in budget, that is a formality)

Tasks/Steps/Elements

Seee: https://github.com/jupyter/jupyter_client/blob/master/jupyter_client/multikernelmanager.py

Versions checklist

• Spyder version: >= 5
• Python version: 3.7, 3.8
• Qt version: all
• PyQt version: all
• Operating System name/version:
  • server : Ubuntu 20.04 Linux (aarch64 on RPI)
  • client : Windows/Linux/macOS

Document the workings of spyder-remote by means of a screencast (= test of the QScreenCast plugin)