smartbiotech / pbrcontrol Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 1.0 13.92 MB

Python 99.50% Shell 0.50%

pbrcontrol's People

Contributors

Stargazers

Watchers

Forkers

mvalle21

pbrcontrol's Issues

Parameter for outlier tolerance change when is pump on

In PhenometricsPumpManager, the lower outlier tolerance is changed when pump is turned on (L128) and changed back when pump is turned off (L148). Now the current value is multiplied by a constant (number 2 particularly). This should be generalised to the multiplication by a variable defined in the protocol as pump_on_outlier_factor (feel free to come up with a better name).

Response 200 when node is not running

From the documentation it is clear that when a node is not running, the response 400, 'Node_id not provided' should be returned. Instead, response 200, [] is obtained.

How to replicate:

start PBRcontrol
call this:

>>> r = requests.get('https://localhost:5000/log?node_id=1', verify=False, auth=('BioArInEO', 'sybila'))
>>> print(r.status_code, r.text)
200 []

Ping not working

Ping is not working anymore. Also its documentation is not sufficient!

[2020-03-06 22:08:34,649] ERROR in app: Exception on /ping [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/dist-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.7/dist-packages/flask_restful/__init__.py", line 468, in wrapper
    resp = resource(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/flask/views.py", line 89, in view
    return self.dispatch_request(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/flask_restful/__init__.py", line 583, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/bioarineo/PBRcontrol/DBmanager/DBapp.py", line 289, in get
    response['jvm'] = JVMController.isJVMStarted()
AttributeError: 'function' object has no attribute 'isJVMStarted'

Problem with /end

Got response 200 after executing this code:

requests.get('https://localhost:5000/end?node_id=1', verify=False, auth=('BioArInEO', 'sybila'))

Yet there was an exception on the PBRcontrol server

Exception in thread 1PBR-checker:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/xtrojak/PycharmProjects/PBRcontrol/DataManager/executioner.py", line 55, in run
    device.end()
  File "/home/xtrojak/PycharmProjects/PBRcontrol/DataManager/base_interpreter.py", line 27, in end
    self.log.update_log(datetime.datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S"), self.device_details['node_id'], response)
TypeError: update_log() missing 4 required positional arguments: 'command_id', 'target_arguments', 'response', and 'source'

Protocol checker only works with NodeInitiation endpoint

The checker is not ready to be used for other endpoints (AddDevice)

By restarting PBRcontrol we lose information

I'm not sure whether this is critical, so it has probably low priority.

Anyway... in case we restart the PBRcontrol, it automatically loses information about which data have already been checked. It is natural since this information is only stored "in code" when it is running. Then when get_log is called again, all available data from DB are provided.

This could be solved in two ways:

the information is stored in DB
it could be solved on Bioarineo side - when connection stops working I will remember the last time stamp when I obtained some data, if I reconnect I will call get_log with the time stamp.

For solution 2, however, we would need some "ping" function which I could use to check whether the PBRcontrol is accessible again (pinging the server is not enough, it does not imply the tool is running too).

Any other ideas or preferences?

set_pump_state might fail

The periodical measurement calls set_pump_state to turn on/off the pump. However, the case when this command is not successful (indicated by False return value), the program is not prepared and it continues without checking whether the command was successful.

Ping on PBRcontrol API

Implementation of a separate resource on the API which would return data about the current state of PBRcontrol should be implemented.

Possible response codes:
<200> -> indicates that the API can respond to other requests
<404> -> indicates that the API is not running

Could be later used for reaching for data about active nodes - this feature must be delayed until Issue #6 is resolved.

ePBR pump usage

Compared to PSI PBR, the change of the state of pump in ePBR is more complicated. It cannot be simply switched on and off. The pump works as following:

it is turned on and additional medium is added to the bioreactor (takes 20 seconds)
then the pump is turned off and another pump extracts extra medium from the bioreactor

The problem with point 2. is that the extraction pump is possibly shared with other devices.

Considering the given information, we propose the following implementation of set_pump_state for ePBR (has to be verified)

OD = measure_od()
while OD > min_OD:
     setAux1(1)             # this turns on the pump
     time.sleep(40)        # sleep 40 seconds, should be enough to accomplish steps 1. and 2.
     OD = measure_od()

such that min_OD is given to ePBR during initialisation.

Question for Jan:

We are still not sure how correctly use setAux1. It takes as argument 0/1, which should turn off/on the pump. As described above it is quite clear how setAux1(1) should be used. What is not clear, however, is whether after the "40 seconds" the state is set back to 0. Also it is not clear whether setAux1(0) has any effect at all - and if it does, what does it actually mean in the context of steps described above and on the pseudo-code provided above.

Get log - time stamp ignored for manual (external) records

Having some manual (source = "external") records in log table, the execution of

t = datetime.datetime.utcnow().strftime("%Y%m%d%H%M%S")
requests.get('https://localhost:5000/log?node_id=1&time=' + t, verify=False)

returns all records with time stamp after the given time t but also all manual records with time stamp before the given time t.

Missing PBR commands

There are some additional commands in AbstractPBR which are not reflected in interpreterPBR !

Init time not working properly

Getting data from log with a time stamp (parameter &time= in URL) returns also data with earlier time stamp (the difference is several minutes).

Power outage restoration

In the case of power outage, it would be great if PBRcontrol could be restored to the previous state. This includes two steps:

start PBRcontrol API on system startup
restore all running experiments

Step 2 is quite tricky, it requires to add more state logging to DB such it is possible to recognize whether an experiment was running before PBRcontrol was terminated and also which node with particular settings were running.

Relevant to issue #11.

Authentication

Currently it is unclear how the authentication process will be provided in the final product. This should be clarified. Also any login details should be stored in a DB/config file on the side of the server, not in the code.

I can see some work on this was done:

DBmanager/DBapp.py, L18 - L19:

self.username = api.app.config['USERNAME']
self.password = api.app.config['PASSWORD']

yet it wasn't documented.

/log retrieves incorrect data in the first time-unspecific request (under certain conditions)

The problem is that after the initial request with time specification is sent, PBRcontrol must remember the log_id of the last entry in database it retrieves so that it can continue from that point in the table when the following non-time-specific requests are made. However, if there are no data yet since the init time, the log_id PBRcontrol remembers will default to 0 and in the next, non-time-specific request, ALL data for the node from the table will be fetched.

inf and nan in measurement values

Seems like some implementations of PBR might return nan or inf as a value of a measurement. This should be fixed and recognised as an error, these values cannot be processed by Bioarineo.

Multidevice support

Multidevice support of Java devices is not working (as expected). Problem seems to be in starting JVM multiple times. Follows the request with data sent & error message:

import datetime
node = {
    1 : {
        'experiment_details' : {'sleep_time' : 20},
        'devices' : [{
            'device_type' : 'PBR',
            'device_class' : 'PSI_java',
            'device_id': 'PBR01',
            'host_address': '/dev/ttyUSB0',
            'setup' : {
                'initial_commands' : [],
                'lower_outlier_tol' : 2,
                'upper_outlier_tol' : 3,
                'max_outliers' : 6,
                'min_OD' : -100,
                'max_OD' : 100,
                'pump_id' : 1,
                'ft_channel': 5,
                'OD_channel': 1
            }
        },
        {
            'device_type': 'GAS',
            'device_class': 'PSI_java',
            'device_id': 'GAS01',
            'host_address': '/dev/ttyUSB1',
            'setup': {
                'initial_commands': []
            }
        },
        {
            'device_type': 'GMS',
            'device_class': 'PSI_java',
            'device_id': 'GMS01',
            'host_address': '/dev/ttyUSB2',
            'setup': {
                'initial_commands': []
            }
        }]
    }
}

response = equests.post('https://localhost:5000/initiate', str(node), verify=False, auth=('BioArInEO', 'sybila'))
print(response)
'{"1": {"PBR": 1, "GAS": 1, "GMS": 1}}\n'

127.0.0.1 - - [03/Mar/2020 10:20:45] "POST /initiate HTTP/1.1" 200 -                                                                                                                                                                          
Exception in thread 1-PSI_java-GMS-checker:                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                                            
  File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner                                                                                                                                                                       
    self.run()                                                                                                                                                                                                                                
  File "/home/bioarineo/PBRcontrol/DataManager/executioner.py", line 48, in run
    device = interpreter.DeviceManager(*arguments)  # initiate the physical device and its interpreter
  File "/home/bioarineo/PBRcontrol/DataManager/interpreterGMS.py", line 7, in __init__
    super(DeviceManager, self).__init__(device_details, device_class, log)
  File "/home/bioarineo/PBRcontrol/DataManager/base_interpreter.py", line 17, in __init__
    self.device = device_class(*args)
  File "/home/bioarineo/PBRcontrol/HWdevices/PSI_java/GMS.py", line 6, in __init__
    super(GMS, self).__init__(ID, address, "HWdevices/PSI_java/lib/config/device_GMS.config")
  File "/home/bioarineo/PBRcontrol/HWdevices/PSI_java/Device.py", line 17, in __init__
    self.device = self.connect(device_config)
  File "/home/bioarineo/PBRcontrol/HWdevices/PSI_java/Device.py", line 22, in connect
    JVMController.startJVM()
  File "/usr/local/lib/python3.7/dist-packages/jpype/_core.py", line 164, in startJVM
    raise OSError('JVM cannot be restarted')
OSError: JVM cannot be restarted

/usr/local/lib/python3.7/dist-packages/jpype/_core.py:217: UserWarning:
-------------------------------------------------------------------------------
Deprecated: convertStrings was not specified when starting the JVM. The default
behavior in JPype will be False starting in JPype 0.8. The recommended setting
for new code is convertStrings=False.  The legacy value of True was assumed for
this session. If you are a user of an application that reported this warning,
please file a ticket with the developer.
-------------------------------------------------------------------------------

  """)
Mar 03, 2020 10:20:48 AM psi.bioreactor.core.config.Config open
INFO: Config file opened: HWdevices/PSI_java/lib/config/device_GAS.config propCount=2
Mar 03, 2020 10:20:49 AM psi.bioreactor.core.config.Config open
INFO: Config file does not exists: HWdevices/PSI_java/lib/config/device_GAS.session
Mar 03, 2020 10:20:49 AM psi.bioreactor.message.command.CommandDefs initialize
INFO: Loading command manifests from HWdevices/PSI_java/lib/commands/commands_GAS...
Mar 03, 2020 10:20:49 AM psi.bioreactor.core.plugin.PluginManager loadPlugins
INFO: Loading plugins...
Mar 03, 2020 10:20:49 AM psi.bioreactor.core.plugin.PluginManager addDefaultServiceLoaders
INFO: Adding built-in plugins loader
Mar 03, 2020 10:20:49 AM psi.bioreactor.core.plugin.PluginManager addDefaultServiceLoaders
INFO: Searching plugins in HWdevices/PSI_java/lib/jar
Mar 03, 2020 10:20:49 AM psi.bioreactor.core.plugin.PluginManager loadPlugins
INFO: Loading plugins...
Mar 03, 2020 10:20:49 AM psi.bioreactor.core.plugin.PluginManager addDefaultServiceLoaders
INFO: Adding built-in plugins loader
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x76072bf0, pid=2165, tid=2191Mar 03, 2020 10:20:49 AM psi.bioreactor.core.plugin.PluginManager addDefaultServiceLoaders                                                                                              
INFO: Searching plugins in HWdevices/PSI_java/lib/jar

#
# JRE version: OpenJDK Runtime Environment (11.0.6+10) (build 11.0.6+10-post-Debian-1deb10u1)
# Java VM: OpenJDK Server VM (11.0.6+10-post-Debian-1deb10u1, mixed mode, serial gc, linux-)
# Problematic frame:
# C  [_jpype.cpython-37m-arm-linux-gnueabihf.so+0x20bf0]  JPypeTracer::trace1(char const*)+0x200
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/bioarineo/PBRcontrol/hs_err_pid2165.log
#
# If you would like to submit a bug report, please visit:
#   https://bugs.debian.org/openjdk-11
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted

After this it was not possible to reproduce error, because the devices are not reachable anymore (on EUCZ_00101)

JVM cannot be restarted

Restarting a node (eg. end and start again) resulted to the following error:

192.168.17.1 - - [24/Oct/2019 18:02:19] "POST /initiate HTTP/1.1" 200 -

Exception in thread 1PBR-checker:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/pi/PBRcontrol/DataManager/executioner.py", line 48, in run
    device = interpreter.DeviceManager(*arguments)  # initiate the physical device and its interpreter
  File "/home/pi/PBRcontrol/DataManager/interpreterPBR.py", line 49, in __init__
    super(DeviceManager, self).__init__(device_details, device_class, log)
  File "/home/pi/PBRcontrol/DataManager/base_interpreter.py", line 9, in __init__
    self.device = device_class(self.device_details['device_type'], self.device_details['address'])
  File "/home/pi/PBRcontrol/HWdevices/PSI_java/PBR.py", line 7, in __init__
    super(PBR, self).__init__(ID, address, "/home/pi/PBRcontrol/HWdevices/PSI_java/lib/config/device_PBR.config")
  File "/home/pi/PBRcontrol/HWdevices/PSI_java/Device.py", line 16, in __init__
    self.device = self.connect(device_config)
  File "/home/pi/PBRcontrol/HWdevices/PSI_java/Device.py", line 20, in connect
    "-Djava.class.path=/home/pi/PBRcontrol/HWdevices/PSI_java/lib/jar/bioreactor-commander-0.8.7.jar")
  File "/home/pi/.local/lib/python3.5/site-packages/jpype/_core.py", line 157, in startJVM
    raise OSError('JVM cannot be restarted')
OSError: JVM cannot be restarted

Remove SSL/TLS certification

Because security of connection is ensured by VPN, certification and communication over HTTPS is redundant and complicates matters on both sides of communication.

Implement device_id generator

ePBRs must have a valid ID to be able to connect - this must be provided in the initiation protocol, otherwise the experiment will not run. To ensure unity across all device classes, each device will have a unique ID assigned to it in case it is not provided.