Comments (24)
michalpokusa
commented on September 4, 2024
From the code that you provided it seems that this is a continuous stream of data. The way that a ChunkedResposne is intended to use is mainly when returning a large response to split it, so that you do not need to store the whole response in memory at any moment. If this function never returns, the chunked response will never be "finished".
Also, there is no oficial support for async at the moment, although it stil might be possible to use that.
Depending on what you are trying to achieve, you might want to try websockets, MQTT or even saving values to list and returning them on request.
Keep in mind, that CircuitPython is single threaded, and the adafruit_httpserver can handle one request as the time, and only after it is finished, it will handle the next one.
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
ty
Please could you provide some links to an example that uses websockets?
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
ty
Please could you provide some links to an example that uses websockets?
I have not worked with websockets in CircuitPython, but i know Neradoc has a library for that, you will find some examples there.
https://github.com/Neradoc/websockets-for-circuitpython
from adafruit_circuitpython_httpserver.
Neradoc
commented on September 4, 2024
Note that we don't have a websockets server implementation that I know of, only client.
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
Note that we don't have a websockets server implementation that I know of, only client.
Oh, i didn't know that. Thanks for clearing that out.
In that case MQTT might be the way to go.
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
What's the simplest, easiest and fastest way to stream data from the Pi Pico W to a client?
It must be a common use case to read data from some sensor or device, possibly cache it, and continuously graph it in a browser.
- Websockets isn't an option - no websockets on the http server - what does it take to get it implemented? Looks like it's possible in micropython https://www.donskytech.com/using-websocket-in-micropython-a-practical-example/ - time to switch?
- A hack might be to write to the local filesystem, poll this file from the browser and truncate/replace/switch it as it fills up.
- Then there is Redis - there's this for micropython https://micropython-redis.readthedocs.io/en/latest/ might not be too hard to port it to socketpools.
- MQTT looks like overkill for a simple point to point solution ...
btw - am I right in assuming that circuit python "workflows" are really circuit python kernel implementations for reading/writing data to the device? Took me forever to have the aha.
I believe there is a defect in assignment to wifi.radio.hostname. If it's reused in the same execution new values are merged and do not replace the existing value. E.g. wifi.radio.hostname = 'some-long-foo' then later wifi.radio.hostname = 'short-name' the resulting value will be: short-namefoo
I discovered this in code to uniqueify my hostname, be setting the hostname to some random string, running a getaddrinfo to find the hostname I want to use. Resolving clashes with the desired name by suffixing the name and looking again until there's no match, stop station, set the hostname to the unique value and connect again.
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
Regarding the hostname problem, I tested your scenario on ESP32-S2 and everything works as expected, I do not have a Pico W with me at the moment so can't test that. If you provide a exact snippet that you used I can you it to check again. Maybe it is a Pico specific thing.
When it comes to your project, the most appropriate solution highly depends on:
- how many Pico W will send data
- do you need historic data, like from 5minutes/2 hours ago, or only after a client connects
- do you need it in real time, <1s delay, or can it be more e.g. 10s
- do you want to store the data, or only use it at runtime
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
I really appreciate your help :-) tysm
I checked in the code here: https://github.com/adrianblakey/slot-car-data-logger
I hope the README.md answers the questions - if not please tell me
btw I am running: Adafruit CircuitPython 8.2.0-rc.1 on 2023-06-27; Raspberry Pi Pico W with rp2040
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
So, after reading the README.md and some research I found the SSE (Server-Sent Events), which seem to suit your needs. Today I managed to write experimental logic for supporting it in this repo branch, I believe it would be a good addition to lib. There are some limitations, like max number of connected client, which seems to be 1 because of the limit on TCP sockets etc.
This should work for your application, the usage is limited, but Websockets probably would have the same problems for the same reason, but we have to remember we operate on microcontrollers, not fully-featured computers.
If you happen to need help implementing SSE in your project feel free to contact me, or ask on Adafruit's Discord Server.
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
I tried the example - a very cool solution, ty.
The first GET request to /client downloads the script, which GET's /connect-client which returns the initial response, and the infinite while loop runs the send_message - forever.
However - subsequent GET's to /client (by refreshing the browser), fail - is this caused by the maximum number of connected clients issue, or is this a browser issue https://developer.mozilla.org/en-US/docs/Web/API/EventSource ?
Here's the traceback when refreshing the browser:
0;π192.168.178.89 | code.py | 8.2.0-rc.1Started development server on http://192.168.178.89:80
192.168.178.74 -- "GET /connect-client" 408 -- "200 OK" 101
192.168.178.74 -- "GET /client" 510 -- "200 OK" 474
Traceback (most recent call last):
File "/lib/adafruit_httpserver/server.py", line 351, in poll
File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request
File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler
File "code.py", line 60, in connect_client
File "/lib/adafruit_httpserver/response.py", line 483, in close
File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes
BrokenPipeError: 32
Traceback (most recent call last):
File "code.py", line 67, in <module>
File "/lib/adafruit_httpserver/server.py", line 380, in poll
File "/lib/adafruit_httpserver/server.py", line 351, in poll
File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request
File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler
File "code.py", line 60, in connect_client
File "/lib/adafruit_httpserver/response.py", line 483, in close
File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes
BrokenPipeError: 32
0;π192.168.178.89 | 117@/lib/adafruit_httpserver/respons BrokenPipeError | 8.2.0-rc.1
Code done running.
def connect_client(request: Request):
response = SSEResponse(request)
if connected_client.response is not None:
connected_client.response.close() # Close any existing connection <<<= line 60
connected_client.response = response
return response
The connected_client.response is obviously not None but it's in some state that does not allow it to be closed. CircuitPython is happy to recover by soft rebooting - but ...
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
I am unable to replicate thsi error, which browser are you using? It also might be the Pico thing, I am using ESP32-S2 Feather and it simply switches to new client.
EDIT1: Also, I am using stable CP 8.1.0, try with this version, maybe something changed in 8.2.0.
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
It's lovely Chrome Version 114.0.5735.198 (Official Build) (64-bit) on ArchLinux _ suppose I could give it a shot w 8.1.0 and report back ...
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
With 8.1.0 - it now behaves differently
The REPL ...
0;π192.168.178.89 | code.py | 8.1.0Started development server on http://192.168.178.89:80
192.168.178.74 -- "GET /connect-client" 408 -- "200 OK" 101
connected client response <SSEResponse object at 0x2002eda0>
192.168.178.74 -- "GET /connect-client" 408 -- "200 OK" 101
connected client response <SSEResponse object at 0x2001aef0>
...
and in the browser console ...
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
So to make this clear, you try to connect multiple clients, and there is no error on server side, but on both clients you get errors?
The CONNECTION_REFUSED is a bit diffrent from what i expected, but since example allows only one client and switches between them, it is normal that clients throw erros, as they in fact are constantly disconnected and connected again, like they are fighting for their place.
The output I get is on both clients is 2-3 "Event data:...", then error, and then it repeats.
Nevertheless, it kind of proves that there was in fact a change in CP itself.
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
I reran the test again. I noticed I had some tabs open before that were also submitting the requests thanks to Chrome's persistence...
The test is:
- ctl-D to start the code.py
- http://192.168.178.89/client (F12 to open the console display in the browser) - events appear correctly
- Open new browser tab
- Run another http://192.168.178.89/client
- Browser console errors: SSE error: EventΒ {isTrusted: true, type: 'error', target: EventSource, currentTarget: EventSource, eventPhase: 2,Β β¦} every 2/3 responses in both browser tabs
- REPL shows ...
192.168.178.74 -- "GET /connect-client" 378 -- "200 OK" 101
connected client response <SSEResponse object at 0x20026330>
192.168.178.74 -- "GET /connect-client" 408 -- "200 OK" 101
connected client response <SSEResponse object at 0x2002cc60>
If I repeat the same test of refreshing the browser it behaves the same way as 8.2.0 :-)
192.168.178.74 -- "GET /client" 506 -- "200 OK" 474
192.168.178.74 -- "GET /connect-client" 430 -- "200 OK" 101
192.168.178.74 -- "GET /client" 510 -- "200 OK" 474
connected client response <SSEResponse object at 0x2001da00>
Traceback (most recent call last):
File "/lib/adafruit_httpserver/server.py", line 351, in poll
File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request
File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler
File "code.py", line 61, in connect_client
File "/lib/adafruit_httpserver/response.py", line 483, in close
File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes
BrokenPipeError: 32
Traceback (most recent call last):
File "code.py", line 68, in <module>
File "/lib/adafruit_httpserver/server.py", line 380, in poll
File "/lib/adafruit_httpserver/server.py", line 351, in poll
File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request
File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler
File "code.py", line 61, in connect_client
File "/lib/adafruit_httpserver/response.py", line 483, in close
File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes
BrokenPipeError: 32
0;π192.168.178.89 | 117@/lib/adafruit_httpserver/respons BrokenPipeError | 8.1.0
Code done running.
Press any key to enter the REPL. Use CTRL-D to reload.
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
If every browser tabs error every 2/3 successfull event data logs, then it works as expected, because as I said before, the clients are both fighting for the only one space.
When it comes to BrokenPipeError, I cannot replicate it. It may be browser, CP or Pico related, do you have any other microcontroller you can test that on?
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
Is there something I could do to gather more data about the BrokenPipeError and debug it? This is the only mcu I have.
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
Nothing comes to my mind right now.
Considering we are both using the same code, browser and CP version, I say it is a CP implementation thing. It would not be the first time, in the past there were also some diffrences in socket behavior between ESPs and Pico.
In my opinion, you could make an issue on CP itself about that, it probably is causesd by missing the close on socket or something similar.
Another way, although a bit hacky, is supressing this error, of course only it your program works doing so.
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
btw - here's the Firefox behavior
0;π192.168.178.89 | code.py | 8.1.0Started development server on http://192.168.178.89:80
192.168.178.74 -- "GET /client" 344 -- "200 OK" 474
192.168.178.74 -- "GET /connect-client" 336 -- "200 OK" 101
192.168.178.74 -- "GET /favicon.ico" 298 -- "404 Not Found" 90
192.168.178.74 -- "GET /client" 344 -- "200 OK" 474
connected client response <SSEResponse object at 0x2001bb40>
Traceback (most recent call last):
File "/lib/adafruit_httpserver/server.py", line 351, in poll
File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request
File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler
File "code.py", line 61, in connect_client
File "/lib/adafruit_httpserver/response.py", line 483, in close
File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes
OSError: [Errno 9] EBADF
Traceback (most recent call last):
File "code.py", line 68, in <module>
File "/lib/adafruit_httpserver/server.py", line 380, in poll
File "/lib/adafruit_httpserver/server.py", line 351, in poll
File "/lib/adafruit_httpserver/server.py", line 283, in _handle_request
File "/lib/adafruit_httpserver/route.py", line 154, in wrapped_handler
File "code.py", line 61, in connect_client
File "/lib/adafruit_httpserver/response.py", line 483, in close
File "/lib/adafruit_httpserver/response.py", line 117, in _send_bytes
OSError: [Errno 9] EBADF
0;π192.168.178.89 | 117@/lib/adafruit_httpserver/respons OSError | 8.1.0
Code done running.
Press any key to enter the REPL. Use CTRL-D to reload.
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
Try wrapping the _close_connection in SSEResponse or the body od _close_connection itself in try except:
def _close_connection(self) -> None:
try:
self._request.connection.close()
except:
passMaybe Pico closes the socket on it's own or other dark magic causes this. Β―\(γ)/Β―
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
Back on 8.2.0 rc1. Yes the connection does get closed ... so this seems to do the trick
def _close_connection(self) -> None:
try:
self._request.connection.close()
except (BrokenPipeError, OSError) as e: # Client closed the connection already
pass
and
def close(self):
"""
Close the connection.
**Always call this method when you are done sending events.**
"""
try:
self._send_bytes(self._request.connection, b"event: close\n")
self._close_connection()
except (BrokenPipeError, OSError) as e: # Client closed the connection already
pass
and on send_event - about line 477
try:
self._send_bytes(self._request.connection, message.encode("utf-8"))
except (BrokenPipeError, OSError) as e: # Might fail due to another open browser tab
pass
Interesting behavior -
- Start 1 tab.
- Refresh this tab - all is now fine.
- Start another tab - crashes the VM, which does a soft reboot and services the new tab - not the old.
- Refresh the first and second tab and they now (sort of) cooperate emitting say 2-3 events each before giving an error
How do we get the VM crash fixed?
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
What do you mean by VM?
from adafruit_circuitpython_httpserver.
adrianblakey
commented on September 4, 2024
The python interpreter.
When the code has an issue or the interpreter has an issue doesn't it reboot (which is what happens the first time new tab is opened and the request submitted)? Maybe there is another issue with python code that throws a runtime excp that's not being caught and I am mistaken ...
Also, is there any way to speed up the rate at which the events are fired (send_message is called)? Seems like it occurs about once a second - I need this at about every 1/10 sec. Could I go back to playing games with asyncio, caching say 10 events and emitting them from the cache in send_message?
from adafruit_circuitpython_httpserver.
michalpokusa
commented on September 4, 2024
Regarding more frequent messages, yes, you can increase that, line 28 on example:
self.next_message = monotonic() + <time in seconds here>
from adafruit_circuitpython_httpserver.
Related Issues (20)
- Create an httpserver_simpletest example HOT 7
- How do I safely disable all access to filesystem? HOT 1
- Rapid requests causing hangs HOT 2
- PyPi Release not importable
- Setting the hostname - Pico W HOT 2
- mdns setting error - trying to set a custom hostname HOT 5
- Release notes has bad link to Docs HOT 2
- Submitting empty form, or forms with only checkboxes, causes crash in web server HOT 2
- request.form_data.get(...) returns the string "None" if a field doesn't exist HOT 2
- "ImportError: no module named 'hashlib'" on the WIZnet W5500-EVB-Pico HOT 7
- instructions error HOT 1
- Simple Test (examples/httpserver_simpletest_auto.py) error on M5stack Timer Camera X HOT 2
- CircuitPython 9.0.0 will require SO.REUSEADDR for socket reuse HOT 1
- Airlift Support? HOT 11
- HTTPS Server now works on ESP32-S3 but MemoryError on Pico W and ESP32-S2
- Pico W EOPNOTSUPP On CircuitPython 9.0.0 HOT 2
- HTTPServer stops working on CircuitPython version >= 9.0 HOT 3
- RPI Pico W Circuitpython Webserver on AP HOT 5
- add support for "multipart/x-mixed-replace" (for streaming video) HOT 17
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from adafruit_circuitpython_httpserver.