There seems to be an incompatibility with musl related to errno (tested on OpenWrt). See details here.

(Reporting this on someone's behalf.)

With a great sadness I succumb to using "".format() as the default (and only) way to do string formatting in micropython-lib. That means that my dreams of making str.format() optional in uPy won't be realized, and instead str.__modulo__ should be made such.

The reason for that is however obvious on a mid-duration thought: str.format() allows to format with several arguments without allocating intermediate tuple. So simple.

@dpgeorge , @dhylands : FYI

Version: 0.15.-2

client = umqtt.MQTTClient("sddsdsd", "dmarkey.com", 1883)
client.connect()
Traceback (most recent call last):
File "", line 1, in
File "umqtt.py", line 84, in connect
IndexError: bytes index out of range

On my Arch Linux machine import re fails because libpcre.so.3 does not exist. Instead it's libpcre.so.1 (also aliased as libpcre.so).

So should we just change it to load libpcre.so, or try different versions in succession?

Lately, I was looking into parsing XML. Doing my homework, I looked into bunch of small XML parsers, but they all were still too bloated and insisted on well-formedness checking, and had other funky things.

I was looking into really minimal way to parse/extract useful parts from XML, and that's just working with stream of tokens. So, I wrote a simple tokenizer in Python. There's no XML tokenizer module in CPython standard lib, but I guess it would be nice addition to micropython-lib, as other XML APIs are unlikly coming soon.

Thoughts?

I propose "xmltok" for module name.

ExitStack was added in 5557382 from CPython 3.4.2 and subsequently removed in 92ef77c maybe because it wasn't compatible with uPy out of the box.

I am using a slightly modified ExitStack taken from CPython lib for some unittests which need to run on both my development host and on the target device and I am wondering if there is any interest in having ExitStack added to contextlib. Thanks

I'm trying to use urequests to get weather data from the openweathermap.org API. I'm using the following code:

import urequests
r = urequests.get("http://api.openweathermap.org/data/2.5/weather?q=Zurich&appid=XXX").json()

Unfortunately, that throws a ValueError: syntax error in JSON error. Examining the text of the response, I see that it is trucated:

{"coord":{"lon":8.55,"lat":47.37},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10d"}],"base":"cmc stations","main":{"temp":287.38,"pressure":1022,"humidity":89,"temp_min":282.04,"temp_max":292.45

compared to what I get normally with a browser or wget:

{"coord":{"lon":8.55,"lat":47.37},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03d"}],"base":"stations","main":{"temp":286.43,"pressure":1022,"humidity":86,"temp_min":282.04,"temp_max":288.71},"wind":{"speed":1.86,"deg":271.002},"rain":{"3h":0.0725},"clouds":{"all":48},"dt":1466327930,"sys":{"type":3,"id":9109,"message":0.035,"country":"CH","sunrise":1466306928,"sunset":1466364353},"id":2657896,"name":"Zurich","cod":200}

There was a lot of changes lately, time to tag an new release (after landing few more changes).

Using pip-micropython to install micropython's packages will resulting in an error.

$ pip-micropython install micropython-cgi
Warning: MICROPYPATH is not set, assuming default value
Destination library directory: /Users/latyas/.micropython/lib
/tmp/pip-micropy-venv/lib/python2.7/site-packages/pip/commands/install.py:180: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Collecting micropython-cgi
  Downloading http://pypi.doubanio.com/packages/14/a1/397dbc567329e20fbb01c9e8c4f7b122e018368696eba53daa5b0fbc8638/micropython-cgi-3.3.3-2.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    IOError: [Errno 2] No such file or directory: '/private/tmp/pip-build-Qff7fH/micropython-cgi/setup.py'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/tmp/pip-build-Qff7fH/micropython-cgi/

Seems setup.py is missing on pypi, but exists on this repo.

It would be nice to create "meta" package to let people install entire stdlib in "one go" - it would be devoid of actual Python source, but depend on all other packages belonging to stdlib.

This probably will requires some experimenting to get it right, but otherwise the biggest issue is how to name such module. Choices are:

• micropython-all - very intuitive and easy to remember, but thinking further, may get confusing, because it won't really install some abstract "all", but merely stdlib. Would also conflict with "all.py" module packaged (which is unlikely to appear in natural way, but still).
• micropython-meta-all - less intuitive, but way to disambiguate "all.py" module.
• micropython-stdlib - less intuitive than "all", but then formally correct, but conflicting with implausibke stdlib.py
• micropython-meta-stdlib - apparently least intuitive of the choices so far, but resolves other issues

Any opinions/other naming ideas?

There's growing interest to support multiple platforms in micropython-lib besides currently supported POSIX/Linux - MacOSX #7, bare-metal (PyBoard) #8, Windows micropython/micropython#886 (comment). Consider ways how to implement it efficiently.

I tried the following to bind to any adapter on the unix port running on BBB

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("", 9090))

and I got the following error

Traceback (most recent call last):
File "main.py", line 47, in
File "main.py", line 22, in run
File "/root/.micropython/lib/socket.py", line 37, in bind
File "/root/.micropython/lib/socket.py", line 15, in _resolve_addr
OSError: [addrinfo error -5]

In continuously running data logging applications the pid value can overflow the 16 bit field. In this instance struct.pack_into constrains the result to 16 bits but produces an illegal value of 0 (MQTT spec section 2.3.1).

I don't know if this is also a valid concern but cPython's struct.pack_into throws an exception on overflow: if the MicroPython implementation ever became conformant the code would fail.

Implement collections.namedtuple factory function.

Is there any C/asm optimization possible or shall we stick to pure Python version for next few months?

This causes failure in a subsequent call to subscribe() as reported here. The following (tested) fix is one option:

    def check_msg(self):
        self.sock.setblocking(False)
        result = self.wait_msg()
        self.sock.setblocking(True)
        return result

The implementation of this probably belongs to micropython-lib, but as we have more people here, I post RFC here.

So, as we go forward, uPy offers more and more features which are not compatible with CPython. But we surely want to remain good members of Python community, and provide a compatibility layer to run MicroPython-optimized programs on other Python implementations. We had beginnings of that for a long time - https://github.com/micropython/micropython/blob/master/examples/micropython.py , and this RFC is to set conventions and guidelines how to go forward with this.

My proposals:

1. Host such compatibility modules in micropython-lib repo.
2. To distinguish them from native uPy modules, have their dir names prefixed with "cpython-".
3. For PyPi, stick with existing convention of "micropython-" prefix, which them will be followed by "cpython-" infix.

As an example, CPython compatibility shim for MicroPython's "micropython" module will be possible to install from PyPi as "micropython-cpython-micropython".

I know this must be a very rookie question, but I have to ask... I've managed to install micropython on my Mac and it can also connect with my pyboard and do some basic things through REPL. What should I do if I'd like to connect the pyboard with an IMU (e.g. MPU9250) and install revelant libraries onto pyboard so that the pyboard can automatically run the programme and send data once connected to a serial port.

Many thanks!

I've already done bunch of changes towards supporting baremetal systems. More things to discuss/do:

1. Upstream pip doesn't seem to have easy programmatical interface to its functionality (it has stuff like pip.commands.install.InstallCommand, which is definitely not what we need). So, proposed interface is upip.install(package_name_or_list, path=None).
2. If path above is None, it will be deduced, specifically sys.path[1] taken for baremetal system (/lib for esp8266 e.g., others should be updated to have preferred-to-install path as sys.path[1]).
3. The idea is to use 4K dictionary space for gzip decompression. This will require repacking and re-pushing all micropython-lib packages to PyPI.

@dpgeorge , let's discuss.

For modules imported from CPython stdlib, I saw far just set version as 1.0, and then added/incremented micro version if any tweaks were later required.

But I guess, it would be more informative to set version to CPython's version from which module was imported, and then probably add revision suffix for any changes, e.g. "3.3.3-1". Hyphen in revision suffix of course assumed PyPI is ok with it (needs testing).

Comments welcome.

Hi all,

Im trying to make an HTTPS request from an ESP8266 (specifically on a nodemcu). Ive added the urequests module from this repo and am trying the following with no luck. Is HTTPS possible from the ESP8266? Do I need to change something?

>>> import urequests                                                                                                                        
>>> urequests.get('https://www.google.com')                                                                                                 
ssl_handshake_status: -261                                                                                                                  
Traceback (most recent call last):                                                                                                          
  File "<stdin>", line 1, in <module>                                                                                                       
  File "urequests.py", line 101, in get                                                                                                     
  File "urequests.py", line 56, in request                                                                                                  
OSError: [Errno 5] EIO                                                                                                                      
>>>  

Thanks, Mike

The cycle implementation does not make a copy of the elements iterated over and as such only works for proper sequences like lists, but not for all types of iterables:

from itertools import cycle

def reverse(l):
    while l:
        yield l.pop()

for x in cycle(reverse([1, 2, 3])):
    print(x)

output:

3
2
1

Proposed fix: just copy the equivalent python implementation?

def cycle(iterable):
    saved = []
    for element in iterable:
        yield element
        saved.append(element)
    while saved:
        for element in saved:
              yield element

File uploading is not supported now, is there any plan to add this feature?

For example:

micropython -m upip install micropython-sqlite3

gives following output:

Installing to: /home/teq/.micropython/lib/
Warning: pypi.python.org SSL certificate is not validated
Installing micropython-sqlite3 0.2.3 from https://pypi.python.org/packages/17/07/4f39d7c00eed557e765837c970040a6fbd225f418cb6de05016a32b81347/micropython-sqlite3-0.2.3.tar.gz
Traceback (most recent call last):
  File "upip.py", line 285, in <module>
  File "upip.py", line 278, in main
  File "upip.py", line 202, in install
  File "upip.py", line 192, in install
  File "upip.py", line 162, in install_pkg
  File "upip.py", line 88, in install_tar
  File "upip.py", line 49, in _makedirs
  File "upip.py", line 48, in _makedirs
  File "upip.py", line 44, in _makedirs
OSError: [Errno 2] ENOENT

The problem is in _makedirs function, it doesn't work with absolute path returned by get_install_path (see my commit for details).

I did a quickfix here: https://github.com/Teq/micropython-lib/commit/443c599123b7997d6482115205d44af4c74a79b1

On a raspian (IP=10.0.0.32) I installed mosquitto:
service mosquitto status
gives me: [ ok ] mosquitto is running.

An ESP8266 is in the same network:
FW: esp8266-20161017-v1.8.5.bin
'>>> sta_if.ifconfig()
gives me: ('10.0.0.105', ...)

On the raspian-terminal I can also ping my ESP8266:
ping 10.0.0.105
gives me: 64 bytes from 10.0.0.105: icmp_req=1 ttl=255 time=2.62 ms ...

But I am failing to connect with the MQTT:
'>>> from umqtt.simple import MQTTClient
'>>> c = MQTTClient("client", "10.0.0.32")
'>>> c.connect()
gives me:
Traceback (most recent call last):
File "", line 1, in
File "umqtt/simple.py", line 84, in connect
IndexError: bytes index out of range

Implement functools.partial function.

Is there any C/asm optimization possible or shall we stick to pure Python version for next few months?

I'd like to give micropython-lib a try, but I don't know how to build it and set it up.

Can you document the required steps ?

os.urandom uses "open" to open /dev/urandom, but that refers to the re-defined open in the os module. It should use the builtin open.

I have the following test code:

import select
epoll = select.epoll()
epoll.register(0, select.EPOLLIN)
epoll.poll()

It'll wait until there's data on stdin to read. If you press ctrl-C then the poll returns with EINTR and is supposed to finish early. On 64-bit x86-64 it works as expected (KeyboardInterrupt is raised) but on 32-bit x86 it leads to seg fault.

Any ideas?

core.py uses time.time() in its run_forever() loop. This produces unexpected results on the Pyboard because time.time() returns an integer. This causes the following code to behave incorrectly but understandably for a while.

There is a second problem which occurs after a period, in which the '1.3' coroutine gets repeatedly scheduled very rapidly, before it again settles down. The code (with trivial name changes) works as expected on cPython.

from uasyncio.core import get_event_loop, Sleep
loop = get_event_loop()
def cb(t, s):
    print(s)
    loop.call_later(t, cb, t, s)
loop.call_soon(cb, 1, 'one second')
loop.call_soon(cb, 1.3, '1.3')
loop.run_forever()

Output:

one second
1.3
one second
1.3
one second
1.3
one second
1.3
................. OK, I get this. repeats for a while. But then:
1.3
1.3
1.3
1.3
1.3
1.3
1.3
1.3
1.3
1.3
................. Numerous repetitions omitted. Then:
one second
1.3
one second
1.3
one second

The reason appears to be that the GNU and BSD versions of 'find' require different options:

bryans-computer:micropython-lib bryan$ make install
find: illegal option -- m
find: illegal option -- a
find: illegal option -- e
find: illegal option -- p
find: illegal option -- t
find: illegal option -- h
find: 1: No such file or directory

I haven't yet figured out how to modify the command to work on OSX, will continue to look into it.
Reference for BSD (OSX) find (the man page on OSX is basically the same):
https://www.freebsd.org/cgi/man.cgi?query=find(1)&sektion=
Reference for GNU find:
http://linux.die.net/man/1/find

Subscribing is working but when I just do:

c.publish(b"topic",b"This is a test") I get :
TypeError: function takes 2 positional arguments but 3 were given

How can that be?

See http://bugs.python.org/issue24449

@dpgeorge : FYI

We'd like to be able to install packages directly on a system running using MicroPython itself, including small self-hosted system like MicroPython bare-metal ports (pyboard, etc.)

So, a self-hosted package manager (codenamed "upip") should be written (initial prototyping is actually in progress for some time). Complete self-hosting has many obstacles, for example, currently PyPI usages mandates SSL access, so until SSL network connections are supported, it's not possible (See micropython/micropython#952).

So, to not block on individual issues (like SSL issue above), the approach should be incremental - first write proof of concept minimal installer running in MicroPython unix port with the help of external utilities (wget, etc.), then gradually get rid of dependencies to support full self-hostedness.

Is it desired to have a tiny distutils stub in micropython-lib that implements some library functions of it in a tiny way?
One example would be here:
http://bues.ch/gitweb?p=awlsim.git;a=tree;f=libs/tiny_distutils/distutils;hb=HEAD
This implements distutils.spawn.find_executable() in a very simple way, so my application can use it.
Would that be a candidate for merging into micropython-lib?

is this missing or deprecated, whats best way to append an iterable?

If I've understood the way ussl works correctly, the default behaviour is to skip certificate verification, as used to be the case with CPython prior to PEP 476.

This is a problem not only for ussl, but also for upip specifically, as this means any use of the "download" option remains subject to man-in-the-middle attacks, even when using HTTPS rather than HTTP.

Rather than silently creating a security failure, it would be desirable to fail to download over HTTPS if the ability to verify certificates hasn't been configured correctly (if the protocol is HTTP, it's at least obvious the connection isn't secure - the "https" in the URL means an unvalidated HTTPS connection claims to be secure, but in fact is not).

If I connect up an MQTT client and publish (sucessfully), then stop the broker, subsequent publishes fail without error. I'm not sure if this is the desired functionality or not. In fact, its a bit opaque to me how robust achieves its reconnection functionality.

connected with mihome, channel 6
dhcp client start...
ip:192.168.10.185,mask:255.255.255.0,gw:192.168.10.1

from simple import MQTTClient
c = MQTTClient("adcdd", "192.168.10.195") # Broker not running
c.connect()
0
c.publish("foo", "Hello MQTT")
..# Stop broker
c.publish("foo", "Hello MQTT")

No exception....

Hello guys,

I'm making basic notifier

I want login fb/twitter/gmail etc. with nodemcu module( with uPython) but i can't imported mechanize or bs4.
how to login fb/twitter/gmail etc. ? thanks

umqtt.robust works when WiFi fails completely. In testing there are circumstances when WiFi performance can be degraded to a point where blocking sockets can hang indefinitely without throwing an OSError. The behaviour is hard to replicate consistently. Moving slowly to the limit of WiFi range can (rarely) provoke it. It occurs here every few hours even when well within range, perhaps owing to RF interference. I have seen the following in many hours of testing:

1. Socket read and write methods hanging for long periods.
2. Publish with qos==1 failing to receive a PUBACK.

These will cause an application to hang. Possible fixes:

1. Use a socket timeout rather than a blocking socket.
2. Implement a timeout in the qos==1 loop which waits for a PUBACK. This begs the question as to what to do if a PUBACK is not received when the timeout expires. Arguably the publication should be repeated with dup==1 on the grounds that the broker may not have received the original message.

Hey ,I want to ask whcih file in esp8266's floder describe the xively ,Thank you for your answer!

Hi,

I may be rude now, so sorry in advance.

Could we rethink mqtt api as it is now? Unfortunately as many other libraries I see that mqtt.simple tries to make it's own 'socket'. But, does it really makes sense? It's network communication protocol, making the connection and getting a way to write/receive data should not be it's business. It's business is to make data with some sense from incoming bytes and send sensible bytes.

Problems:

• you will start to bloat protocol api with socket/ssl options
  • ssl header "ssl.wrap_socket(sock, keyfile=None, certfile=None, server_side=False, cert_reqs=CERT_NONE, ca_certs=None)" got pretty range of args to be fully supported :)
• you practically blocked people from using something else then socket, while not many people could think about it, but with protocol you could still send it's data over bluetooth/i2c/spi/uart.

Solution:

• mqtt.simple should be able to work in two modes:
  • either make simple socket
    • connect() is relevant
  • or get 'socket' api friendly object
    • connect() is not relevant, we want only initialize the mqtt protocol connection

This way I can prepare connection in any way I want on anything I want, and just pass proper object to mqtt "hi, here you got com object, use as you want".

So, what you think?

P.S: I am in state that I need to pass cert arguments to ssl, so I should be able to come with patch for this eventually

https://github.com/micropython/micropython-lib/blob/master/uasyncio/uasyncio/__init__.py

starts

import errno
import uselect as select

uselect no longer exists in micropython-lib, but is in select. I fixed this in my copy and then ran up against ffilib and ffi issues, ie:
ffi no longer exists,
on line 11 of select.py libc.func has no attribute NoneType, etc, etc.

Basically, I'm trying to get the asyncio http server running with a view to being able to write
asyncio network code. Is there a better branch to be working with?

Hi,

the umqtt.robust module is pretty reliable but I found a case when it stops sending:

As soon as I restart the MQTT broker, I have to connect via WebREPL and press CTRL+C.

I use the following minimal test code:
https://github.com/phieber/uPython-ESP8266-01-umqtt

Can you reproduce this issue when restarting the broker?

br
Patrick

I noticed there's no license information in the repository -- is it safe to assume the code is under the same MIT license as Micro Python?

Cheers,
Roger

If I try to do a publish on a yet to be connected client the following happens.

state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 2
cnt

connected with mihome, channel 6
dhcp client start...
ip:192.168.10.185,mask:255.255.255.0,gw:192.168.10.1

from simple import MQTTClient
c = MQTTClient("adcdd", "192.168.10.195") # Broker not running
c.publish("foo", "Hello MQTT")
Traceback (most recent call last):
File "", line 1, in
File "simple.py", line 110, in publish
AttributeError: 'NoneType' object has no attribute 'write'

It would be nice if the exception was more explicit about the cause.

Example script:

import time
from umqtt.simple import MQTTClient

def sub_cb(topic, msg):
    print((topic, msg))

def main(server="localhost"):
    c = MQTTClient("umqtt_client", server)
    c.set_callback(sub_cb)
    c.connect()
    c.subscribe(b"foo_topic")
    #time.sleep_ms(100)
    #c.check_msg()
    c.subscribe(b"bar_topic")
    while True:
        c.wait_msg()

    c.disconnect()

Running with mosquitto MQTT broker seems to work fine with:

$ mosquitto_pub -t "foo_topic" -m "1"
$ mosquitto_pub -t "bar_topic" -m "1"
$ mosquitto_pub -t "bar_topic" -m "3"

However, having a retained message,

$ mosquitto_pub -t "foo_topic" -m "1" -r

the start of the script fails with:

Traceback (most recent call last):
  File "example_sub_multi.py", line 21, in <module>
  File "example_sub_multi.py", line 14, in main
  File "/home/frederik/Downloads/micropython-lib/umqtt.simple/umqtt/simple.py", line 104, in subscribe
AssertionError:

Having a look in the code shows that subscribe() is expecting the
answer of the second subscribe (0x90), but since the message of the
first subscribe arrived, the AssertionError is thrown. So,
uncommenting sleep_ms() and check_msg() results in a working
script.

@pfalcon: This is a bug or by intention?

It's the first line of _libc/init.py

It looks like there's supposed to be a module called 'ffi', however it's not present in _libc, nor does it seem to be a builtin on the pyboard.

Is this supposed to be part of the _libc module, or some component I need to get from elsewhere?

Request for inspect module to be ported