Adding musl c library to micro python - musl branch about micropython HOT 26 CLOSED

Thanks Hagen!

It would be great if we can merge this (or parts of it) soon into the master branch.

Does the musl regex implement the same syntax and semantics as Python needs for its regex module?

from micropython.

Also, how do we keep in sync with the musl development? It would be nice if we didn't modify musl (as much as that's possible) but instead just started using the parts of their library as necessary. That way we can pull in any new changes they make.

from micropython.

That's a good question about regex, I'll have to look into that to know the answer. Seems to me if it doesn't implement the same then it's probably not a good idea to bring in this feature.

I'll try to code in small milestones that would be easy to merge back into the master.

I've struggled with the idea of keeping in sync with musl development. My only experience in this, is with svn, which is not pretty. The model we use is to keep a separate 'Vendor' tree and then hand merge local changes into a 'Local Vendor' tree anytime one of the 'Vendor' libraries has a significant update.

I'm not a git expert, so what I did was just import the repo along with history. I guess we could consider this a branch of some sorts. The best idea I have now is to hand merge any significant changes to musl master into the micro python 'branch'. Unless there is a better 'git' way of doing this. I'm open to any suggestions.

The last bit of work I did with a C library was to enhance the Bionic C library on the Android phones to add wide character support and other items that were missing, sort of the reverse of what I'm trying to do here. From my experience doing that, the C library tends to be like a sweater and when you pull on a thread a lot more comes along with it. So I believe we'll have to do some local changes so the whole sweater doesn't come along with the threads we are pulling.

from micropython.

Also, how do we keep in sync with the musl development?

Indeed, why import big 3rd-party libs into MicroPython? Why exactly musl? There are/can be lot of C libs around - it's nice to test them for uPy compilation, but why force specific one?

What would be useful is to limit core MicroPython usage to specific subset of C library, like for example no stdio (POSIX file i/o should be enough). But that won't help with 3rd-party extension modules anyway.

from micropython.

Generally the C library in the open source world comes from two historical sources - GNU and BSD - the difference being the license model. Since micro python is MIT licensed, the most compatible license to use is the c library that originates from the BSD source.

The musl c library is a carefully crafted version of the c library that carries the MIT license. By bringing in a snapshot of this library (all of the code) into micro python a paper trail has been created of where this source originally came from. Now we can pick and choose what portions of the C library we can repurpose for micro python rather than write code from scratch.

The only other library I can think of that would fit the license would be the Bionic C library which caries the Apache License. However this library has already been stripped down to not include UTF8 support, so there would be quite a bit of work getting the proper code to bring that support back in.

There is a lot of value of re-using this code, even something as simple as a printf function has a lot of nuances that have already been figured out by teams of programmers before us.

from micropython.

But what's the use case of crafting C library "just for MicroPython"? Currently, MicroPython can be built with any C lib, and 2 actual targets have good libs already - for Cortex-M bare metal that's apparently newlib, for POSIX system (codenamed "unix") - that's host libc, which for most people would be glibc. So, current usecases are covered, what are yours?

from micropython.

newlib and glibc are GPL license. Since the micro python code would be statically linking in these libraries, the license of micro python would change from it's existing MIT license which has an overall attractiveness that out weighs just statically linking in a GPL license library.

from micropython.

Okay, well, it seems we need to have a think about the issue of a C library.

1. We need some C library for the STM port. If newlib is GPL, then we need musl (or something else) that is MIT.
2. Would be good to keep the unix version C lib agnostic, so that it can compile on any posix system.
3. We don't need much in the way of C lib functionality. I've implemented myself what is needed for the STM port (strxxx, memxxx, printf is basically it).
4. I think it's nice to include a C lib in the Micro Python repo so the STM port is completely self contained.
5. Since it's just the C lib, it should be easy to replace anything we choose with something else, since a proper C lib is conforming, and we'll write conforming code.

The main thing we need is UTF-8 support. What is the standard C lib way of doing UTF-8?

For regex, it would be nice to take it from somewhere, but it needs to be exactly Python's regex. We might be able to use the Python code, if it has the correct license.

from micropython.

regex lib comparisons with licenses:

• http://en.wikipedia.org/wiki/Comparison_of_regular_expression_engines

The pypi module re is a replacement (but not exact match) for inbuilt and is licensed under the PSFL. Is it compatible with the MIT ?

• https://pypi.python.org/pypi/regex
• it does do unicode

from micropython.

I fear I've scared people by talking of tailoring a c library. It's really not that scary, but it is a bit of work. There are lots of things to consider when trying to answer a question like adding UTF-8 support and such. The first that comes to mind is simple things like a c function isalpha()

In the ASCII world this is a-f,A-F but start dipping into the UTF-8 world and now is isalpha() true for é or ö. This is the 'thread' that starts to unravel the 'sweater'. Now you need to bring in wide character support, locale, etc.

It's really not that hard to bring in a C library, but you need to know how much of the sweater you want to bring in.

Hope this makes sense...

from micropython.

Hello,

On Sun, 29 Dec 2013 16:24:30 -0800
hagenkaye [email protected] wrote:

newlib and glibc are GPL license. Since the micro python code would
be statically linking in these libraries, the license of micro python
would change from it's existing MIT license which has an overall
attractiveness that out weighs just statically linking in a GPL
license library.

Newlib is a hack of BSD libc, with most of the hacks themselves
released as BSD: https://sourceware.org/newlib/COPYING.NEWLIB

Glibc is _L_GPL. Yes, it does NOT have static linking exception, like
some libs do. But It makes no sense to static-link Glibc anyway,
because it's bloat. Glibc users use it dynamically and don't have
problems with it.

So, there's no real need to fight windmills. There're fun reasons to
use other libc's and I understand that kind of fun, and will be doing it
to MicroPython like I did to some other projects - unless you beat me on
it ;-). But for that kind of fun, there's no need to do large random
code drops into MicroPython repository. Use the alternative libc's as
they are. If there're good reasons to not use them as they are, fork
them into your own repos, and use that. When you have some results,
create a wiki page titled "How to link a uPy with arbitrary alternative
libc". Well, that's how I'd do it, YMMV.

Best regards,
Paul mailto:[email protected]

from micropython.

Btw, it would be nice to open separate ticket for regex library - it does not have to come from libc.

from micropython.

It's really not that scary

Custom-tailored C libs is not scary, it's lotsa fun. What's scary is random code drops into main repo ;-). Let's try to keep the project clean and lean. Custom-tailored lib can start as a separate repo. It can also start from STM code as pointed by @dpgeorge . Or it can start from another lib, but not with approach "drop lot of iunk, then try to clean it up", but with "take one useful piece, then pull useful dependencies and prune useless". In that regard, the best thing about musl is the full match of license name. Because otherwise it "tries to be correct and modern", which means threads supports, and thread-safety is a gore (when you don't need it). So, there're so many nice choices and tries to do before landing code in the main repo ;-).

from micropython.

I'll respectively disagree with the pull in everything and prune approach. As this method definitively shows where source code came from (from that hash of the entire source tree as the first commit).

Anyways, I guess I could put this in a different repo, but I prefer just to create a branch as this allows everyone to see the code in one place easily. I'll let Damien decide if anything I do for fun is worthy of merging back to master.

from micropython.

newlib and glibc are GPL license.

When newlib originated at RedHat its main license was based on BSD. As
code was added from other sources, their respective licenses came into
effect. But I understand only builds of newlib for Linux targets are
covered by GPL (possibly due to use of code from glibc). Builds for
standalone (no OS) ARM targets should not be covered by GPL. This seems
consistent with the newlib license document:

ftp://sources.redhat.com/pub/newlib/COPYING.NEWLIB
...
(21) Free Software Foundation LGPL License (-linux targets only)
...
(22) Xavier Leroy LGPL License (i[3456]86--linux targets only)
...

musl libc may not support ARM Cortex-M targets. According to this table
by Rich Felker (main musl developer), microcontrollers are not one of
its core targets.

http://www.etalabs.net/compare_libcs.html

                                                  musl  uClibc 

dietlibc eglibc
...
MMU-less microcontrollers no yes no no

newlib is larger than other libc implementations for microcontrollers,
but would be a reasonable default for Micro Python builds for standalone
ARM targets. Some commercial GCC distributions and ARM's own compiler
provide C libraries with better ARM optimizations. Those could be
supported later on for projects that require smaller binaries.

http://support.code-red-tech.com/CodeRedWiki/RedlibAndNewlib
Redlib

http://www.rowleydownload.co.uk/arm/documentation/home.htm
http://www.rowleydownload.co.uk/arm/documentation/index.htm?http://www.rowleydownload.co.uk/arm/documentation/home.htm
CrossWorks C library

http://www.keil.com/arm/microlib.asp
MicroLib

from micropython.

Another note for licensing purists: currently "unix"-targetted "py" interpreter is linked with GNU readline, and that's what causes that binary to fall under GPL.

from micropython.

The "unix" and "stm" components should be considered separately with regards to libc and license choice. Both of these components rely on the "py" component, and the "py" component should have absolute minimal dependencies (eg no file access, no printf).

C lib functionality is needed mainly at the "unix" or "stm" or other port level. For the "stm" component, it would be nice to include a minimal libc, and musl looks good for this. We don't need any of its threading support, just math and a few other things.

@pfalcon: it seems you are most interested in the core "py" and "unix" components, and I agree that we should keep these (at least "py") as minimal as possible. I definitely don't want to have "py" depending on musl, or anything at all really.

@hagenkaye: what was your original reason for wanting to pull in a C library? I agree we want to provide functionality at the "stm" component level.

from micropython.

Hi Damien,

I pulled in the entire musl library to preserve the integrity of where the original source originated from. I'll leave the micro python/musl directory intact and make no modifications. In fact this musl directory doesn't even have to merged to master, we can keep the musl branch intact with the entire musl c library source code.

I'm now putting together a micropython/micromusl directory. As I grab code from the musl directory into micromusl the paper trail remains intact. At any time someone can git log --follow a file in the micromusl directory and get a complete historical account of where this source came from. Any local modifications I make will be in the micromusl directory. When you merge the code back into master, you'll just grab the commits from the micromusl directory.

The end result is a nice clean source tree in master branch, with a good historical account of the source in the musl branch.

Right now, the milestone I'm working on is to replace micro python/stm/string0.c with source from the musl c library. All the to-do's in string0.c (align 32 bit copies, copy backwards, etc.) will be completed by replacing this file with the code in musl.

The goal is to provide a really stripped down libc.a that uses only MIT licensed code. Since I'm using standard headers etc. the stm code can be linked with this micromusl library or linked with an alternative library in the future.

Next steps after replacing string0.c will be to bring in the code for printf and then the math library.

Hope this makes sense.

from micropython.

@pfalcon: it seems you are most interested in the core "py" and "unix" components, and I agree that we should keep these (at least "py") as minimal as possible.

Yep, I'd say I'm interested in a general-purpose implementation which scales down to MCUs (nowadays people mostly think about "scale up", which is sigh). Good plan on having separation between "core" and ports/extension. That's exactly my concern - it's already clear (well, to me) that MicroPython is such small because lot of "standard" CPython stuff is not yet implemented (say, string methods). If we just keep adding stuff, what will ensue is maybe a bit different, but big and bloaty CPython. So, instead of adding anything and everything, it's better to work on making it possible to integrate anything and everything, while still keeping core nicely separate.

from micropython.

I also opened #13 to discuss regex libs suitable for uPy, please add stuff there.

from micropython.

@hagenkaye yes, what you are doing sounds good to me. To fix up stm/string0.c is a very good start. Note that printf will need some work to port from musl: I'm using an "environment" (function and data) passed along to all parts of printf so that the output can go through a generic "print_strn" function. For the math stuff, that needs some thought because STM is natively 32bit float, not 64bit double, and I want to keep the Micro Python floating-point type as 32bit float.

from micropython.

@pfalcon yes, if we keep adding everything to Micro Python, it's likely to grow quite big. Once I've finished the core stuff, that should remain relatively stable and of fixed size. Then it's a matter of implementing the Python standard library and allowing that to be optionally compiled in, so you can use it if you have the space.

In the end, when/if Micro Python supports all CPython functionality (a hard task!), the main differences will be: a more compact compiler, ability to compile to machine code, an inline assembler, small-ints stuffed inside the object pointer, no reference counting, careful use of the heap to minimise its use, smaller object representation, and probably lots of other minor things. These are the features that make it suitable for a microcontroller.

from micropython.

I've been having some second thoughts on how to do this. Rather than making this a branch and creating this feature in the branch, I'm thinking that it probably would be best to create a repo specifically to take the musl library and strip it down to create a micromusl library - a small c library that runs on micro controllers. Doing it this way solves this issues that have come to mind

• running this feature as a branch creates work to keep the branch up to date with respect to the master
• the idea of cherry picking commits from the branch back to the master just creates extra work for Damien

As well, as I look at the musl code, I see there are possibilities of where some modifications to the musl library could benefit the musl library and I'd like to submit these back to the original musl library. And of course I'd like to keep the micromusl library in sync with the musl library so any changes made there can be brought into micromusl easily.

The micromusl library repo would just focus on making a libc library with a MIT license and wouldn't have any micro python code in it. Just the library with a separate test app to run tests to ensure the library is bug free before it gets pulled into micro python.

from micropython.

a small c library that runs on micro controllers

@hagenkaye, +1. If you'd need another test case for your lib, feel free to use https://github.com/pfalcon/squirrel-lang ;-)

from micropython.

okay here is my new grand experiment on bringing in changes from musl, creating a small c library, that can easily be brought back into micro python.

I created a repo for micromusl at

https://github.com/hagenkaye/micromusl.git

In this repo I added a submodule to bring in the musl repo

git submodule add -- git://git.musl-libc.org/musl musl

Since this is a submodule, at any time I can pull in updates from musl into micromusl

Now I think the easy way to get any changes from micromusl to micro python is to create a submodule in micro python to brings in the micromusl repo. From the micropython repo:

git submodule add -- https://github.com/hagenkaye/micromusl.git micromusl

I believe pull requests can be made, or @dpgeorge can just do:

cd micromusl
git pull

to bring in the latest.

This should be a better way to develop this feature and easily bring in changes. I'm not a git expert, so any comments and questions are welcome.

from micropython.

Closed. The musl branch has been deleted. Development will continue as above, in a separate repo.

from micropython.