Giter VIP home page Giter VIP logo

koreader-base's Introduction

koreader-base CircleCI

This is the base framework for creating document readers like KOReader, an e-ink device oriented reader application for various document formats.

It uses the MuPDF library (see http://mupdf.com/), djvulibre library, CREngine library, libk2pdfopt library and it is scripted using Lua (see http://www.lua.org/). To gain good speed at that, it uses the LuaJIT compiler.

It all started as the KindlePDFviewer application, which has since been greatly enhanced and now reflects this in having a new name, KOReader.

The code is distributed under the GNU AGPL v3 license (read the COPYING file).

Building

Follow these steps:

  • automatically fetch thirdparty sources with Makefile:

    • make sure you have patch, wget, unzip, git and svn installed
    • run make fetchthirdparty.
  • run make TARGET=kindlepw2 For Kindle models >= Paperwhite 2.

  • run make TARGET=kindle For Kindle models >= Kindle 4 < Paperwhite 2.

  • run make TARGET=kindle-legacy for Kindle DXG/2/3 devices.

  • or run make TARGET=kobo for Kobo devices.

  • or run make TARGET=pocketbook for PocketBook devices.

  • or run make TARGET=cervantes for BQ Cervantes devices.

  • or run make TARGET=android for Android devices.

  • or run make TARGET=win32 for Windows.

  • or run make TARGET=generic-arm for generic ARM devices.

  • or run make TARGET=ubuntu-touch for Ubuntu Touch.

  • or run make TARGET=appimage for the desktop AppImage.

  • or run make TARGET=debian, debian-armel or debian-armhf for a Debian package.

  • or run make KODEBUG=1 to build with debugging symbols.

  • or run make for the emulator on Linux.

Use ccache

Ccache can speed up recompilation by caching previous compilations and detecting when the same compilation is being repeated. In other words, it will decrease build time when the sources have already been built. Ccache support has been added to KOReader's build system. Before using it, you will need to install a ccache in your system.

  • in Ubuntu use:sudo apt-get install ccache
  • in Fedora use:sudo yum install ccache
  • install from source:
  • to disable ccache, use export USE_NO_CCACHE=1 before make.
  • for more detail about ccache. visit:

http://ccache.samba.org

Device emulation

The code also features a device emulation. You need SDL headers and library for this. It allows to develop on a standard PC and saves precious development time. It might also create a suboptimal desktop PDF viewer, depending on your view.

If you are using Fedora Core Linux, run yum install SDL2 SDL2-devel. If you are using Debian or Ubuntu, install the libsdl2-dev package.

By default, the build system builds in "emulation mode", so following is all you need to build the emulator:

make

The emulator uses 800x600 as the default resolution. It can be changed at runtime by changing the following environment variables:

EMULATE_READER_W=746 EMULATE_READER_H=1024

The emulator uses a default DPI of 160. This can also be changed with an environment variable:

EMULATE_READER_DPI=300

KOReader supports "viewports", i.e. displaying only in a rectangular excerpt of the screen. This is useful on devices where the framebuffer is larger than the area that is actually visible. In order to simulate such a viewport using the emulator, specify a specially crafted environment variable:

EMULATE_READER_VIEWPORT="{x=50,w=600,y=10,h=680}"

You can also simulate e-ink refresh with the emulator. When active, only refreshed areas of the screen are actually updated and you also get a visual inverse flash feedback. In order to activate that mode, set an environment variable to the number of milliseconds you want the flash to endure:

EMULATE_READER_FLASH=100

koreader-base's People

Contributors

benoit-pierre avatar chrox avatar clenton avatar cramoisi avatar cyphar avatar dmalinovsky avatar dpavlin avatar ezdiy avatar frankyifei avatar frenzie avatar galunid avatar germanc avatar houqp avatar hrdl-github avatar hugleo avatar hwhw avatar hzj-jie avatar mroethke avatar niluje avatar onde2rock avatar pazos avatar poire-z avatar rjd22 avatar tchaikov avatar thotypous avatar tigran123 avatar tns-hun avatar toromtomtom avatar yparitcher avatar zwim avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

koreader-base's Issues

Android aarch64 build.

It has a few fancy things (NEON and VFPv4)

  • SDK API needs to be bumped from API14 to API21 (that's easy and doesn't affect other builds: we can have both armeabi-v7a and x86 with a min api of 14 and build aarch64 with a min api of 21). The build system just need to have the target api installed (which is 28 right now)

  • NDK API can stay in r15c, AFAICT. That's cool because I'm unable to build luajit with something newer.

New pattern file is not in kobo package

The file Dutch.pattern isn't cloned into base/kpvcrlib/crengine/cr3gui/data/hyph. What is the reason for that? Is the koreader build still linked to an older base?

Remove extr.c?

extr.c is code written by @tigran123 to build a mupdf-based attachment extractor. That was used with kindlevncviewer, but with koreader, we currently do not support such attachments.

My proposal would be to remove the code since now it's a bit distracting.
On the other hand, there's no real need to do so other than keeping our code footprint small.

I would offer to introduce an "attachment API" based on the extr.c code into the mupdf interface, so in perspective, we could add extraction (and probably attachment listing) functionality to KoReader, too.

[Android framebuffer]: add support for e-ink updates.

Did a few test myself, using onyx sdk for rk3026 devices. Things work nice on "native" sdk applications but I can't get the same results on KOReader. The program don't crash and execute the commands on the java side accordly, but I can see any screen updates.

Digging a bit I found that, to be able to manipulate the current view from java land we might need to create a SurfaceView in MainActivity and get a reference to it using AnativeWindow_fromSurface. So we can write to the framebuffer but without holding the ownership, allowing java code to handle things on that particular surface.

Notes on updating MuPDF

The external fonts patch needs to be updated, but that looks like it should be simple.

Commit 4d45a50c authored a year ago by Tor Andersson's avatar Tor Andersson
Link required thirdparty libraries into one library: libmupdfthird.a

So that's set(LINK_OPTS "${LINK_OPTS} build/release/libmupdfthird.a") instead of build/release/libopenjpeg.a build/release/libmujs.a build/release/libjbig2dec.a separately. Also in Makefile.defs.

commit 15cab201d3c98dc6580c8cf592d94ab226f96db5
Author: Sebastian Rasmussen [email protected]
Date: Sat Aug 13 17:45:14 2016 +0800

Make fz_archive a generic archive type.

Previously it was inherently tied to zip archives and directories.
Now these are separated out into distinct subclasses. This prepares
for support for further archive formats.

That mainly means include/mupdf/fitz/unzip.h was changed to include/mupdf/fitz/unarchive.h.

But even for 1.9 wrap-mupdf.c will definitely need a look. I didn't want to investigate that now, but I'm leaving these notes so I or someone else can start right there rather than wasting 10 minutes on the basics.

PS Good to know: you can search for files in the git log. Well, I knew that, but I didn't know it also worked for deleted files. :-)

thirdparty/mupdf/build/git_checkout/mupdf$ git log -- include/mupdf/fitz/unzip.h

Remove PocketBook Harfbuzz workaround

https://github.com/harfbuzz/harfbuzz/releases/tag/2.5.0
Support for gcc 4.8 and earlier has been dropped.

In case you missed it, in https://github.com/harfbuzz/harfbuzz/releases for 2.5.2:

Fix build on gcc 4.8. That's supported again.

Some talks about dropping, less about resupporting it again in https://github.com/harfbuzz/harfbuzz/issues/1724 , relevant commit is https://github.com/harfbuzz/harfbuzz/commit/e4e518f33d933a02058bad86a6aae714e59814db

Originally posted by @poire-z in #917 (comment)

Clang build MD5 test broken

The good news. I can sometimes reproduce the Clang issue locally. The bad news, results aren't stable.

One test run:

[  FAILED  ] ./spec/base/unit/md5_spec.lua @ 10: MD5 module should calculate correct MD5 hashes
[  FAILED  ] ./spec/base/unit/md5_spec.lua @ 23: MD5 module should calculate MD5 sum of a file

Another test run:

[  FAILED  ] ./spec/base/unit/md5_spec.lua @ 16: MD5 module should calculate MD5 sum by updating
[  FAILED  ] ./spec/base/unit/md5_spec.lua @ 23: MD5 module should calculate MD5 sum of a file

Also if I run the commands myself in the LuaJIT console the output is correct.

ko@b4a54b3a1a82:~/koreader/base$ ./build/x86_64-pc-linux-gnu/luajit 
LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2017 Mike Pall. http://luajit.org/
JIT: ON SSE2 SSE3 SSE4.1 BMI2 fold cse dce fwd dse narrow loop abc sink fuse
> md5=require"ffi/MD5"
> print(md5.sum(""))
d41d8cd98f00b204e9800998ecf8427e
> md5.new():sum("\0")
> print(md5.new():sum("\0"))
93b885adfe0da089cdf634904fd59f71
> print( md5.new():sum("0123456789abcdefX"))
1b05aba914a8b12315c7ee52b42f3d35
> m = md5.new()
> m:update("0123456789")
> m:update("abcdefghij")
> print(m:sum())
644be06dfc54061fd1e67f5ebbabcd58
> print(md5.sum("0123456789abcdefghij"))
644be06dfc54061fd1e67f5ebbabcd58

Fix GloHD mxcfb handling

Note to self for when I have some free time:

  • The driver was slightly updated, there are header changes (in particular a 'new' wait_for_update_complete, w/ collision info, like on Kindles).
  • Potentially confirmation that _AUTO enables REAGLD automatically behind the scenes, yay?!

TODO:

  • Update strace patch
  • Update frankenstein mxfb header
  • Update wait_for_update_complete handling

CAVEATS:

Said new wait_for_update_complete may or may not be actually used on production devices (cf. MX50_IOCTL_IF). And I don't have a device to check, so, if anyone could strace nickel, that'd help :).

Untangle invert logic

Just FYI:
I have a bunch of patches in the pipeline that will un-entangle the situation with regard to the invertation of framebuffers and blitbuffers. For historic reasons, we jump through some hoops in that area:
The original eink framebuffer has 0=white, 15=black. Virtually everything else, including the framebuffers starting with Kindle Touch, have 0=black, 15=white (or the according RGB16 values, like Kobo's framebuffer has, or the according RGB32 values on everything else).
But since a lot of the code started back then, we did everything the old way. And thus we had to convert the data we got from 3rd party libs (like all rendering engines), and we had to convert it again for the modern framebuffers - arguably 99.999% of the devices we run on.

I've now rolled that situation up. Patches are in
https://github.com/hwhw/koreader-base/tree/master
and
https://github.com/hwhw/koreader/tree/noninvert

I'll wait with a PR until that library de-entangling I've started in base is cleaned up and merged into the main koreader repo, then I'll send PRs for this one.

Compile for arm-linux-gnueabihf?

Hi! since Kobo switched to armhf from armel in their latest firmware release, I'm trying to compile koreader for this architecture. I tried to simply modify Makefile.defs putting CHOST?=arm-linux-gnueabihf in it, but it looks like this setting is respected only partially since in the end the linker seems to complain that part of the binaries are for armhf and part of them are for armel (looks like thirdparty libraries are still built for armel). What else should I change?

Implement lua-http for HTTP(S) 1.0, 1.1 and 2.0

Alternative lazy option, patch luasocket and luasec, see lunarmodules/luasec#38

# expose parseRequest() from https://github.com/diegonehab/luasocket/pull/133 (without ftp.lua and smtp.lua)
set(PATCH_CMD "${PATCH_CMD} && patch -N -p1 < ${CMAKE_CURRENT_SOURCE_DIR}/parserequest.patch || true")

parserequest.patch for luasocket updated for current master

diff --git a/src/ftp.lua b/src/ftp.lua
index 6286f90..a894c1c 100644
--- a/src/ftp.lua
+++ b/src/ftp.lua
@@ -34,8 +34,13 @@ _M.PASSWORD = "[email protected]"
 -----------------------------------------------------------------------------
 local metat = { __index = {} }
 
-function _M.open(server, port, create)
-    local tp = socket.try(tp.connect(server, port or _M.PORT, _M.TIMEOUT, create))
+function _M.open(params)
+    local tp = socket.try(tp.connect(
+        params.server, 
+        params.port or _M.PORT, 
+        _M.TIMEOUT, 
+        function() return params:create() end  -- wrap create as a method call
+      ))
     local f = base.setmetatable({ tp = tp }, metat)
     -- make sure everything gets closed in an exception
     f.try = socket.newtry(function() f:close() end)
@@ -203,7 +208,7 @@ end
 local function tput(putt)
     putt = override(putt)
     socket.try(putt.host, "missing hostname")
-    local f = _M.open(putt.host, putt.port, putt.create)
+    local f = _M.open(putt)
     f:greet()
     f:login(putt.user, putt.password)
     if putt.type then f:type(putt.type) end
@@ -235,7 +240,7 @@ end
 local function tget(gett)
     gett = override(gett)
     socket.try(gett.host, "missing hostname")
-    local f = _M.open(gett.host, gett.port, gett.create)
+    local f = _M.open(gett)
     f:greet()
     f:login(gett.user, gett.password)
     if gett.type then f:type(gett.type) end
@@ -262,7 +267,8 @@ _M.command = socket.protect(function(cmdt)
     cmdt = override(cmdt)
     socket.try(cmdt.host, "missing hostname")
     socket.try(cmdt.command, "missing command")
-    local f = _M.open(cmdt.host, cmdt.port, cmdt.create)
+    cmdt.create = cmdt.create or socket.tcp
+    local f = _M.open(cmdt)
     f:greet()
     f:login(cmdt.user, cmdt.password)
     f.try(f.tp:command(cmdt.command, cmdt.argument))
@@ -277,6 +283,7 @@ _M.put = socket.protect(function(putt, body)
       tput(putt)
       return table.concat(putt.target)
     else 
+      putt.create = putt.create or socket.tcp
       return tput(putt) 
     end
 end)
@@ -287,6 +294,7 @@ _M.get = socket.protect(function(gett)
       tget(gett)
       return table.concat(gett.target)
     else 
+      gett.create = gett.create or socket.tcp
       return tget(gett) 
     end
 end)
diff --git a/src/http.lua b/src/http.lua
index 550634c..d926e72 100644
--- a/src/http.lua
+++ b/src/http.lua
@@ -106,15 +106,15 @@ end
 -----------------------------------------------------------------------------
 local metat = { __index = {} }
 
-function _M.open(host, port, create)
-    -- create socket with user connect function, or with default
-    local c = socket.try((create or socket.tcp)())
+function _M.open(reqt)
+    -- create socket with user connect function
+    local c = socket.try(reqt:create())   -- method call, passing reqt table as self!
     local h = base.setmetatable({ c = c }, metat)
     -- create finalized try
     h.try = socket.newtry(function() h:close() end)
     -- set timeout before connecting
     h.try(c:settimeout(_M.TIMEOUT))
-    h.try(c:connect(host, port or PORT))
+    h.try(c:connect(reqt.host, reqt.port or PORT))
     -- here everything worked
     return h
 end
@@ -294,7 +294,7 @@ end
     -- we loop until we get what we want, or
     -- until we are sure there is no way to get it
     local nreqt = adjustrequest(reqt)
-    local h = _M.open(nreqt.host, nreqt.port, nreqt.create)
+    local h = _M.open(nreqt)
     -- send request line and headers
     h:sendrequestline(nreqt.method, nreqt.uri)
     h:sendheaders(nreqt.headers)
@@ -352,9 +352,10 @@ end
 _M.request = socket.protect(function(reqt, body)
-    if base.type(reqt) == "string" then return srequest(reqt, body)
+    if  base.type(reqt) == "string" then 
+      local t, code, headers, status = reqt.target, socket.skip(1, _M.request(reqt))
       return table.concat(t), code, headers, status
-    else return trequest(reqt) end
+    else
+      reqt.create = reqt.create or socket.tcp
+       return trequest(reqt) 
+     end
 end)
diff --git a/src/smtp.lua b/src/smtp.lua
index b113d00..ea52873 100644
--- a/src/smtp.lua
+++ b/src/smtp.lua
@@ -113,9 +113,13 @@ function metat.__index:send(mailt)
     self:data(ltn12.source.chain(mailt.source, mime.stuff()), mailt.step)
 end
 
-function _M.open(server, port, create)
-    local tp = socket.try(tp.connect(server or _M.SERVER, port or _M.PORT,
-        _M.TIMEOUT, create))
+function _M.open(mailt)
+    local tp = socket.try(tp.connect(
+        mailt.server or _M.SERVER, 
+        mailt.port or _M.PORT,
+        _M.TIMEOUT, 
+        function() return mailt:create() end  -- wrap to do a method call
+      ))
     local s = base.setmetatable({tp = tp}, metat)
     -- make sure tp is closed if we get an exception
     s.try = socket.newtry(function()
@@ -245,7 +249,8 @@ end
 -- High level SMTP API
 -----------------------------------------------------------------------------
 _M.send = socket.protect(function(mailt)
-    local s = _M.open(mailt.server, mailt.port, mailt.create)
+    mailt.create = mailt.create or socket.tcp
+    local s = _M.open(mailt)
     local ext = s:greet(mailt.domain)
     s:auth(mailt.user, mailt.password, ext)
     s:send(mailt)

k2pdfopt compiled with mupdf support, but mupdf support actually broken

#286 led me to investigate why we have the

error: cannot open 
error: cannot load document ''

errors. It's from pdf_open_document_no_run().

It triggers when we call libk2pdfopt's k2pdfopt_reflow_bmp() function in the MuPDF FFI wrapper.

This is the full backtrace for the non-LuaJIT side of things:

(gdb) bt
#0  0x00007ffff17b0d60 in pdf_open_document_no_run () from libs/libmupdf.so
#1  0x00007ffff17a9259 in pdf_open_document () from libs/libmupdf.so
#2  0x00007ffff070ed86 in wtextchars_fill_from_page_ex () from libs/libk2pdfopt.so.2
#3  0x00007ffff06c1e68 in k2ocr_ocrwords_get_from_ocrlayer () from libs/libk2pdfopt.so.2
#4  0x00007ffff06c2a29 in k2ocr_ocrwords_fill_in_ex () from libs/libk2pdfopt.so.2
#5  0x00007ffff06bc2d7 in masterinfo_add_bitmap () from libs/libk2pdfopt.so.2
#6  0x00007ffff06cb940 in bmpregion_add () from libs/libk2pdfopt.so.2
#7  0x00007ffff06ca0d9 in bmpregion_analyze_justification_and_line_spacing () from libs/libk2pdfopt.so.
#8  0x00007ffff06cba88 in bmpregion_add () from libs/libk2pdfopt.so.2
#9  0x00007ffff06cc228 in bmpregion_add_textrows () from libs/libk2pdfopt.so.2
#10 0x00007ffff06ccd06 in bmpregion_source_box_process () from libs/libk2pdfopt.so.2
#11 0x00007ffff06cd73f in bmpregion_source_page_add () from libs/libk2pdfopt.so.2
#12 0x00007ffff06afc03 in k2pdfopt_reflow_bmp () from libs/libk2pdfopt.so.2
...

Looking more closely at it, this is the OCR support in k2pdfopt. We have support for PDF "pseudo" OCR with MuPDF in k2pdfopt. However, in order to use it, we would have to set up the "MASTERINFO" struct properly with source file name, and masterinfo_new_source_page_init() would have to be called with the right page number. All of this isn't ever done, since we pass bitmaps to k2pdfopt, not PDF files.

The relevant part is in https://github.com/koreader/libk2pdfopt/blob/30552b25b964aaeff31bb4d664a12d9dc99816c1/koptreflow.c#L52 and following lines.

What I'll try next is to build k2pdfopt without mupdf integration. It should work well anyway if I'm not mistaken?

Code reorganization: Split into modules/shared libraries

A thing I wanted to do long ago is to clean up the koreader-base building and the organization. While statically compiling everything into one large binary might have its nice aspects, its somewhat clumsy and we might get into problems in the long run.

So I want to split everything up into single Lua modules that can be "require"d. So there won't be a big executable anymore, but rather a standard LuaJIT interpreter/compiler that will dynamically load the modules, which themselves drag in the corresponding dynamic libraries for the functionality they provide.

In fact, I did this today. Have a look at https://github.com/hwhw/koreader-base - it now builds the bigger parts as shared libraries, then will build libraries using the C/Lua API for dynamically loading.

I'm not sending a Pull Request yet, since I want to do some more cleaning up. I'd like to have some proper output directories for emulator and native code, so from the perspective of the frontend code repositories, both can be switched on the fly. And I want to straighten the naming of the Makefile variables and - if possible - clean up the Makefile structure a bit.

Feel free to try out what I've done so far!

In the longer run, I want to replace some interfacing with LuaJIT's FFI. It's rapidly faster than the Lua/C API we're using now. But in order to get there, I must take step by step, so my first step was to split it into components.

Building djvulibre For Kobo

I'm trying to build koreader-base cc7f266 on 64-bit Debian Sid using the CodeSourcery toolchain from the KoboReader GitHub repo.

djvulibre fails to build with the following error when caling:

TARGET_DEVICE=KOBO make all customupdate

This is the error:

make[1]: Entering directory /home/rob/src/koreader/koreader-base/djvulibre/build/tools' arm-none-linux-gnueabi-g++ -DHAVE_CONFIG_H -I.. -I../.. -I../../libdjvu -I../../tools -DNDEBUG -Wall -O3 -Wno-non-virtual-dtor -pthread -DTHREADMODEL=POSIXTHREADS -ffast-math -pipe -fomit-frame-pointer -march=armv6j -mtune=arm1136jf-s -mfpu=vfp -mfloat-abi=softfp -marm -fno-stack-protector -U_FORTIFY_SOURCE -D_GNU_SOURCE -fno-finite-math-only -fno-use-cxa-atexit -c ../../tools/bzz.cpp /bin/sh ../libtool --mode=link \ arm-none-linux-gnueabi-g++ -o bzz -DHAVE_CONFIG_H -I.. -I../.. -I../../libdjvu -I../../tools -DNDEBUG -Wall -O3 -Wno-non-virtual-dtor -pthread -DTHREADMODEL=POSIXTHREADS -ffast-math -pipe -fomit-frame-pointer -march=armv6j -mtune=arm1136jf-s -mfpu=vfp -mfloat-abi=softfp -marm -fno-stack-protector -U_FORTIFY_SOURCE -D_GNU_SOURCE -fno-finite-math-only -fno-use-cxa-atexit bzz.o ../libdjvu/libdjvulibre.la -Wl,-O1 -Wl,--as-needed -lm libtool: link: arm-none-linux-gnueabi-g++ -o .libs/bzz -DHAVE_CONFIG_H -I.. -I../.. -I../../libdjvu -I../../tools -DNDEBUG -Wall -O3 -Wno-non-virtual-dtor -pthread -DTHREADMODEL=POSIXTHREADS -ffast-math -pipe -fomit-frame-pointer -march=armv6j -mtune=arm1136jf-s -mfpu=vfp -mfloat-abi=softfp -marm -fno-stack-protector -U_FORTIFY_SOURCE -D_GNU_SOURCE -fno-finite-math-only -fno-use-cxa-atexit bzz.o -Wl,-O1 -Wl,--as-needed ../libdjvu/.libs/libdjvulibre.so -lm -pthread /home/rob/CodeSourcery/Sourcery_G++_Lite/bin/../lib/gcc/arm-none-linux-gnueabi/4.4.1/../../../../arm-none-linux-gnueabi/bin/ld: .libs/bzz: hidden symbol__sync_lock_test_and_set_4' in /home/rob/CodeSourcery/Sourcery_G++_Lite/bin/../lib/gcc/arm-none-linux-gnueabi/4.4.1/libgcc.a(linux-atomic.o) is referenced by DSO
/home/rob/CodeSourcery/Sourcery_G++_Lite/bin/../lib/gcc/arm-none-linux-gnueabi/4.4.1/../../../../arm-none-linux-gnueabi/bin/ld: final link failed: Nonrepresentable section on output
collect2: ld returned 1 exit status
make[1]: *** [bzz] Error 1
make[1]: Leaving directory `/home/rob/src/koreader/koreader-base/djvulibre/build/tools'

I am unable to fix this by adding -lgcc_s -lgcc to the flags as suggested by a web search for similar problems, but I can get it to build by hacking config.h to not set HAVE_INTEL_ATOMIC_BUILTINS .

gcc-lua-cdecl usage

We could use gcc-lua-cdecl to avoid incorrect FFI declarations (see #46 and #50). I could write a Wiki page with instructions or could integrate it in our build system. Integrating it in the build system has the advantage of increasing portability: the headers will be processed by the same toolchain as used to compile koreader-base, so there is no risk of different struct declarations because of platform variation etc. The drawbacks are that we would need to import two new git repositories as submodules (gcc-lua and gcc-lua-cdecl), and add more building dependencies (gcc-4.x-plugin-dev in Debian/Ubuntu, and Lua itself).

My idea is that we would have for instance a ffi/utildecl.c file containing:

#define _BSD_SOURCE
#include <unistd.h>
#include <sys/statvfs.h>
#include <sys/time.h>
#include "cdecl.h"

cdecl_struct(timeval)
cdecl_func(gettimeofday)
cdecl_func(sleep)
cdecl_func(usleep)
cdecl_struct(statvfs)
cdecl_func(statvfs)

And the building system would autogenerate a ffi/utildecl.lua file containing:

local ffi = require("ffi")
ffi.cdef[[
struct timeval {
  long int tv_sec;
  long int tv_usec;
};
int gettimeofday(struct timeval *restrict, struct timezone *restrict) __attribute__((__nothrow__, __leaf__));
unsigned int sleep(unsigned int);
int usleep(unsigned int);
struct statvfs {
  long unsigned int f_bsize;
  long unsigned int f_frsize;
  long unsigned int f_blocks;
  long unsigned int f_bfree;
  long unsigned int f_bavail;
  long unsigned int f_files;
  long unsigned int f_ffree;
  long unsigned int f_favail;
  long unsigned int f_fsid;
  int __f_unused;
  long unsigned int f_flag;
  long unsigned int f_namemax;
  int __f_spare[6];
};
int statvfs(const char *restrict, struct statvfs *restrict) __attribute__((__nothrow__, __leaf__));
]]

What do you think? Any suggestions?

porting to a new device and UBUNTU_COMPAT_FLAGS

Hi, I'm trying to port KOReader to a bq cervantes 4. The first build went well but can't run because luajit was complaining about glibc. So I lurked into the code and set UBUNTU_COMPAT_FLAGS for this platform.

Now luajit works but KOReader fails with the crash.log:

---------------------------------------------
                launching...
  _  _____  ____                _
 | |/ / _ \|  _ \ ___  __ _  __| | ___ _ __
 | ' / | | | |_) / _ \/ _` |/ _` |/ _ \ '__|
 | . \ |_| |  _ <  __/ (_| | (_| |  __/ |
 |_|\_\___/|_| \_\___|\__,_|\__,_|\___|_|

 It's a scroll... It's a codex... It's KOReader!

 [*] Current time: 10/12/18-21:46:27
 [*] Version: v2015.11-1878-g9b45a930_2018-10-12

ffi.load: blitbuffer
ffi.load (warning): libblitbuffer.so: cannot open shared object file: No such file or directory
ffi.load: SDL2
ffi.load (warning): libSDL2.so: cannot open shared object file: No such file or directory
ffi.load: SDL
ffi.load (warning): libSDL.so: cannot open shared object file: No such file or directory
10/12/18-21:46:27 INFO  SoftwareVersion:  openFW
10/12/18-21:46:27 INFO  initializing for device Cervantes
10/12/18-21:46:27 INFO  framebuffer resolution: {
    ["h"] = 1448,
    ["w"] = 1072
}
ffi.load: libs/libfreetype.so.6
ffi.load: libs/libmupdf.so
ffi.load (warning): /lib/arm-linux-gnueabi/libm.so.6: version `GLIBC_2.27' not found (required by libs/libmupdf.so)
./luajit: ./setupkoenv.lua:30: Not able to load dynamic library: libs/libmupdf.so
stack traceback:
	[C]: in function 'error'
	./setupkoenv.lua:30: in function 'load'
	./ffi/mupdf.lua:25: in main chunk
	[C]: in function 'require'
	frontend/ui/widget/htmlboxwidget.lua:10: in main chunk
	[C]: in function 'require'
	frontend/ui/widget/scrollhtmlwidget.lua:6: in main chunk
	[C]: in function 'require'
	frontend/ui/widget/dictquicklookup.lua:18: in main chunk
	[C]: in function 'require'
	frontend/apps/reader/modules/readerdictionary.lua:4: in main chunk
	[C]: in function 'require'
	frontend/apps/reader/readerui.lua:26: in main chunk
	[C]: in function 'require'
	./reader.lua:212: in main chunk
	[C]: at 0x00013a05

Any hint on that?, thanks

GTK3 filechooser broken in newer GTK3

You get this crash:

12/18/18-13:12:37 DEBUG Opening FileChooser: gtk-3
./luajit: ./ffi/filechooser_gtk-3.lua:32: bad argument #1 to 'gtk_dialog_run' (cannot convert 'struct _GtkWidget *' to 'struct _GtkDialog *')
stack traceback:
	[C]: in function 'gtk_dialog_run'
	./ffi/filechooser_gtk-3.lua:32: in function 'open'
	frontend/device/input.lua:316: in function 'waitEvent'
	frontend/ui/uimanager.lua:931: in function 'handleInput'
	frontend/ui/uimanager.lua:979: in function 'run'
	./reader.lua:210: in main chunk
	[C]: at 0x5638956bf170
~/src/kobo/koreader

Crash avoided with some casting, but it gets stuck in the gtk_events_pending() loop.

diff --git a/ffi/filechooser_gtk-3.lua b/ffi/filechooser_gtk-3.lua
index f8dbb9f..37f858c 100644
--- a/ffi/filechooser_gtk-3.lua
+++ b/ffi/filechooser_gtk-3.lua
@@ -21,7 +21,7 @@ Proof of concept, https://github.com/Alloyed/nativefiledialog might be better.
 function FileChooser:show(action, button, title)
     gtk.gtk_init(nil, nil)
 
-    local d = gtk.gtk_file_chooser_dialog_new(
+    local dialog = gtk.gtk_file_chooser_dialog_new(
         title,
         nil,
         action,
@@ -29,10 +29,10 @@ function FileChooser:show(action, button, title)
         button, ffi.cast("const gchar *", gtk.GTK_RESPONSE_OK),
         nil)
 
-    local response = gtk.gtk_dialog_run(d)
-    local filename = gtk.gtk_file_chooser_get_filename(d)
+    local response = gtk.gtk_dialog_run( ffi.cast("GtkDialog *", dialog) )
+    local filename = gtk.gtk_file_chooser_get_filename( ffi.cast("GtkFileChooser *", dialog) )
 
-    gtk.gtk_widget_destroy(d)
+    gtk.gtk_widget_destroy(dialog)
 
     while gtk.gtk_events_pending() do
         gtk.gtk_main_iteration()

android: unused DT entries

android linker shows a few warnings related to unused DT entries on the shared libraries. I'm using pidcat for easily logging:

pidcat org.koreader.launcher -t linker

                            Process org.koreader.launcher created for activity org.koreader.launcher/.MainActivity
                            PID: 23600   UID:    GIDs: 

                 # starting the program
                 linker  W  "/data/data/org.koreader.launcher/files/libs/libkoreader-lfs.so" unused DT entry: type 0xf arg 0x309
                         W  "/data/data/org.koreader.launcher/files/libs/libblitbuffer.so" unused DT entry: type 0xf arg 0x5e7
                         W  "/data/data/org.koreader.launcher/files/libs/libjpeg.so.8" unused DT entry: type 0xf arg 0xf7f
                         W  "/data/data/org.koreader.launcher/files/libs/libmupdf.so" unused DT entry: type 0xf arg 0xb4a0
                         W  "/data/data/org.koreader.launcher/files/libs/libwrap-mupdf.so" unused DT entry: type 0xf arg 0xb01
                         W  "/data/data/org.koreader.launcher/files/libs/libkoreader-xtext.so" unused DT entry: type 0xf arg 0x1479
                         W  "/data/data/org.koreader.launcher/files/common/serialize.so" unused DT entry: type 0xf arg 0x2af
                         W  "/data/data/org.koreader.launcher/files/rocks/lib/lua/5.1/lpeg.so" unused DT entry: type 0xf arg 0x4a9
                         W  "/data/data/org.koreader.launcher/files/libs/libcrypto.so.1.1" unused DT entry: type 0xf arg 0x15878
                         W  "/data/data/org.koreader.launcher/files/libs/libssl.so.1.1" unused DT entry: type 0xf arg 0x4c2f
                         W  "/data/data/org.koreader.launcher/files/common/ssl.so" unused DT entry: type 0xf arg 0x1235

                 # opening an epub
                         W  "/data/data/org.koreader.launcher/files/libs/libpng16.so.16" unused DT entry: type 0xf arg 0x153d
                         W  "/data/data/org.koreader.launcher/files/libs/libcrengine.so" unused DT entry: type 0xf arg 0x21c6a
                         W  "/data/data/org.koreader.launcher/files/libs/libkoreader-cre.so" unused DT entry: type 0xf arg 0x2172

                 # opening a pdf
                         W  "/data/data/org.koreader.launcher/files/libs/liblept.so.5" unused DT entry: type 0xf arg 0xc0ab
                         W  "/data/data/org.koreader.launcher/files/libs/libtesseract.so.3" unused DT entry: type 0xf arg 0x3c191
                         W  "/data/data/org.koreader.launcher/files/libs/libk2pdfopt.so.2" unused DT entry: type 0xf arg 0x51be

The warnings don't affect normal operation and probably won't need a fix, but termux is using https://github.com/termux/termux-elf-cleaner to remove these dynamic sections and the warnings.

Docker container. Cannot initialize SDL

Hi,
Want to use docker for developing instead of vagrant with VirtualBox
I've mounted koreader sources directories into docker

Modified virdevenv/docker/ubuntu/kokindle/Makefile

docker run --detach-keys "ctrl-q,ctrl-q"  -v /Users/me/Projects/kodev:/home/ko/kodev --rm -t -i $(USER)/kokindle:$(VERSION) bash -l

Then

./kodev build

Built without any errors. Then execute next command:

./kodev run

returned

ko@6f429626715a:~/kodev/koreader$ ./kodev run
[*] Building KOReader...
make -C base
make[1]: Entering directory `/home/ko/kodev/koreader/base'
# set up some needed paths and links
install -d build/x86_64-linux-gnu/{cache,history,clipboard,fonts} /home/ko/kodev/koreader/base/build/x86_64-linux-gnu/plugins/evernote.koplugin/thrift
ln -sf /home/ko/kodev/koreader/base/thirdparty/kpvcrlib/cr3.css build/x86_64-linux-gnu/data/
# setup Evernote SDK
cd /home/ko/kodev/koreader/base/thirdparty/evernote-sdk-lua/build/x86_64-linux-gnu/evernote-sdk-lua-prefix/src/evernote-sdk-lua && \
		cp -r *.lua evernote /home/ko/kodev/koreader/base/build/x86_64-linux-gnu/plugins/evernote.koplugin && \
		cp thrift/*.lua /home/ko/kodev/koreader/base/build/x86_64-linux-gnu/plugins/evernote.koplugin/thrift
test -e build/x86_64-linux-gnu/rocks/share/lua/5.1/re.lua && chmod 664 build/x86_64-linux-gnu/rocks/share/lua/5.1/re.lua || true  # hot fix re.lua permission
make[1]: Leaving directory `/home/ko/kodev/koreader/base'
install -d koreader-emulator-x86_64-linux-gnu/koreader
rm -f koreader-emulator-x86_64-linux-gnu/koreader/git-rev; echo v2015.11-658-g683b038 > koreader-emulator-x86_64-linux-gnu/koreader/git-rev
cp -f base/ev_replay.py koreader-emulator-x86_64-linux-gnu/koreader/
[*] create symlink instead of copying files in development mode
cd koreader-emulator-x86_64-linux-gnu/koreader && \
		ln -sf ../../base/build/x86_64-linux-gnu/* .
[*] install front spec only for the emulator
cd koreader-emulator-x86_64-linux-gnu/koreader/spec && test -e front || \
		ln -sf ../../../../spec ./front
cd koreader-emulator-x86_64-linux-gnu/koreader/spec/front/unit && test -e data || \
		ln -sf ../../test ./data
for f in reader.lua frontend resources defaults.lua datastorage.lua l10n tools README.md COPYING; do \
		ln -sf ../../$f koreader-emulator-x86_64-linux-gnu/koreader/; \
	done
[*] Install plugins
cp -r plugins/* koreader-emulator-x86_64-linux-gnu/koreader/plugins/
[*] Installresources
cp -r -pL resources/fonts/* koreader-emulator-x86_64-linux-gnu/koreader/fonts/
install -d koreader-emulator-x86_64-linux-gnu/koreader/{screenshots,data/{dict,tessdata},fonts/host,ota}
[*] Running KOReader with arguments: ...
~/kodev/koreader/koreader-emulator-x86_64-linux-gnu/koreader ~/kodev/koreader
---------------------------------------------
                launching...
  _  _____  ____                _
 | |/ / _ \|  _ \ ___  __ _  __| | ___ _ __
 | ' / | | | |_) / _ \/ _` |/ _` |/ _ \ '__|
 | . \ |_| |  _ <  __/ (_| | (_| |  __/ |
 |_|\_\___/|_| \_\___|\__,_|\__,_|\___|_|

 [*] Current time: 11/10/16-21:22:39

./luajit: ./ffi/SDL2_0.lua:38: cannot initialize SDL
stack traceback:
	[C]: in function 'error'
	./ffi/SDL2_0.lua:38: in function 'open'
	./ffi/framebuffer_SDL2_0.lua:13: in function 'init'
	./ffi/framebuffer.lua:59: in function 'new'
	frontend/device/sdl/device.lua:36: in function 'init'
	frontend/device.lua:39: in main chunk
	[C]: in function 'require'
	frontend/ui/uimanager.lua:1: in main chunk
	[C]: in function 'require'
	./reader.lua:111: in main chunk
	[C]: at 0x00404dc0
~/kodev/koreader
ko@6f429626715a:~/kodev/koreader$

Cervantes 3 model detection is incorrect

In #745 @pazos implemented support for the bq Cervantes line of devices, with the Cervantes 4 being the test device. I've had some time today to test out my Cervantes 3, and have found that koreader won't start. There's a few separate bugs which I'll describe here; I can resolve one of them but I usually program further up the stack and I'm hitting the limits of my low-level powers.

The first issue is that koreader wouldn't start. I tracked this down to frontend/device/cervantes/device.lua in koreader and will open a PR there -- it appears that there isn't a default/fall-through for unimplemented methods like hasNaturalLight, so I added a no method (the inverse of yes, defined at the top of that file) and managed to get koreader starting.

Unfortunately, the screen never does anything. fbink appears to work, and a bit of digging led me to the following:

11/10/18-09:41:21 DEBUG refresh: partial -1 -1 1074 1450
11/10/18-09:41:21 DEBUG mxc_update: 1072x1448 region @ (0, 0) with marker 43 (WFM: 257 & UPD: 0)
11/10/18-09:41:21 DEBUG MXCFB_SEND_UPDATE ioctl failed: Inappropriate ioctl for device

Google led me to strace and the following:

write(1, " mxc_update: 1072x1448 region @ "..., 73 mxc_update: 1072x1448 region @ (0, 0) with marker 43 (WFM: 257 & UPD: 0)) = 73
write(1, "\n", 1
)                       = 1
ioctl(4, 0x4040462e, 0x2c613458)        = -1 ENOTTY (Inappropriate ioctl for device)

I note that the second argument there, 0x4040462e is the MXCFB_SEND_UPDATE for the Pocketbook. The Cervantes, as per ffi/framebuffer_mxcfb.lua should be 0x4044462e.

So -- it appears that this argument is wrong, but I'm stuck getting much further here. The value in mxcfb_cervantes_h.lua is correct, and I believe that's getting imported in framebuffer_mxcfb.lua (EDIT -- ah, but that's based on some device detection and it checks if it isPocketBook before it gets to the Cervantes -- I'm digging here now).

If anyone can suggest what might've gone wrong, I'd appreciate it! I'm happy to open a PR when we manage to dig the fix up :).

Use after free w/ PicDocument image decoding or rendering & MuPDF scaling

c.f. #816 for the original crash report.

The quickest repro I could get is simply to alternate between opening a PNG and viewing its cover (possibly throwing an unscale/rescale pass in the process).

i.e.,

  1. Long-press on image > "View full size cover"
  2. Original size
  3. Scale
  4. Back to FM
  5. Tap on image to "open" it
  6. Back to FM
  7. Long-press on image > "View full size cover"
  8. KABLOOEY

Tracked down to ffi/pic.lua's doc.image_bb:setAllocated(1) in openPNGDocument.
I don't know when exactly it gets free'd, but stuff blows up in ffi/mupdf.lua's fz_scale_pixmap in scaleBlitBuffer.

I'm not familiar with the BB intricacies as far as the allocated flag is concerned, and the FFI + GC interactions make my brain hurt. (Plus, LodePNG's use of a double pointer for the output buffer doesn't help).

Deprecate SDL 1.2

SDL 1.2 has been deprecated since 2014 or so.

https://wiki.libsdl.org/Support

SDL 1.2 is deprecated and not recommended for new projects.

Qemu recently followed suit: https://patchwork.kernel.org/patch/10164537/

I haven't actually tested it in years, so the status of SDL 1.2 in KOReader is that it may not be broken.

I think that other than the base CI, nothing really uses it anymore.

- libsdl1.2-dev

I briefly entertained the idea of testing both SDL 1.2 as well as SDL 2 (cf. koreader/koreader#4631 (comment)) but the additional maintenance burden isn't worth it.

MuPDF 1.13

Pinging @TnS-hun :-)

It's not mentioned on https://mupdf.com/news.html yet but the 1.13.0 tag exists in the repo.

Because of the "Improvements to annotation editing API." mentioned in the changelog the build fails. I didn't investigate what changed exactly; this was just a quick test to see what would happen.

In file included from wrap-mupdf.c:24:0:
wrap-mupdf.h:119:21: error: unknown type name ‘fz_annot_type’; did you mean ‘fz_annot’?
     pdf_page *page, fz_annot_type type)
                     ^
wrap-mupdf.h:32:46: note: in definition of macro ‘MUPDF_WRAP’
     ret_type wrapper_name(fz_context *ctx, ##__VA_ARGS__);
                                              ^~~~~~~~~~~
In file included from wrap-mupdf.c:154:0:
wrap-mupdf.h:119:21: error: unknown type name ‘fz_annot_type’; did you mean ‘fz_annot’?
     pdf_page *page, fz_annot_type type)
                     ^
wrap-mupdf.h:49:53: note: in definition of macro ‘MUPDF_WRAP’
     extern ret_type wrapper_name(fz_context *ctx, ##__VA_ARGS__) { \
                                                     ^~~~~~~~~~~

screenshot_2018-04-20_16-42-46

Edit: btw, MuPDF Explored was updated too.

Makefile and CMake style

I want to get rid of uppercase commands in the CMake files. It's currently a bit of a hodgepodge with a majority lowercase and I think it negatively affects maintainability (i.e., it takes me more time to scan than I think it should). Cf. https://community.kde.org/Policies/CMake_Coding_Style#Upper.2Flower_casing

Enforcement isn't completely self-evident but cmake-lint would probably do the job for some basic checks, also see https://gitlab.kitware.com/cmake/cmake/issues/17441

On a related note, https://github.com/mrtazz/checkmake exists to do similar things for Makefiles. There's also https://github.com/softprops/mint but at a glance that doesn't seem to do much other than checking if all targets exist.

How to use in my app

how to include and use it in my app? Where can I find a sample? I need to create an app with inside a document viewer.

LPEG 1.0.2

@NiLuJe It doesn't seem to be in Gentoo yet. I suppose that'll be rectified soon enough?

http://www.inf.puc-rio.br/~roberto/lpeg/lpeg-1.0.2.tar.gz

The changelog isn't super informative:

HISTORY for LPeg 1.0.2

* Changes from version 1.0.1 to 1.0.2
  ---------------------------------
  + some bugs fixed

* Changes from version 0.12 to 1.0.1
  ---------------------------------
  + group "names" can be any Lua value
  + some bugs fixed
  + other small improvements

http://lua.2524044.n2.nabble.com/ANN-LPeg-1-0-2-td7685779.html

How to set up and/or use Busted properly?

make test uses the -l flag which seems to have been removed. I winged it for util_spec.lua by defining the function in my test file itself, but for more complicated stuff that won't do. Besides, I imagine this will affect the automated builds as well at some point in the future.

PocketBook isn't equipped to deal with custom TC location

tl;dr It's defeating the purpose of the Docker image by downloading it from scratch.

Needs a little adjustment in the definitions here as well as in the kodev script in front.

Something with the path as well as the SYSROOT var…

[[ ":$PATH:" == *":pocketbook-toolchain:"* ]]
[ -z "${SYSROOT}" ]

Android toolchain: what GCC (?) version?

After @NiLuJe 's awesome work, I'm a bit at loss - or rather, my Android toolchain is. I built it with the "android-toolchain" Makefile target, and it defaults to create a gcc-4.6 based one. That, however, does not know about "-mtune=generic-armv6".

So what toolchain do we want to use? And we should probably adapt the android-toolchain target to reflect that, I think.

Switch to 8bit buffers in some or all places

Blitbuffers are currently 4bit based. That looked like a good idea since it saves memory (e.g. on the KDX: ~500kB vs. ~1MB for a fullscreen buffer) and resembled the framebuffer's layout.

However, blitting is very slow for a lot of cases (odd source or target X coordinates and rotated or modified buffers, namely). And we lose information that we could use e.g. for contrast/gamma setting, dithering or other occasions. Painting single pixels is awful for the memory cache as well.

My suggestion is to convert a lot of the buffers to 8bpp and to do the final step to 4bpp only when blitting to the framebuffer. I think we could make it interoperable with existing code in KPV.

All our rendering backends generate >=8bpp renderings anyway - the higher memory usage would just be for cached tiles.

make fetchthirdparty failed

define USE_FONTCONFIG 1

MuPDF patch: use external fonts

cd mupdf && patch -N -p1 < ../mupdf.patch
can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?

The text leading up to this was:

|diff --git a/pdf/pdf_font.c b/pdf/pdf_font.c
|index 33a1a65..c2fdee3 100644
|--- a/pdf/pdf_font.c

|+++ b/pdf/pdf_font.c

File to patch:
Skip this patch? [y]
Skipping patch.
3 out of 3 hunks ignored
can't find file to patch at input line 48
Perhaps you used the wrong -p or --strip option?

The text leading up to this was:

|diff --git a/pdf/pdf_fontfile.c b/pdf/pdf_fontfile.c
|index 99565da..a91380f 100644
|--- a/pdf/pdf_fontfile.c

|+++ b/pdf/pdf_fontfile.c

File to patch:
Skip this patch? [y]
Skipping patch.
2 out of 2 hunks ignored

can not open pdf

get this error on my kindle and there is no problem on emulator
# cannot open document /mnt/us/documents/1.pdf ./ffi/koptcontext.lua:13 /lib/libc.so.6 version 'GLIBC_2.22' not found (required by libs/liblept.so.3)
compiled with GCC 5.3 shipped with ubuntu 16.04, and I dont know if this is the reason

random crashes in luajit gc because of memory corruption

Has anybody besides me ever experienced segfaults or memory corruption in koreader in a scenario like the one I describe below?

Sometimes, I get random segfault crashes when using the FileChooser. I only observe it when I change the order of requires in Lua source code, which ends up somehow triggering a bad state.

Another common strange thing which occurs for me is to get memory corruption in a single font glyph. Recently, I had the third character of "Table of contents" in the menu to display always with a corrupted glyph ("b" turned into a square). When using Portuguese i18n, I got corrupted a glyph present in the same string ("á" in "Sumário"). Then the same glyph displays incorrectly in any menu which uses the corresponding character. I think this might be also related to this issue, but of course I cannot be sure about that.

Back to the segfault, I had built koreader using this tree but removing the require "ui/widget/container" in uimanager.lua (which I originally had put there with the only purpose of avoiding this segfault!). In order to better debug the code, I checked out the latest stable LuaJIT repo, disabled JIT and enabled debug in LuaJIT Makefile. I also changed BASE_CFLAGS of koreader-base to -g etc.

diff --git a/Makefile.defs b/Makefile.defs
index fbb91f9..bcee582 100644
--- a/Makefile.defs
+++ b/Makefile.defs
@@ -43,7 +43,7 @@ HOSTCXX:=g++
 HOSTAR:=ar

 # Base CFLAGS, without arch. We'll need it for luajit, because its Makefiles do some tricky stuff to differentiate HOST/TARGET
-BASE_CFLAGS:=-O2 -ffast-math -pipe -fomit-frame-pointer
+BASE_CFLAGS:=-g -pipe


 # Misc GCC tricks to ensure backward compatibility with the K2, even when using a fairly recent TC (Linaro/MG).
diff --git a/src/Makefile b/src/Makefile
index 0469934..f82f762 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -57,7 +57,7 @@ CCOPT_mips=
 #
 CCDEBUG=
 # Uncomment the next line to generate debug information:
-#CCDEBUG= -g
+CCDEBUG= -g
 #
 CCWARN= -Wall
 # Uncomment the next line to enable more warnings:
@@ -103,7 +103,7 @@ XCFLAGS=
 #XCFLAGS+= -DLUAJIT_ENABLE_LUA52COMPAT
 #
 # Disable the JIT compiler, i.e. turn LuaJIT into a pure interpreter.
-#XCFLAGS+= -DLUAJIT_DISABLE_JIT
+XCFLAGS+= -DLUAJIT_DISABLE_JIT
 #
 # Some architectures (e.g. PPC) can use either single-number (1) or
 # dual-number (2) mode. Uncomment one of these lines to override the

After playing a little with the FileChooser, I get the following backtrace:

(gdb) bt
#0  gc_mark (g=g@entry=0x40e401f0, o=<optimized out>) at lj_gc.c:58
#1  0x40a6d51c in gc_traverse_tab (t=0x416de6b8, g=0x40e401f0) at lj_gc.c:193
#2  propagatemark (g=0x40e401f0, g@entry=0x5f4) at lj_gc.c:307
#3  0x40a6deac in gc_onestep (L=0x40e401f0, L@entry=0x40e401c0) at lj_gc.c:611
#4  0x40a6e334 in lj_gc_step (L=0x40e401c0) at lj_gc.c:676
#5  0x40a6a7c4 in $a () at lj_vm.s:648
#6  0x40a76f74 in lua_pcall (L=<optimized out>, nargs=<optimized out>, nresults=<optimized out>, errfunc=<optimized out>) at lj_api.c:1052
#7  0x0000edf4 in docall (L=0x40e401c0, narg=0, clear=1) at koreader-base.c:77
#8  0x0000f078 in main (argc=4, argv=0xbe9efca4) at koreader-base.c:121
(gdb) info reg
r0             0x40e401f0   1088684528
r1             0x40000000   1073741824
r2             0x0  0
r3             0x1  1
r4             0x40000000   1073741824
r5             0x40e401f0   1088684528
r6             0x1  1
r7             0x416dd450   1097716816
r8             0x0  0
r9             0x0  0
r10            0x40e401f0   1088684528
r11            0x1  1
r12            0x5e 94
sp             0xbe9efa28   0xbe9efa28
lr             0x40a6d51c   1084675356
pc             0x40a6d1f4   0x40a6d1f4 <gc_mark+28>
cpsr           0x80000010   2147483664
(gdb) disassemble
Dump of assembler code for function gc_mark:
   0x40a6d1d8 <+0>: push    {r3, r4, r5, lr}
   0x40a6d1dc <+4>: mov r5, r0
   0x40a6d1e0 <+8>: mov r4, r1
   0x40a6d1e4 <+12>:    ldrb    r2, [r4, #4]
   0x40a6d1e8 <+16>:    ldrb    r3, [r4, #5]
   0x40a6d1ec <+20>:    and r2, r2, #252    ; 0xfc
   0x40a6d1f0 <+24>:    cmp r3, #12
=> 0x40a6d1f4 <+28>:    strb    r2, [r4, #4]
   0x40a6d1f8 <+32>:    beq 0x40a6d21c <gc_mark+68>
   0x40a6d1fc <+36>:    cmp r3, #5
   0x40a6d200 <+40>:    beq 0x40a6d258 <gc_mark+128>
   0x40a6d204 <+44>:    cmp r3, #10
[...]

I will try to debug further, but does anyone have some idea on what might be wrong?

mupdf tar fixes

Building with system libraries

I've set up a buildroot for my ereader, and I am already building many of the libraries used by koreader. What would be the best way to direct the build to allow searching for system libraries instead of vendoring?

Why "platform=android-9" is fixed used when building on android?

I think platform=android-x is an open selection when building on different android targets (2.3, 4.0, 4.1, 4.4 or 5.x). But I found in nearly all android related build scripts the "platform=android-9" is hard-coded. May I ask why should it be done like this rather than providing it for user determination? Thanks

Download GNU deps from mirror

This is a simple one. https://ftp.gnu.org/ is currently offline but we shouldn't actually be using it anyway.

https://www.gnu.org/server/mirror.en.html

First, for users/downloaders: the address http://ftpmirror.gnu.org/ multiplexes between the mirrors, trying to choose one that is nearby and up to date. E.g., http://ftpmirror.gnu.org/emacs/ goes to a mirror's directory of GNU Emacs. We recommend using this generic ftpmirror.gnu.org address wherever possible in links, documentation, etc., to reduce load on the main GNU server.

tl;dr http://ftpmirror.gnu.org should be more stable.

Mupdf: create FFI-based binding

Just a heads-up: I'm almost done converting the Lua-API based mupdf binding to a FFI-based one. In the first step, I'll just leave the Lua-facing API as-is, i.e. there will be the same methods with the same signatures. We might want to clean that up a bit later.

I'm planning to do the same with djvu and maybe coolreader lib, but one step at a time.

I'm solving the problem of mupdf's exception handling by creating wrapper functions that will catch the exception and return a designated "error" value.

I'll send a PR when it's ready for preview. And I might just add a bunch of tests, too, while I'm at it.

Fix Kobo Mark7 mxcfb handling

Creating a new issue, as a followup of #390 ;).

The Mark7 kernel exposes a new API, but still appears to support the old one (for now?).
There's currently code in to handle the new API, but it's commented out.

Long story short, much like in the issue mentioned earlier, I need a strace dump from Nickel, on a Mark7 device running the latest FW, to decide if we need to flip the switch to the new API on those devices ;).

Quoting the relevant instructions:


The binaries are available here.

Basically, you need SSH access under Nickel, and with the attached binary, you trace nickel's ioctls, like so: ./strace -fitv -e trace=ioctl -p $(pidof nickel). Then you move around a bit, first in the Library, then in a book (where you make sure you do so long enough to trigger at least once a flashing update), moving around in a couple of menus while you're there, and then post the output here ;).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.