kiwix / kiwix-build Goto Github PK

Kiwix & openZIM build engine

License: GNU General Public License v3.0

Python 87.84% Shell 4.10% Batchfile 4.26% Makefile 2.82% JavaScript 0.98%

kiwix-build's Introduction

Kiwix Build

Kiwix Build provides advanced tools to (cross-)compile easily Kiwix & openZIM softwares and libraries and deploy them. They have been tested on Fedora 35+ & Ubuntu 20.04+.

Kiwix Build audience is:

Advanced users who don't want/can handle all the dependencies compilations manually
Kiwix developer team for its own CI/CD

Prerequisites

You will need a recent version of Meson (>= 0.34) and Ninja (>= 1.6) If your distribution provides a recent enough versions for them, just install them with your package manager. Continue to read the instructions otherwise.

Before anything else you need to install Python3 related tools. On Debian based systems:

sudo apt-get install python3-pip virtualenv

Create a virtual environment to install python module in it instead of modifying the system.

virtualenv -p python3 ./ # Create virtualenv
source bin/activate      # Activate the virtualenv

Then, download and install kiwix-build and its dependencies:

git clone https://github.com/kiwix/kiwix-build.git
cd kiwix-build
pip3 install .
hash -r # Refresh bash paths

Compilation

The compilation is handled by the kiwix-build command. It will compile everything. If you are using a supported platform (Redhat or Debian based) it will install missing packages using sudo. You can get kiwix-build usage like this:

kiwix-build --help

Target

You may want to compile a specific target so you will have to specify it on the

command line:

kiwix-build libkiwix # will build kiwix-build and its dependencies
kiwix-build kiwix-desktop # will build kiwix-desktop and its dependencies
kiwix-build zim-tools # will build zim-tools and its dependencies

By default, kiwix-build will build kiwix-tools .

To see the whole list of available targets run with non existing target, ex:

kiwix-build not-existing-target
...
invalid choice: 'not-existing-target' (choose from 'alldependencies', 'android-ndk',
...

Config

If no config is specified, the default will be native_dyn.

You can select another config using the option --config. For now, there is ten different supported platforms:

native_dyn
native_mixed
native_static
win32_dyn
win32_static
android
android_arm
android_arm64
android_x86
android_x86_64
flatpak

All native_* config means using the native compiler without any cross-compilation option. Other may simply use cross-compilation or may download a specific toolchain to use.

If you want to compile kiwix-tools for win32 using static linkage:

kiwix-build --config win32_dyn

Android

kiwix-android (https://github.com/kiwix/kiwix-android) depends of the libkiwix project.

When building libkiwix, you should directly use the target-platform android_<arch>:

kiwix-build libkiwix --config android_arm

You may directly use the special config android which will build different android architectures

kiwix-build --config android libkiwix

By default, it will build for all android architecture, you can limit this with option --android-arch:

kiwix-build libkiwix --config android --android-arch arm # aar with arm architecture
kiwix-build libkiwix --config android --android-arch arm --android-arch arm64 # aan with arm and arm64 architectures

To build kiwix-android itself, you should see the documentation of kiwix-android.

iOS

When building for ios, we may want to compile a "fat library", a library for several architectures.

To do so, you should directly use the target-platfrom ios_multi. As for android, kiwix-build will build the library several times (once for each platform) and then create the fat library.

kiwix-build --config iOS_multi libkiwix

You can specify the supported architectures with the option --ios-arch:

kiwix-build --config iOS_multi libkiwix # all architetures
kiwix-build --config iOS_multi --ios-arch arm --ios-arch arm64 # arm and arm64 arch only

Outputs

Kiwix-build.py will create several directories:

ARCHIVES: All the downloaded archives go there.
SOURCES: All the sources (extracted from archives and patched) go there.
BUILD_<config>: All the build files go there.
BUILD_<config>/INSTALL: The installed files go there.
BUILD_<config>/LOGS: The logs files of the build.

If you want to install all those directories elsewhere, you can pass the --working-dir option to kiwix-build:

Troubleshooting

If you need to install Meson "manually":

virtualenv -p python3 ./ # Create virtualenv
source bin/activate      # Activate the virtualenv
pip3 install meson       # Install Meson
hash -r                  # Refresh bash paths

If you need to install Ninja "manually":

wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip
unzip ninja-linux.zip ninja -d $HOME/bin

License

GPLv3 or later, see LICENSE for more details.

kiwix-build's People

Contributors

Stargazers

Watchers

kiwix-build's Issues

Verify that Windows cross-compile works fine on Ubuntu 16.04

Integrate fat lib building procedure for kiwix-lib targeting iOS

A lot of times we need to use lipo to combine libs for multiple arches into one fat lib. It would be a great idea to add this function to kiwix-build.

macOS: ld: library not found for -lrt

on macOS, build natively, encountered the following error
lrt_error.txt

Quote comment by @mgautierfr in issue #9

For what I see, clang cannot found librt. The flag -lr comes from the xapian pkg-config if the configure detect (and found) the timer_create function. This part was a bit buggy and I fix it with the e6673e2.
Try to remove all xapian and kiwix-tools stuff (SOURCE and BUILD) and launch a new compilation.
If it doesn't work, you will have to fix xapian (first find if you really need librt, and then find either why xapian add it or why kiwix-tools cannot found it)

The question is:

I assume @mgautierfr means -lrt is carried from xapian. (There was no flag -lr) If this is true, how come xapian build successfully?
How do I know if librt is needed? I don't know what it does and how xapian use it

Unable to compile pugixml on macOS

Running ./kiwix-build.py

prepare sources kiwix-tools :
  gitclone kiwix-tools : OK
  gitupdate kiwix-tools : OK
[BUILD]
build pugixml :
  configure pugixml : OK
  compile pugixml : OK
  test pugixml : ERROR
Traceback (most recent call last):
  File "/Users/kelson/kiwix-build/bin/mesontest", line 19, in <module>
    from mesonbuild import mesonmain
  File "/Users/kelson/kiwix-build/lib/python3.6/site-packages/mesonbuild/mesonmain.py", line 18, in <module>
    from . import environment, interpreter, mesonlib
  File "/Users/kelson/kiwix-build/lib/python3.6/site-packages/mesonbuild/environment.py", line 17, in <module>
    from . import coredata
  File "/Users/kelson/kiwix-build/lib/python3.6/site-packages/mesonbuild/coredata.py", line 20, in <module>
    from .mesonlib import MesonException, commonpath
  File "/Users/kelson/kiwix-build/lib/python3.6/site-packages/mesonbuild/mesonlib.py", line 60, in <module>
    meson_command = python_command + [detect_meson_py_location()]
  File "/Users/kelson/kiwix-build/lib/python3.6/site-packages/mesonbuild/mesonlib.py", line 51, in detect_meson_py_location
    raise RuntimeError('Could not determine how to run Meson. Please file a bug with details.')
RuntimeError: Could not determine how to run Meson. Please file a bug with details.
run command 'mesontest --verbose'
current directory is '/Users/kelson/kiwix-build/BUILD_native_dyn/pugixml-1.2'
env is :
  TERM_PROGRAM : 'Apple_Terminal'
  SHELL : '/bin/bash'
  TERM : 'xterm-256color'
  TMPDIR : '/var/folders/s7/frvr3wkd3_34_kkd353f70k40000gn/T/'
  Apple_PubSub_Socket_Render : '/private/tmp/com.apple.launchd.snnquLCEZx/Render'
  TERM_PROGRAM_VERSION : '388.1.1'
  OLDPWD : '/Users/kelson/kiwix-build/ninja'
  TERM_SESSION_ID : '70AA5DAE-B08E-4294-BDBE-01DE72B3E10E'
  USER : 'kelson'
  SSH_AUTH_SOCK : '/private/tmp/com.apple.launchd.l21yy24m1l/Listeners'
  __CF_USER_TEXT_ENCODING : '0x1F5:0x0:0x0'
  VIRTUAL_ENV : '/Users/kelson/kiwix-build'
  PATH : '/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/bin:/Users/kelson/kiwix-build/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin'
  PWD : '/Users/kelson/kiwix-build'
  XPC_FLAGS : '0x0'
  PS1 : '(kiwix-build) \\h:\\W \\u\\$ '
  XPC_SERVICE_NAME : '0'
  SHLVL : '1'
  HOME : '/Users/kelson'
  LOGNAME : 'kelson'
  LC_CTYPE : 'UTF-8'
  SECURITYSESSIONID : '186a8'
  _ : './kiwix-build.py'
  PKG_CONFIG_PATH : ':/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/lib/pkgconfig'
  LD_LIBRARY_PATH : ':/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/lib:/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/lib'
  CPPFLAGS : '-I/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/include '
  LDFLAGS : '-L/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/lib -L/Users/kelson/kiwix-build/BUILD_native_dyn/INSTALL/lib '

Stopping build due to errors

macOS: error when build kiwix-lib natively

Using code from branch apple-platforms, building on macOS natively, encountered errors when building kiwix-lib. Please see log for detail.
log.txt

Automatically install ninja

It's not compulsory, but would simplify the installation by avoiding some manual steps

kiwix-build install dir should be customizable

The scripts should allow to:

specify the install directory on the command line (necessary for the packaging)
run the install separatly from the compilation

how can I do a custom android app build on a file already downloaded

Alert for "downloading list of catalogs" is not dismissable

The pop up indicates "do not ask me again," which only registers with the app if you say "yes" and not if you say "no." So you have to dismiss it all the time. (Saw this on Reno's computer at the hackathon)

Meson encountered an error in file meson.build, line 39, column 4:

Error encountered: For custom ctpp2_prefix_install you need a meson version >=0.31.0

root@deb9-demo:/opt/tools/kiwix-build# meson -v
0.37.1

Write nightly builds at the right place

in http://download.kiwix.org/nightly/bin//

kiwix-build.py should detect early if a dependency (svn for example) is not available (and complain)

on a fresh new Debian stretch install, I followed the instructions of this readme and Kiwix failed to compile because SVN was missing. it should detect that and just install subversion through apt-get install subversion.

(similarly, why aren't meson and ninja installed that way? just apt-get install meson was enough here...)

Use the provided CMake file to build pugixml.

Pugixml has a Cmake build system who generate a pkg-config file.
We do not need a custom meson file.

See kiwix/libkiwix#49

Fix dependency nightmare with icu4c and mingw32.

With the recent update of glibc 2.26 (2017-08-02), the xlocale.h header has been removed (https://sourceware.org/glibc/wiki/Release/2.26#Removal_of_.27xlocale.h.27)
icu4c 58.1 is using xlocale.h
icu project has fix this in the release 60.1. => So we will have to use last icu version to compile on system with recent glibc. (Fedora 27 at least)
icu4c 60.1 is using the function ResolveLocaleName on Windows
mingw32 doesn't export this function in kernel32. There is patch for this (https://sourceforge.net/p/mingw-w64/mingw-w64/ci/6dc5579d5e71f2662316416502b01c2d3ace0232) but no release has been made yet.
Once mingw32 made a release, distributions will have to update their packages and travis (who uses ubuntu) will have to update its image to allow use to compile all libzim/kiwix stack again for win32.

Cross-compile kiwix-tools on MIPS

This is to get kiwix-serve working on most OpenWRT devices

Make CI temporary dependencies available through HTTP

These dependencies are necessary to compile many repositories without having each time to recompile everything and even more important without having to use kiwix-build.

This dependencies needs to be also available to everybody, otherwise the CI won't be able to test merge requests hosted on external git repositories.

Crash by trying to build custom app

$ ./kiwix-build.py --target-platform android_arm --android-custom-app wikimed --zim-file-size $ZIM_SIZE kiwix-android-custom
[INSTALL PACKAGES]
 - automake : SKIP
 - cmake : SKIP
 - git : SKIP
 - subversion : SKIP
 - ccache : SKIP
 - pkg-config : SKIP
 - gcc : SKIP
 - default-jdk : SKIP
SKIP, No package to install.
[PREPARE]
prepare sources for toolchain android-ndk :
  download android-ndk : SKIP
  extract android-ndk : SKIP
prepare sources for toolchain android-sdk :
  download android-sdk : SKIP
  extract android-sdk : SKIP
prepare sources Gradle :
  download Gradle : SKIP
  extract Gradle : SKIP
prepare sources pugixml :
  download pugixml : SKIP
  extract pugixml : SKIP
  patch pugixml : SKIP
prepare sources zlib :
  download zlib : SKIP
  extract zlib : SKIP
  patch zlib : SKIP
prepare sources lzma :
  download lzma : SKIP
  extract lzma : SKIP
prepare sources uuid :
  download uuid : SKIP
  extract uuid : SKIP
prepare sources xapian-core :
  download xapian-core : SKIP
  extract xapian-core : SKIP
  patch xapian-core : SKIP
prepare sources icu4c :
  svncheckout icu4c_native : SKIP
  svnupdate icu4c_native : OK
  patch icu4c_native : SKIP
prepare sources libzim :
  gitclone libzim : SKIP
  gitupdate libzim : OK
prepare sources kiwix-lib :
  gitclone kiwix-lib : SKIP
  gitupdate kiwix-lib : OK
prepare sources kiwix-android :
  gitclone kiwix-android : SKIP
  gitupdate kiwix-android : OK
prepare sources kiwix-android-custom :
  gitclone kiwix-android-custom : SKIP
  gitupdate kiwix-android-custom : OK
[BUILD]
build toolchain android-ndk :
  build_platform android-ndk : OK
  fix_permission_right android-ndk : OK
build toolchain android-sdk :
  build_platform android-sdk : SKIP
  fix_licenses android-sdk : SKIP
build Gradle :
  install Gradle : OK
build pugixml :
  configure pugixml : OK
  compile pugixml : OK
  install pugixml : OK
build zlib :
  pre_build_script zlib : OK
  configure zlib : OK
  compile zlib : OK
  install zlib : OK
build lzma :
  configure lzma : OK
  compile lzma : OK
  install lzma : OK
build uuid :
  configure uuid : OK
  compile uuid : OK
  install uuid : OK
build xapian-core :
  configure xapian-core : OK
  compile xapian-core : OK
  install xapian-core : OK
build icu4c_native :
  configure icu4c_native : OK
  compile icu4c_native : OK
  install icu4c_native : SKIP
build icu4c_cross-compile :
  configure icu4c_cross-compile : OK
  compile icu4c_cross-compile : OK
  install icu4c_cross-compile : OK
build libzim :
  configure libzim : OK
  compile libzim : OK
  install libzim : OK
build kiwix-lib :
  configure kiwix-lib : OK
  compile kiwix-lib : OK
  install kiwix-lib : OK
build kiwix-android :
SKIP
build kiwix-android-custom :
  configure kiwix-android-custom : ERROR
Traceback (most recent call last):
  File "./kiwix-build.py", line 947, in <module>
    builder.run()
  File "./kiwix-build.py", line 885, in run
    self.build()
  File "./kiwix-build.py", line 875, in build
    builder.build()
  File "/home/kelson/kiwix-build/dependencies.py", line 412, in build
    self.command('configure', self._configure)
  File "/home/kelson/kiwix-build/dependency_utils.py", line 206, in command
    return self.target.command(*args, **kwargs)
  File "/home/kelson/kiwix-build/dependency_utils.py", line 50, in command
    ret = function(*args, context=context)
  File "/home/kelson/kiwix-build/dependencies.py", line 441, in _configure
    shutil.copytree(pj(self.buildEnv.install_dir, 'kiwix-lib'), pj(self.build_path, 'kiwixlib', 'src', 'main'))
  File "/home/kelson/kiwix-build/lib/python3.5/shutil.py", line 303, in copytree
    names = os.listdir(src)
FileNotFoundError: [Errno 2] No such file or directory: '/home/kelson/kiwix-build/BUILD_android_arm/INSTALL/kiwix-lib'

Limit concurrent kiwix-build jobs to prevent 40 min backlog

We should limit the concurrent kiwix-build jobs in travis to 3-4 so that other repos pull requests are not blocked for 50 mins+ as was the case for me tonight.

Modify the code to work on Jessie (python 3.4.2 and context argument)

The default python3 on Raspbian Jessie is 3.4.2

The current code uses 'context' which is first available in Python 3.4.3

Let's see if we can find a way to work on 3.4.2 so kiwix-build can run on Raspbian Jessie.

macOS: error when build kiwix-tools natively

kiwix-build stopped when building kiwix-tools, seems to sdo with -lrt. I am building using apple-platforms branch.

The log is here:
log.txt

increase xapian version dependency

Currently, kiwix-build builds xapian 1.4.0
When compiling zimwriterfs (required for mwoffliner to work), the compilation fails because it requires at least xapian 1.4.2

Expected: compile xapian 1.4.2 in kiwix-build, so that it allows to compile zimwriterfs too

kiwix-index - where is it now ?

Hi,

I'm trying to compile kiwix for an arm64 system but it seems to be missing kiwix-index ?

I've cloned the repo and built everything but kiwix-index seems to be missing

Many thanks

macOS: cannot run kiwis-build.py

Hi @mgautierfr, I am trying to run the build script on macOS 10.12.3, with python 3.6, targeting macOS.
I executed sudo python3 kiwix-build.py --build-static build, got the following error message ERROR: kiwix-build is intented to run only on Linux platform
What should I do?

Bring back the Android universal/multiarch APK

This is essential for people not relying on the Google Play store. This should work for both custom apps and standard Android app.

Impossible to build custom apps

It seems there is a bug with Travis and the Requests. I have open a ticket here:
travis-ci/travis-ci#7982

CI kiwix-build basic integration with Travis

On each commit the CI should build:

On Ubuntu, compile statically kiwix-lib with local deps (downloaded)
On Ubuntu, compile statically kiwix-tools with local deps (downloaded)
On Ubuntu, cross-compile for windows statically local deps (downloaded)

Compile (and publish as nightly) zimwriterfs

Publish kiwix-tools for i586 and i686 archictectures

From @holta on November 22, 2017 18:20

Currently on OLPC's 32-bit XO laptops in the developing world we are using:
http://download.kiwix.org/bin/0.9/kiwix-0.9-linux-i686.tar.bz2 (30-Oct-2014)

As this is more than 3 years old and failing with new content, we're also experimenting with:
http://download.kiwix.org/bin/0.10/kiwix-0.10-linux-i686.tar.bz2 (26-Oct-2016)

With essentially all new ZIM content coming online at the end of 2017 having an internal index in the ZIM file, the above strategies are no longer working :-(

Can compiler flags hopefully be set to publish kiwix-tools (kiwix-serve, kiwix-manage) for the i686 architecture as well? Ideally publishing to http://download.kiwix.org/nightly/ and also upholding the i686 tradition @ http://download.kiwix.org/bin/ ?

Context: 3 million of OLPC's XO laptops were manufactured, and several 100,000 are today in active use, where Wikipedia on Kiwix is often the most adored of all apps/content when it is installed! A special thank you to @georgejhunt & @darkenvy whose field community/integration has made this work a centerpiece in Haiti, Ghana, Mexico etc, where we distribute different versions of WikipediaFR, WikipediaHT, WikipediaEN and WikipediaES.

Copied from original issue: kiwix/kiwix-tools#112

requirements_build_custom_app.txt is missing in git repo

Impossible to go further with that documentation:
https://github.com/kiwix/kiwix-build/wiki/Build-and-publish-a-custom-app

Dependence between kiwix and openzim.

The recent PR #21 allow kiwix-build to build the zimwriterfs project.
This add no extra dependencies between the kiwix projects and zimwriterfs. It just said to kiwix-build how to build zimwriterfs.

This PR leads to a discussion between @kelson42 and me about what is the difference between the two organization openzim and kiwix.
I will try to be objective here, but I may not succeed, I'm part of the discussion. @kelson42, do not hesitate to correct or complete me.

The main argument of @kelson42 is that the openzim and kiwix are two different projects and must stay as it. People should be able to use zim (through zimlib, zimwriterfs or any other tools) without using at all kiwix or kiwix-related projects. Thus, he asks me to think of how we could separate the two projects and do not add zimwriterfs to kiwix-build. (But we agree that for now, it worth to accept the PR as we could easily remove zimwriterfs or split/duplicate kiwix-build if necessary)

I globally agree with this will. The zim format is a (somehow advanced) archive format and should stay as it. Kiwix is a application that use the zim format to store and display web content (mainly mediawiki content, but not only)

But, saying that, I don't see why the kiwix team couldn't have tools to use and compile zimwriterfs.
We already have tool to compile other dependencies as xapian, pugixml, .. and zimlib.
zimwriterfs is not intended to be used by kiwix end user, but I don't see why we (kiwix) should avoid publish a tool to compile it. If someone asks about it, we just could answer «At kiwix, we build zimwriterfs this way. It simpler this way, but do as you want. Use it, or not, we don't care.»
We already greatly use zimlib (kiwix/kiwix-lib and thus kiwix/kiwix-tools, kiwix/kiwix-android, kiwix/kiwix-IOS, kiwix/node-libzim) and zimwriterfs (kiwix/mwoffliner, kiwix/sotoki, kiwix/phet, kiwix/youtube, kiwix/TED, kiwix/gutemberg). And I'm not aware of other projects using libzim.

The other solutions are to duplicate kiwix-build or move kiwix-build to a generic and independent builder. And I'm appealing by none of them.

Error building iOS_arm64 on macOS 10.13.2

output:

(venv) Chriss-MBP:kiwix-build chrisli$ ./kiwix-build.py kiwix-lib --target-platform iOS_arm64
WARNING: kiwix-build has not been tested on MacOS platfrom.
Tests, bug reports and patches are welcomed.
[INSTALL PACKAGES]
SKIP
[PREPARE]
prepare sources pugixml :
  download pugixml : SKIP
  extract pugixml : SKIP
  patch pugixml : SKIP
prepare sources zlib :
  download zlib : SKIP
  extract zlib : SKIP
  patch zlib : SKIP
prepare sources lzma :
  download lzma : SKIP
  extract lzma : SKIP
prepare sources xapian-core :
  download xapian-core : SKIP
  extract xapian-core : SKIP
prepare sources icu4c :
  svncheckout icu4c_native : SKIP
  svnupdate icu4c_native : OK
  patch icu4c_native : SKIP
prepare sources libzim :
  gitclone libzim : SKIP
  gitupdate libzim : OK
prepare sources kiwix-lib :
  gitclone kiwix-lib : SKIP
  gitupdate kiwix-lib : OK
[BUILD]
build pugixml :
  configure pugixml : ERROR
Traceback (most recent call last):
  File "./kiwix-build.py", line 1030, in <module>
    builder.run()
  File "./kiwix-build.py", line 964, in run
    self.build()
  File "./kiwix-build.py", line 948, in build
    builder.build()
  File "/Users/chrisli/Developer/kiwix-build/dependency_utils.py", line 224, in build
    self.command('configure', self._configure)
  File "/Users/chrisli/Developer/kiwix-build/dependency_utils.py", line 219, in command
    return self.target.command(*args, **kwargs)
  File "/Users/chrisli/Developer/kiwix-build/dependency_utils.py", line 50, in command
    ret = function(*args, context=context)
  File "/Users/chrisli/Developer/kiwix-build/dependency_utils.py", line 346, in _configure
    if not self.target.force_native_build and self.buildEnv.meson_crossfile:
  File "./kiwix-build.py", line 342, in __getattr__
    return getattr(self.options, name)
AttributeError: 'Namespace' object has no attribute 'meson_crossfile'

Integrate post-command `change_install_name` for movable kiwix-lib on mac_os

To be able to move library on mac_os we need to change the post script to change the install_name "inside" the binary.

Error building on macOS 10.13.1

(kiwix-build) Chriss-MacBook-Pro:kiwix-build chrisli$ ./kiwix-build.py --help
usage: kiwix-build.py [-h] [--working-dir WORKING_DIR] [--libprefix LIBPREFIX]
                      [--target-platform {native_dyn,native_static,win32_dyn,win32_static,armhf_dyn,armhf_static,android_arm,android_arm64,android_mips,android_mips64,android_x86,android_x86_64,iOS_arm64}]
                      [--verbose] [--hide-progress] [--skip-source-prepare]
                      [--build-deps-only] [--no-cert-check] [--clean-at-end]
                      [--android-custom-app ANDROID_CUSTOM_APP]
                      [--zim-file-url ZIM_FILE_URL]
                      [--zim-file-size ZIM_FILE_SIZE]
                      [TARGET]

positional arguments:
  TARGET

optional arguments:
  -h, --help            show this help message and exit
  --working-dir WORKING_DIR
  --libprefix LIBPREFIX
  --target-platform {native_dyn,native_static,win32_dyn,win32_static,armhf_dyn,armhf_static,android_arm,android_arm64,android_mips,android_mips64,android_x86,android_x86_64,iOS_arm64}
  --verbose, -v         Print all logs on stdout instead of in specific log
                        files per commands
  --hide-progress       Hide intermediate progress information.
  --skip-source-prepare
                        Skip the source download part
  --build-deps-only     Build only the dependencies of the specified targets.

advanced:
  --no-cert-check       Skip SSL certificate verification during download
  --clean-at-end        Clean all intermediate files after the (successfull)
                        build

custom app:
  Android custom app specific options

  --android-custom-app ANDROID_CUSTOM_APP
                        The custom android app to build
  --zim-file-url ZIM_FILE_URL
                        The url of the zim file to download
  --zim-file-size ZIM_FILE_SIZE
                        The size of the zim file.
(kiwix-build) Chriss-MacBook-Pro:kiwix-build chrisli$ ./kiwix-build.py --target-platform native_dyn
WARNING: kiwix-build has not been tested on MacOS platfrom.
Tests, bug reports and patches are welcomed.
[INSTALL PACKAGES]
 - autoconf : SKIP
 - automake : SKIP
 - libtool : SKIP
 - cmake : SKIP
 - pkg-config : SKIP
SKIP, No package to install.
[PREPARE]
prepare sources pugixml :
  download pugixml : OK1.50%
  extract pugixml : OK
  patch pugixml : OK
prepare sources zlib :
  download zlib : OK0.41%
  extract zlib : OK
  patch zlib : OK
prepare sources lzma :
  download lzma : OK0.02%
  extract lzma : OK
prepare sources xapian-core :
  download xapian-core : OK0.26%
  extract xapian-core : OK
prepare sources icu4c :
  svncheckout icu4c : OK
  svnupdate icu4c : OK
  patch icu4c : OK
prepare sources libzim :
  gitclone libzim : OK
  gitupdate libzim : OK
prepare sources kiwix-lib :
  gitclone kiwix-lib : OK
  gitupdate kiwix-lib : OK
prepare sources libmicrohttpd :
  download libmicrohttpd : OK0.31%
  extract libmicrohttpd : OK
prepare sources kiwix-tools :
  gitclone kiwix-tools : OK
  gitupdate kiwix-tools : OK
[BUILD]
build pugixml :
  configure pugixml : OK
  compile pugixml : OK
  test pugixml : OK
  install pugixml : OK
build zlib :
  pre_build_script zlib : OK
  configure zlib : OK
  compile zlib : OK
  install zlib : OK
build lzma :
  configure lzma : OK
  compile lzma : OK
  install lzma : OK
build xapian-core :
  configure xapian-core : OK
  compile xapian-core : OK
  install xapian-core : OK
build icu4c :
  configure icu4c : OK
  compile icu4c : OK
  install icu4c : OK
build libzim :
  configure libzim : OK
  compile libzim : ERROR
[1/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/file_part.cpp.o' -MF 'src/zim@sha/file_part.cpp.o.d' -o 'src/zim@sha/file_part.cpp.o' -c ../../SOURCE/libzim/src/file_part.cpp
[2/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/articlesource.cpp.o' -MF 'src/zim@sha/articlesource.cpp.o.d' -o 'src/zim@sha/articlesource.cpp.o' -c ../../SOURCE/libzim/src/articlesource.cpp
[3/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/envvalue.cpp.o' -MF 'src/zim@sha/envvalue.cpp.o.d' -o 'src/zim@sha/envvalue.cpp.o' -c ../../SOURCE/libzim/src/envvalue.cpp
[4/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/fileheader.cpp.o' -MF 'src/zim@sha/fileheader.cpp.o.d' -o 'src/zim@sha/fileheader.cpp.o' -c ../../SOURCE/libzim/src/fileheader.cpp
[5/45] cc  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -std=c11 -O0 -g -pthread -MMD -MQ 'src/zim@sha/md5.c.o' -MF 'src/zim@sha/md5.c.o.d' -o 'src/zim@sha/md5.c.o' -c ../../SOURCE/libzim/src/md5.c
[6/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/dirent.cpp.o' -MF 'src/zim@sha/dirent.cpp.o.d' -o 'src/zim@sha/dirent.cpp.o' -c ../../SOURCE/libzim/src/dirent.cpp
[7/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/cluster.cpp.o' -MF 'src/zim@sha/cluster.cpp.o.d' -o 'src/zim@sha/cluster.cpp.o' -c ../../SOURCE/libzim/src/cluster.cpp
[8/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/article.cpp.o' -MF 'src/zim@sha/article.cpp.o.d' -o 'src/zim@sha/article.cpp.o' -c ../../SOURCE/libzim/src/article.cpp
[9/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/file_compound.cpp.o' -MF 'src/zim@sha/file_compound.cpp.o.d' -o 'src/zim@sha/file_compound.cpp.o' -c ../../SOURCE/libzim/src/file_compound.cpp
[10/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/file_reader.cpp.o' -MF 'src/zim@sha/file_reader.cpp.o.d' -o 'src/zim@sha/file_reader.cpp.o' -c ../../SOURCE/libzim/src/file_reader.cpp
FAILED: src/zim@sha/file_reader.cpp.o
c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/file_reader.cpp.o' -MF 'src/zim@sha/file_reader.cpp.o.d' -o 'src/zim@sha/file_reader.cpp.o' -c ../../SOURCE/libzim/src/file_reader.cpp
../../SOURCE/libzim/src/file_reader.cpp:128:12: error: moving a local object in a return statement prevents copy elision [-Werror,-Wpessimizing-move]
    return std::move(buffer);
           ^
../../SOURCE/libzim/src/file_reader.cpp:128:12: note: remove std::move call here
    return std::move(buffer);
           ^~~~~~~~~~      ~
../../SOURCE/libzim/src/file_reader.cpp:138:12: error: moving a local object in a return statement prevents copy elision [-Werror,-Wpessimizing-move]
    return std::move(ret_buffer);
           ^
../../SOURCE/libzim/src/file_reader.cpp:138:12: note: remove std::move call here
    return std::move(ret_buffer);
           ^~~~~~~~~~          ~
2 errors generated.
[11/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/file.cpp.o' -MF 'src/zim@sha/file.cpp.o.d' -o 'src/zim@sha/file.cpp.o' -c ../../SOURCE/libzim/src/file.cpp
[12/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/blob.cpp.o' -MF 'src/zim@sha/blob.cpp.o.d' -o 'src/zim@sha/blob.cpp.o' -c ../../SOURCE/libzim/src/blob.cpp
[13/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/template.cpp.o' -MF 'src/zim@sha/template.cpp.o.d' -o 'src/zim@sha/template.cpp.o' -c ../../SOURCE/libzim/src/template.cpp
[14/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/md5stream.cpp.o' -MF 'src/zim@sha/md5stream.cpp.o.d' -o 'src/zim@sha/md5stream.cpp.o' -c ../../SOURCE/libzim/src/md5stream.cpp
[15/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/buffer.cpp.o' -MF 'src/zim@sha/buffer.cpp.o.d' -o 'src/zim@sha/buffer.cpp.o' -c ../../SOURCE/libzim/src/buffer.cpp
[16/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/uuid.cpp.o' -MF 'src/zim@sha/uuid.cpp.o.d' -o 'src/zim@sha/uuid.cpp.o' -c ../../SOURCE/libzim/src/uuid.cpp
[17/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/search_iterator.cpp.o' -MF 'src/zim@sha/search_iterator.cpp.o.d' -o 'src/zim@sha/search_iterator.cpp.o' -c ../../SOURCE/libzim/src/search_iterator.cpp
FAILED: src/zim@sha/search_iterator.cpp.o
c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/search_iterator.cpp.o' -MF 'src/zim@sha/search_iterator.cpp.o.d' -o 'src/zim@sha/search_iterator.cpp.o' -c ../../SOURCE/libzim/src/search_iterator.cpp
In file included from ../../SOURCE/libzim/src/search_iterator.cpp:20:
../../SOURCE/libzim/src/xapian/myhtmlparse.h:45:7: error: 'zim::MyHtmlParser::parse_html' hides overloaded virtual function [-Werror,-Woverloaded-virtual]
        void parse_html(const string &text, const string &charset_,
             ^
../../SOURCE/libzim/src/xapian/htmlparse.h:46:15: note: hidden overloaded virtual function 'zim::HtmlParser::parse_html' declared here: different number of parameters (1 vs 3)
        virtual void parse_html(const string &text);
                     ^
In file included from ../../SOURCE/libzim/src/search_iterator.cpp:24:
../../SOURCE/libzim/src/search_internal.h:31:1: error: 'InternalData' defined as a struct here but previously declared as a class [-Werror,-Wmismatched-tags]
struct Search::InternalData {
^
../../SOURCE/libzim/include/zim/search.h:58:10: note: did you mean struct here?
         class InternalData;
         ^~~~~
         struct
In file included from ../../SOURCE/libzim/src/search_iterator.cpp:24:
../../SOURCE/libzim/src/search_internal.h:39:1: error: 'InternalData' defined as a struct here but previously declared as a class [-Werror,-Wmismatched-tags]
struct search_iterator::InternalData {
^
../../SOURCE/libzim/include/zim/search_iterator.h:60:9: note: did you mean struct here?
        class InternalData;
        ^~~~~
        struct
3 errors generated.
[18/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/fileimpl.cpp.o' -MF 'src/zim@sha/fileimpl.cpp.o.d' -o 'src/zim@sha/fileimpl.cpp.o' -c ../../SOURCE/libzim/src/fileimpl.cpp
[19/45] c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/search.cpp.o' -MF 'src/zim@sha/search.cpp.o.d' -o 'src/zim@sha/search.cpp.o' -c ../../SOURCE/libzim/src/search.cpp
FAILED: src/zim@sha/search.cpp.o
c++  -Isrc/zim@sha -Isrc -I../../SOURCE/libzim/src -Iinclude -I../../SOURCE/libzim/include -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -Werror -Wall -pthread -MMD -MQ 'src/zim@sha/search.cpp.o' -MF 'src/zim@sha/search.cpp.o.d' -o 'src/zim@sha/search.cpp.o' -c ../../SOURCE/libzim/src/search.cpp
In file included from ../../SOURCE/libzim/src/search.cpp:22:
../../SOURCE/libzim/src/search_internal.h:31:1: error: 'InternalData' defined as a struct here but previously declared as a class [-Werror,-Wmismatched-tags]
struct Search::InternalData {
^
../../SOURCE/libzim/include/zim/search.h:58:10: note: did you mean struct here?
         class InternalData;
         ^~~~~
         struct
In file included from ../../SOURCE/libzim/src/search.cpp:22:
../../SOURCE/libzim/src/search_internal.h:39:1: error: 'InternalData' defined as a struct here but previously declared as a class [-Werror,-Wmismatched-tags]
struct search_iterator::InternalData {
^
../../SOURCE/libzim/include/zim/search_iterator.h:60:9: note: did you mean struct here?
        class InternalData;
        ^~~~~
        struct
2 errors generated.
ninja: build stopped: subcommand failed.
run command 'ninja -v'
current directory is '/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/libzim'
env is :
  TERM_PROGRAM : 'iTerm.app'
  TERM : 'xterm-256color'
  SHELL : '/bin/bash'
  TMPDIR : '/var/folders/w0/ph55y7r54tld0qb0kyybvbs40000gn/T/'
  Apple_PubSub_Socket_Render : '/private/tmp/com.apple.launchd.nrCFYuutOL/Render'
  TERM_PROGRAM_VERSION : '3.1.5'
  OLDPWD : '/Users/chrisli/Developer/kiwix-build/ninja'
  TERM_SESSION_ID : 'w0t0p0:5C85ED98-7F48-42D8-8599-4475C4AAE225'
  USER : 'chrisli'
  SSH_AUTH_SOCK : '/private/tmp/com.apple.launchd.XjJNJcWRrR/Listeners'
  __CF_USER_TEXT_ENCODING : '0x1F5:0x0:0x0'
  VIRTUAL_ENV : '/Users/chrisli/Developer/kiwix-build'
  PATH : '/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/bin:/Users/chrisli/Developer/kiwix-build/bin:/Users/chrisli/.cargo/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin'
  PWD : '/Users/chrisli/Developer/kiwix-build'
  LANG : 'en_US.UTF-8'
  ITERM_PROFILE : 'Default'
  XPC_FLAGS : '0x0'
  PS1 : '(kiwix-build) \\h:\\W \\u\\$ '
  XPC_SERVICE_NAME : '0'
  SHLVL : '1'
  HOME : '/Users/chrisli'
  COLORFGBG : '15;0'
  ITERM_SESSION_ID : 'w0t0p0:5C85ED98-7F48-42D8-8599-4475C4AAE225'
  LOGNAME : 'chrisli'
  COLORTERM : 'truecolor'
  _ : './kiwix-build.py'
  PKG_CONFIG_PATH : ':/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib/pkgconfig'
  LD_LIBRARY_PATH : ':/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib:/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib'
  CPPFLAGS : '-I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include '
  LDFLAGS : '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib '

Stopping build due to errors

Something's awry with downloading xapian-core in kiwix-build

This may just be me / my environment however I've had problems caused by what seem to be issues either downloading or extracting the xapian-core dependency as part of kiwix-build.py

Here's the error trace when trying to build the arm branch. It also occurs on the same device (a Pi 3) when trying to build from the master branch

I'll try to run the build in another linux environment to triangulate the scope of the problem.

(kiwix)pi@rachel ~/kiwix-dev/kiwix-build $ pyenv exec python kiwix-build.py --target-platform raspberrypi_dyn
[INSTALL PACKAGES]
 - cmake : SKIP
 - automake : SKIP
 - ccache : SKIP
SKIP, No package to install.
[PREPARE]
prepare sources for toolchain armhf :
  gitclone armhf : OK
  gitupdate armhf : OK
prepare sources pugixml :
  download pugixml : SKIP
  extract pugixml : OK
  patch pugixml : OK
prepare sources zlib :
  download zlib : OK
  extract zlib : OK
  patch zlib : OK
prepare sources lzma :
  download lzma : SKIP
  extract lzma : OK
prepare sources uuid :
  download uuid : OK
  extract uuid : OK
prepare sources xapian-core :
  download xapian-core : SKIP
  extract xapian-core : ERROR
Traceback (most recent call last):
  File "/home/pi/kiwix-dev/kiwix-build/utils.py", line 75, in extract_archive
    archive = tarfile.open(archive_path)
  File "/home/pi/.pyenv/versions/3.4.3/lib/python3.4/tarfile.py", line 1553, in open
    raise ReadError("file could not be opened successfully")
tarfile.ReadError: file could not be opened successfully

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "kiwix-build.py", line 862, in <module>
    builder.run()
  File "kiwix-build.py", line 831, in run
    self.prepare_sources()
  File "kiwix-build.py", line 812, in prepare_sources
    source.prepare()
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 117, in prepare
    self.command('extract', self._extract)
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 86, in command
    return self.target.command(*args, **kwargs)
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 50, in command
    ret = function(*args, context=context)
  File "/home/pi/kiwix-dev/kiwix-build/dependency_utils.py", line 106, in _extract
    name=self.source_dir)
  File "/home/pi/kiwix-dev/kiwix-build/utils.py", line 115, in extract_archive
    if archive is not None:
UnboundLocalError: local variable 'archive' referenced before assignment

Add Linux/ARM to CI build targets

Please include support for arm, especially rpi in builds

Desktop app crashes

Reno (at hackathon) was showing me that the Kiwix desktop app crashes a lot on OSX 10.9 (I think). It's on ZIM file open sometimes, and when it happens he can't get it to work again until he clears cache.

Compile (and publish as nightly) zim-tools

Error building native_dyn on macOS 10.13.2

output

(venv) Chriss-MBP:kiwix-build chrisli$ ./kiwix-build.py --target-platform native_dyn
WARNING: kiwix-build has not been tested on MacOS platfrom.
Tests, bug reports and patches are welcomed.
[INSTALL PACKAGES]
 - autoconf : SKIP
 - automake : SKIP
 - libtool : SKIP
 - cmake : SKIP
 - pkg-config : SKIP
SKIP, No package to install.
[PREPARE]
prepare sources pugixml :
  download pugixml : OK1.50%
  extract pugixml : OK
  patch pugixml : OK
prepare sources zlib :
  download zlib : OK0.41%
  extract zlib : OK
  patch zlib : OK
prepare sources lzma :
  download lzma : OK0.02%
  extract lzma : OK
prepare sources xapian-core :
  download xapian-core : OK0.26%
  extract xapian-core : OK
prepare sources icu4c :
  svncheckout icu4c : OK
  svnupdate icu4c : OK
  patch icu4c : OK
prepare sources libzim :
  gitclone libzim : OK
  gitupdate libzim : OK
prepare sources kiwix-lib :
  gitclone kiwix-lib : OK
  gitupdate kiwix-lib : OK
prepare sources libmicrohttpd :
  download libmicrohttpd : OK0.31%
  extract libmicrohttpd : OK
prepare sources kiwix-tools :
  gitclone kiwix-tools : OK
  gitupdate kiwix-tools : OK
[BUILD]
build pugixml :
  configure pugixml : OK
  compile pugixml : OK
  test pugixml : OK
  install pugixml : OK
build zlib :
  pre_build_script zlib : OK
  configure zlib : OK
  compile zlib : OK
  install zlib : OK
build lzma :
  configure lzma : OK
  compile lzma : OK
  install lzma : OK
build xapian-core :
  configure xapian-core : OK
  compile xapian-core : OK
  install xapian-core : OK
build icu4c :
  configure icu4c : OK
  compile icu4c : OK
  install icu4c : OK
build libzim :
  configure libzim : OK
  compile libzim : OK
  test libzim : OK
  install libzim : OK
build kiwix-lib :
  configure kiwix-lib : OK
  compile kiwix-lib : OK
  test kiwix-lib : OK
  install kiwix-lib : OK
build libmicrohttpd :
  configure libmicrohttpd : OK
  compile libmicrohttpd : OK
  install libmicrohttpd : OK
build kiwix-tools :
  configure kiwix-tools : OK
  compile kiwix-tools : ERROR
[1/13] c++  -Isrc/reader/kiwix-read@exe -Isrc/reader -I../../SOURCE/kiwix-tools/src/reader -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/reader/kiwix-read@exe/kiwix-read.cpp.o' -MF 'src/reader/kiwix-read@exe/kiwix-read.cpp.o.d' -o 'src/reader/kiwix-read@exe/kiwix-read.cpp.o' -c ../../SOURCE/kiwix-tools/src/reader/kiwix-read.cpp
../../SOURCE/kiwix-tools/src/reader/kiwix-read.cpp:85:18: warning: unused variable 'contentLength' [-Wunused-variable]
    unsigned int contentLength = 0;
                 ^
1 warning generated.
[2/13] c++  -Isrc/searcher/kiwix-search@exe -Isrc/searcher -I../../SOURCE/kiwix-tools/src/searcher -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/searcher/kiwix-search@exe/kiwix-search.cpp.o' -MF 'src/searcher/kiwix-search@exe/kiwix-search.cpp.o.d' -o 'src/searcher/kiwix-search@exe/kiwix-search.cpp.o' -c ../../SOURCE/kiwix-tools/src/searcher/kiwix-search.cpp
[3/13] c++  -o src/reader/kiwix-read 'src/reader/kiwix-read@exe/kiwix-read.cpp.o' -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -pthread -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64 -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lkiwix -lzim -llzma -lz -licui18n -licuuc -licudata -lpugixml -lxapian -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lmicrohttpd -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lz
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: warning: directory not found for option '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64'
[4/13] /Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/bin/kiwix-compile-resources --cxxfile static/server/server-resources.cpp --hfile static/server/server-resources.h ../../SOURCE/kiwix-tools/static/server/resources_list.txt
jquery-ui/jquery-ui.structure.min.css ['jquery_ui', 'jquery_ui_structure_min_css']
jquery-ui/jquery-ui.min.js ['jquery_ui', 'jquery_ui_min_js']
jquery-ui/external/jquery/jquery.js ['jquery_ui', 'external', 'jquery', 'jquery_js']
jquery-ui/images/ui-bg_flat_0_aaaaaa_40x100.png ['jquery_ui', 'images', 'ui_bg_flat_0_aaaaaa_40x100_png']
jquery-ui/images/ui-bg_flat_75_ffffff_40x100.png ['jquery_ui', 'images', 'ui_bg_flat_75_ffffff_40x100_png']
jquery-ui/images/ui-icons_222222_256x240.png ['jquery_ui', 'images', 'ui_icons_222222_256x240_png']
jquery-ui/images/ui-bg_glass_55_fbf9ee_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_55_fbf9ee_1x400_png']
jquery-ui/images/ui-bg_highlight-soft_75_cccccc_1x100.png ['jquery_ui', 'images', 'ui_bg_highlight_soft_75_cccccc_1x100_png']
jquery-ui/images/ui-bg_glass_65_ffffff_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_65_ffffff_1x400_png']
jquery-ui/images/ui-icons_2e83ff_256x240.png ['jquery_ui', 'images', 'ui_icons_2e83ff_256x240_png']
jquery-ui/images/ui-icons_cd0a0a_256x240.png ['jquery_ui', 'images', 'ui_icons_cd0a0a_256x240_png']
jquery-ui/images/ui-icons_888888_256x240.png ['jquery_ui', 'images', 'ui_icons_888888_256x240_png']
jquery-ui/images/ui-bg_glass_75_e6e6e6_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_75_e6e6e6_1x400_png']
jquery-ui/images/animated-overlay.gif ['jquery_ui', 'images', 'animated_overlay_gif']
jquery-ui/images/ui-bg_glass_75_dadada_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_75_dadada_1x400_png']
jquery-ui/images/ui-icons_454545_256x240.png ['jquery_ui', 'images', 'ui_icons_454545_256x240_png']
jquery-ui/images/ui-bg_glass_95_fef1ec_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_95_fef1ec_1x400_png']
jquery-ui/jquery-ui.theme.min.css ['jquery_ui', 'jquery_ui_theme_min_css']
jquery-ui/jquery-ui.min.css ['jquery_ui', 'jquery_ui_min_css']
home.html.tmpl ['home_html_tmpl']
include.html.part ['include_html_part']
taskbar.css ['taskbar_css']
taskbar.html.part ['taskbar_html_part']
global_taskbar.html.part ['global_taskbar_html_part']
[5/13] c++  -o src/searcher/kiwix-search 'src/searcher/kiwix-search@exe/kiwix-search.cpp.o' -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -pthread -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64 -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lkiwix -lzim -llzma -lz -licui18n -licuuc -licudata -lpugixml -lxapian -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lmicrohttpd -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lz
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: warning: directory not found for option '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64'
[6/13] c++  -Isrc/installer/kiwix-install@exe -Isrc/installer -I../../SOURCE/kiwix-tools/src/installer -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/installer/kiwix-install@exe/kiwix-install.cpp.o' -MF 'src/installer/kiwix-install@exe/kiwix-install.cpp.o.d' -o 'src/installer/kiwix-install@exe/kiwix-install.cpp.o' -c ../../SOURCE/kiwix-tools/src/installer/kiwix-install.cpp
[7/13] c++  -Isrc/manager/kiwix-manage@exe -Isrc/manager -I../../SOURCE/kiwix-tools/src/manager -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/manager/kiwix-manage@exe/kiwix-manage.cpp.o' -MF 'src/manager/kiwix-manage@exe/kiwix-manage.cpp.o.d' -o 'src/manager/kiwix-manage@exe/kiwix-manage.cpp.o' -c ../../SOURCE/kiwix-tools/src/manager/kiwix-manage.cpp
[8/13] c++  -o src/installer/kiwix-install 'src/installer/kiwix-install@exe/kiwix-install.cpp.o' -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -pthread -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64 -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lkiwix -lzim -llzma -lz -licui18n -licuuc -licudata -lpugixml -lxapian -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lmicrohttpd -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lz
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: warning: directory not found for option '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64'
[9/13] c++  -o src/manager/kiwix-manage 'src/manager/kiwix-manage@exe/kiwix-manage.cpp.o' -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -pthread -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64 -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lkiwix -lzim -llzma -lz -licui18n -licuuc -licudata -lpugixml -lxapian -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lmicrohttpd -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -lz
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: warning: directory not found for option '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib64'
[10/13] c++  -Isrc/server/kiwix-serve@exe -Isrc/server -I../../SOURCE/kiwix-tools/src/server -Istatic/server -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/server/kiwix-serve@exe/request_context.cpp.o' -MF 'src/server/kiwix-serve@exe/request_context.cpp.o.d' -o 'src/server/kiwix-serve@exe/request_context.cpp.o' -c ../../SOURCE/kiwix-tools/src/server/request_context.cpp
[11/13] c++  -Isrc/server/kiwix-serve@exe -Isrc/server -I../../SOURCE/kiwix-tools/src/server -Istatic/server -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o' -MF 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o.d' -o 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o' -c ../../SOURCE/kiwix-tools/src/server/kiwix-serve.cpp
FAILED: src/server/kiwix-serve@exe/kiwix-serve.cpp.o
c++  -Isrc/server/kiwix-serve@exe -Isrc/server -I../../SOURCE/kiwix-tools/src/server -Istatic/server -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o' -MF 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o.d' -o 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o' -c ../../SOURCE/kiwix-tools/src/server/kiwix-serve.cpp
../../SOURCE/kiwix-tools/src/server/kiwix-serve.cpp:111:11: warning: comparison of constant 18446744073709551615 with expression of type 'unsigned int' is always true [-Wtautological-constant-out-of-range-compare]
  if (pos != std::string::npos) {
      ~~~ ^  ~~~~~~~~~~~~~~~~~
../../SOURCE/kiwix-tools/src/server/kiwix-serve.cpp:221:35: warning: 'MHD_create_response_from_data' is deprecated: MHD_create_response_from_data() is deprecated, use MHD_create_response_from_buffer() [-Wdeprecated-declarations]
  struct MHD_Response* response = MHD_create_response_from_data(
                                  ^
/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/microhttpd.h:2032:1: note: 'MHD_create_response_from_data' has been explicitly marked deprecated here
_MHD_DEPR_FUNC("MHD_create_response_from_data() is deprecated, use MHD_create_response_from_buffer()") \
^
/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/microhttpd.h:244:44: note: expanded from macro '_MHD_DEPR_FUNC'
#define _MHD_DEPR_FUNC(msg) __attribute__((deprecated(msg)))
                                           ^
../../SOURCE/kiwix-tools/src/server/kiwix-serve.cpp:547:27: error: no member named 'getHtml' in 'kiwix::Searcher'
      content = searcher->getHtml();
                ~~~~~~~~  ^
2 warnings and 1 error generated.
[12/13] c++  -Isrc/server/kiwix-serve@exe -Isrc/server -I../../SOURCE/kiwix-tools/src/server -Istatic/server -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include/ -I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include -Xclang -fcolor-diagnostics -pipe -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -pthread -MMD -MQ 'src/server/kiwix-serve@exe/meson-generated_.._.._.._static_server_server-resources.cpp.o' -MF 'src/server/kiwix-serve@exe/meson-generated_.._.._.._static_server_server-resources.cpp.o.d' -o 'src/server/kiwix-serve@exe/meson-generated_.._.._.._static_server_server-resources.cpp.o' -c static/server/server-resources.cpp
ninja: build stopped: subcommand failed.
run command 'ninja -v'
current directory is '/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/kiwix-tools'
env is :
  TERM_PROGRAM : 'iTerm.app'
  TERM : 'xterm-256color'
  SHELL : '/bin/bash'
  TMPDIR : '/var/folders/w0/ph55y7r54tld0qb0kyybvbs40000gn/T/'
  Apple_PubSub_Socket_Render : '/private/tmp/com.apple.launchd.gRTBqiVMvv/Render'
  TERM_PROGRAM_VERSION : '3.1.5'
  OLDPWD : '/Users/chrisli/Developer/kiwix-build/ninja'
  TERM_SESSION_ID : 'w0t2p0:CF89E50E-4DF2-4263-94D4-A20B4022D45F'
  USER : 'chrisli'
  COMMAND_MODE : 'unix2003'
  SSH_AUTH_SOCK : '/private/tmp/com.apple.launchd.4Ly3PHQaTC/Listeners'
  __CF_USER_TEXT_ENCODING : '0x1F5:0x0:0x0'
  VIRTUAL_ENV : '/Users/chrisli/Developer/kiwix-build/venv'
  PATH : '/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/bin:/Users/chrisli/Developer/kiwix-build/venv/bin:/Users/chrisli/.cargo/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin'
  PWD : '/Users/chrisli/Developer/kiwix-build'
  LANG : 'en_US.UTF-8'
  ITERM_PROFILE : 'Default'
  XPC_FLAGS : '0x0'
  PS1 : '(venv) \\h:\\W \\u\\$ '
  XPC_SERVICE_NAME : '0'
  SHLVL : '1'
  HOME : '/Users/chrisli'
  COLORFGBG : '0;15'
  ITERM_SESSION_ID : 'w0t2p0:CF89E50E-4DF2-4263-94D4-A20B4022D45F'
  LOGNAME : 'chrisli'
  SECURITYSESSIONID : '186a6'
  COLORTERM : 'truecolor'
  _ : './kiwix-build.py'
  PKG_CONFIG_PATH : ':/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib/pkgconfig'
  LD_LIBRARY_PATH : ':/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib:/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib'
  CPPFLAGS : '-I/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/include '
  LDFLAGS : '-L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib -L/Users/chrisli/Developer/kiwix-build/BUILD_native_dyn/INSTALL/lib '

Stopping build due to errors

Please provide a way to clean from a failed, partial run of kiwix-build.py

I remember in Lyon that I had to delete a folder during the early stages of trying to get kiwis-build.py to run successfully to completion. I've run into what seems a similar problem when trying to run kiwis-build.py on a Raspberry Pi (with Python 3.4.3 provided using pyenv)

I've manually deleted various folders, including ARCHIVE, SOURCE and BUILD_native_dyn I've tried to see if there's a --clean option in the script (there doesn't seem to be one), and am now a bit stuck trying to make sense of what's causing the script to first SKIP and then ERROR when trying to obtain xapian-core. This could well be a local issue on my machine (a Raspberry Pi 3 running the latest patches of Raspbian) or perhaps the script isn't yet ready to run on a Pi / build for Pi's.

Anyway, back to the reason for this issue - I'd like a known, reliable way to reset the environment
and also (if practical) some better logging to help me uncover what's been going wrong.

Make official/stable releases with travis

based on git tags

Android nightlies are broken

They are not published anymore

How should I implement iOS sdk Toolchain if no download is necessary

As the discussion between @kelson42 and me, it might be a good idea to simultaneously start cross-platform build development for iOS. So in branch apple_platforms, I have added iOS SDK class

class iOS_sdk(Toolchain):
    pass

But different from android, kiwix-build do not need to download the SDK, since it is already part of Xcode.app. I just need to run xcodebuild -version -sdk iphonesimulator | grep -E '^Path' | sed 's/Path: //' to get the SDK path. How should I implement this class?

Improve kiwix-build README

README should clearly explain the compilation procedure. This means step-by-step procedures which can be follow blindly with short explanation about what is going on:

What are the dependencies (with minimal version)
How to compile kiwix-lib and kiwix-tools under Linux static/dynamic by downloading locally deps
How to cross-compile kiwix-lib and kiwix-tools under Linux for Windows static/dynamic by downloading locally deps

No module named 'PIL' when building apk

on ubuntu 16.4 VM

install dependencies with following script

#!/usr/bin/env bash
# Python3, pip3, ninja
apt-get install python3-pip virtualenv && virtualenv -p python3 ./ && source ./bin/activate && pip3 install meson && hash -r && git clone git://github.com/ninja-build/ninja.git && cd ninja && git checkout release && ./configure.py --bootstrap && mkdir -p ../bin && cp ninja ../bin && cd .. && source ./bin/activate

clone

https://github.com/nataren/kiwix-build
https://github.com/nataren/kiwix-android-custom

(differ only in that embed_zim is true)

run

python ./kiwix-build.py --verbose --target-platform android_arm --android-custom-app wikimedar kiwix-android-custom --zim-file-url https://download.kiwix.org/zim/wikipedia/wikipedia_ar_medicine_2017-08.zim

result:

Installing /opt/cesar-build/kiwix-build/BUILD_android_arm/kiwix-lib/kiwix.pc to /opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib/x86_64-linux-gnu/pkgconfig OK build kiwix-android : SKIP build kiwix-android-custom : configure kiwix-android-custom : run command './gen-custom-android-directory.py wikimedar --output-dir /opt/cesar-build/kiwix-build/BUILD_android_arm/kiwix-android-custom_wikimedar/custom/wikimedar' current directory is '/opt/cesar-build/kiwix-build/SOURCE/kiwix-android-custom' env is : CXX : '/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/bin/arm-linux-androideabi-g++' PS1 : '(cesar-build) \\[\\e]0;\\u@\\h: \\w\\a\\]${debian_chroot:+($debian_chroot)}\\u@\\h:\\w\\$ ' CXXFLAGS : '-fPIC -D_LARGEFILE64_SOURCE=1 -D_FILE_OFFSET_BITS=64 --sysroot=/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/sysroot ' PATH : '/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/bin:/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/bin:/usr/lib/ccache:/opt/cesar-build/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin' LESSCLOSE : '/usr/bin/lesspipe %s %s' PWD : '/opt/cesar-build/kiwix-build' SSH_TTY : '/dev/pts/0' HOME : '/root' PKG_CONFIG_PATH : ':/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib/x86_64-linux-gnu/pkgconfig' SSH_CONNECTION : '192.168.0.56 60203 10.100.0.9 22' MAIL : '/var/mail/root' SHELL : '/bin/bash' SSH_CLIENT : '192.168.0.56 60203 22' LD_LIBRARY_PATH : ':/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib:/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib64' SHLVL : '1' TERM : 'xterm' OLDPWD : '/opt/cesar-build' LOGNAME : 'root' CC : '/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/bin/arm-linux-androideabi-gcc' XDG_SESSION_ID : '49' CFLAGS : '-fPIC -D_LARGEFILE64_SOURCE=1 -D_FILE_OFFSET_BITS=64 --sysroot=/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/sysroot ' USER : 'root' LESSOPEN : '| /usr/bin/lesspipe %s' LDFLAGS : '-L/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib -L/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/lib64 --sysroot=/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/sysroot ' VIRTUAL_ENV : '/opt/cesar-build' NDK_DEBUG : '0' XDG_RUNTIME_DIR : '/run/user/0' LS_COLORS : 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:' LANG : 'en_US.UTF-8' _ : '/opt/cesar-build/bin/python' ANDROID_HOME : '/opt/cesar-build/kiwix-build/TOOLCHAINS/android-sdk-r25.2.3' PKG_CONFIG_LIBDIR : '/opt/cesar-build/kiwix-build/BUILD_android_arm/android-ndk-r13b-arm-14/sysroot/lib/pkgconfig' CPPFLAGS : '-I/opt/cesar-build/kiwix-build/BUILD_android_arm/INSTALL/include ' Traceback (most recent call last): File "./gen-custom-android-directory.py", line 28, in <module> from PIL import Image ImportError: No module named 'PIL' ERROR Stopping build due to errors

build custom apk - KeyError: 'VERSION_CODE'

Continue from #79 with pip3 install pillow

./kiwix-build.py --verbose --target-platform android_arm --android-custom-app wikimedar kiwix-android-custom --zim-file-url https://download.kiwix.org/zim/wikipedia/wikipedia_ar_medicine_2017-08.zim

download_zim kiwix-android-custom : Sha256 for wikipedia_ar_medicine_2017-08.zim not set, do no verify download
OK
compile kiwix-android-custom : ERROR
Traceback (most recent call last):
File "./kiwix-build.py", line 990, in
builder.run()
File "./kiwix-build.py", line 928, in run
self.build()
File "./kiwix-build.py", line 918, in build
builder.build()
File "/opt/cesar-build/kiwix-build/dependencies.py", line 447, in build
self.command('compile', self._compile)
File "/opt/cesar-build/kiwix-build/dependency_utils.py", line 206, in command
return self.target.command(*args, **kwargs)
File "/opt/cesar-build/kiwix-build/dependency_utils.py", line 50, in command
ret = function(*args, context=context)
File "/opt/cesar-build/kiwix-build/dependency_utils.py", line 369, in _compile
gradle_option=self.gradle_option)
File "/opt/cesar-build/kiwix-build/dependencies.py", line 423, in gradle_option
version_code=os.environ['VERSION_CODE'],
File "/opt/cesar-build/lib/python3.5/os.py", line 725, in getitem
raise KeyError(key) from None
KeyError: 'VERSION_CODE'

macOS: error encountered while building uuid

Here is the log

uuid_error.txt

When is the optimal time to run travis cron jobs?

Travis has a concurrent job limit of I think 5 tasks per org. Kiwix build can delay pull request buids by around an hour. Might it be a better idea to schedule these builds at less active times?

Nightly Latest folder empty or has unique names

The nightly folder at http://download.kiwix.org/nightly/latest/ is empty a lot of times. When the folder is not empty, the filenames can be unique, causing scripts or docker builds to fail.

If the names in nightly/latest/ are always the same, then we can always point our software towards the latest build. Which is most helpful when we are all submitting bug reports.

desired:
http://download.kiwix.org/nightly/latest/kiwix-tools_linux64.tar.gz

actual:
http://download.kiwix.org/nightly/latest/kiwix-tools_linux64_2017-11-12.tar.gz
or sometimes blank:
http://download.kiwix.org/nightly/latest/

"TypeError: urlopen() got an unexpected keyword argument 'context'" when building on Debian Jessie

Workaround : edit kiwix-build.py to remove the "context" argument when urlopen() is called (line 418).

It seems to come from the version of python3 distributed with debian jessie (3.4.2-2)

Would it be a good idea to split kiwix-build.py into multiple files?

Currently kiwix-build.py is too huge, I find it hard to navigate. And I cannot put work on different classes in different commits. (i.e. if I added something to both iOS_sdk and iOSTargetInfo, I have to commit them together)

I think it would be a better idea to separate them into files like TargetInfo.py, Toolchain.py, etc.