Use Keras models in C++ with ease

• Introduction
• Usage
• Performance
• Requirements and Installation
• Internals

Would you like to build/train a model using Keras/Python? And would you like run the prediction (forward pass) on your model in C++ without linking your application against TensorFlow? Then frugally-deep is exactly for you.

frugally-deep

• is a small header-only library written in modern and pure C++.
• is very easy to integrate and use.
• depends only on FunctionalPlus, Eigen and json - also header-only libraries.
• supports inference (model.predict) not only for sequential models but also for computational graphs with a more complex topology, created with the functional API.
• re-implements a (small) subset of TensorFlow, i.e. the operations needed to support prediction.
• results in a much smaller binary size than linking against TensorFlow.
• works out of-the-box also when compiled into a 32-bit executable.
• utterly ignores even the most powerful GPU in your system and uses only one CPU core. ;-)
• but is quite fast on one CPU core compared to TensorFlow.

Layer types typically used in image recognition/generation are supported, making many popular model architectures possible (see Performance section).

• Add, Concatenate
• AveragePooling1D/2D, GlobalAveragePooling1D/2D
• Conv1D/2D, SeparableConv2D
• Cropping1D/2D, ZeroPadding1D/2D
• BatchNormalization, Dense, Dropout, Flatten
• MaxPooling1D/2D, GlobalMaxPooling1D/2D
• ELU, LeakyReLU, ReLU, SeLU
• Sigmoid, Softmax, Softplus, Tanh
• UpSampling1D/2D

• multiple inputs and outputs
• nested models
• residual connections
• shared layers
• arbitrary complex model architectures / computational graphs

Currently not supported are the following layer types: ActivityRegularization, AlphaDropout, Average, AveragePooling3D, Bidirectional, Conv2DTranspose, Conv3D, ConvLSTM2D, CuDNNGRU, CuDNNLSTM, Cropping3D, DepthwiseConv2D, Dot, Embedding, GaussianDropout, GaussianNoise, GRU, GRUCell, Lambda, LocallyConnected1D, LocallyConnected2D, LSTM, LSTMCell, Masking, Maximum, MaxPooling3D, Multiply, Permute, PReLU, RepeatVector, Reshape, RNN, SimpleRNN, SimpleRNNCell, StackedRNNCells, Subtract, ThresholdedReLU, TimeDistributed, Upsampling3D, any custom layers

1. Use Keras/Python to build (model.compile(...)), train (model.fit(...)) and test (model.evaluate(...)) your model as usual. Then save it to a single HDF5 file using model.save('....h5', include_optimizer=False). The image_data_format in your model must be channels_last, which is the default when using the TensorFlow backend. Models created with a different image_data_format and other backends are not supported.

2. Now convert it to the frugally-deep file format with keras_export/convert_model.py

3. Finally load it in C++ (fdeep::load_model(...)) and use model.predict(...) to invoke a forward pass with your data.

The following minimal example shows the full workflow:

# create_model.py
import numpy as np
from tensorflow.python.keras.layers import Input, Dense
from tensorflow.python.keras.models import Model

inputs = Input(shape=(4,))
x = Dense(5, activation='relu')(inputs)
predictions = Dense(3, activation='softmax')(x)
model = Model(inputs=inputs, outputs=predictions)
model.compile(loss='categorical_crossentropy', optimizer='nadam')

model.fit(
    np.asarray([[1,2,3,4], [2,3,4,5]]),
    np.asarray([[1,0,0], [0,0,1]]), epochs=10)

model.save('keras_model.h5', include_optimizer=False)

python3 keras_export/convert_model.py keras_model.h5 fdeep_model.json

// main.cpp
#include <fdeep/fdeep.hpp>
int main()
{
    const auto model = fdeep::load_model("fdeep_model.json");
    const auto result = model.predict(
        {fdeep::tensor3(fdeep::shape3(4, 1, 1), {1, 2, 3, 4})});
    std::cout << fdeep::show_tensor3s(result) << std::endl;
}

When using convert_model.py a test case (input and corresponding output values) is generated automatically and saved along with your model. fdeep::load_model runs this test to make sure the results of a forward pass in frugally-deep are the same as in Keras.

In order to convert images to fdeep::tensor3 the convenience function tensor3_from_bytes is provided. In case you want to convert an Eigen::Matrix to fdeep::tensor3, have a look at the following two examples: copy values, reuse memory

Below you can find the durations of one isolated forward pass for some popular models ran on a single core of an Intel Core i5-6600 CPU @ 3.30GHz. frugally-deep was compiled (GCC ver. 5.4.0) with g++ -O3 -mavx (same as TensorFlow 1.6.0 binaries). The processes were started with CUDA_VISIBLE_DEVICES='' taskset --cpu-list 1 ... to disable the GPU and to only allow usage of one CPU.

| Model             | Keras + TensorFlow | frugally-deep |
|-------------------|--------------------|---------------|
| InceptionV3       |             1.04 s |        0.36 s |
| ResNet50          |             0.82 s |        0.22 s |
| VGG16             |             0.68 s |        0.79 s |
| VGG19             |             0.82 s |        0.94 s |
| Xception          |             1.79 s |        0.54 s |
| DenseNet201       |             2.39 s |        0.34 s |
| NASNetLarge       |             4.98 s |        2.23 s |

A C++14-compatible compiler is needed. Compilers from these versions on are fine: GCC 4.9, Clang 3.7 (libc++ 3.7) and Visual C++ 2015.

You can install frugally-deep using cmake as shown below, or (if you prefer) download the code (and the code of FunctionalPlus), extract it and tell your compiler to use the include directories.

git clone https://github.com/Dobiasd/FunctionalPlus
cd FunctionalPlus
mkdir -p build && cd build
cmake ..
make && sudo make install
cd ../..

sudo apt install mercurial
hg clone https://bitbucket.org/eigen/eigen/
cd eigen
mkdir -p build && cd build
cmake ..
make && sudo make install
sudo ln -s /usr/local/include/eigen3/Eigen /usr/local/include/Eigen
cd ../..

git clone https://github.com/nlohmann/json
cd json
mkdir -p build && cd build
cmake ..
make && sudo make install
cd ../..

git clone https://github.com/Dobiasd/frugally-deep
cd frugally-deep
mkdir -p build && cd build
cmake ..
make && sudo make install
cd ../..

Building the tests (optional) requires doctest. Unit Tests are disabled by default – they are enabled and executed by:

# install doctest
wget https://raw.githubusercontent.com/onqtam/doctest/master/doctest/doctest.h
sudo mv ./doctest.h /usr/local/include/doctest.h

# build unit tests
cmake -DFDEEP_BUILD_UNITTEST=ON ..
make unittest

Just add a conanfile.txt with frugally-deep as a requirement and chose the generator for your project.

[requires]
frugally-deep/v0.2.3-p0@dobiasd/stable

[generators]
cmake

Then install it:

$ conan install conanfile.txt

frugally-deep uses channels_first (depth/channels, height, width) as its image_data_format internally. convert_model.py takes care of all necessary conversions. From then on everything is handled as a float32 tensor with rank 3. Dense layers for example take its input flattened to a shape of (n, 1, 1). This is also the shape you will receive as the output of a final softmax layer for example.

In case you would like to use double instead of float for all calculations, simply do this:

#define FDEEP_FLOAT_TYPE double
#include <fdeep/fdeep.hpp>

A frugally-deep model is thread-safe, i.e. you can call model.predict on the same model instance from different threads simultaneously. This way you may utilize up to as many CPU cores as you have predictions to make. With model::predict_multi there is a convenience function available to handle the parallelism for you.

The API of this library still might change in the future. If you have any suggestions, find errors or want to give general feedback/criticism, I'd love to hear from you. Of course, contributions are also very welcome.

Distributed under the MIT License. (See accompanying file LICENSE or at https://opensource.org/licenses/MIT)