antemons / smart-manuscript Goto Github PK

View Code? Open in Web Editor NEW

24.0 1.0 5.0 15.32 MB

This software recognizes (digitizes) handwritten manuscripts and digitizer-pen input.

License: GNU General Public License v3.0

Python 100.00%

digitizer-pen handwritten-manuscripts smart-pens online-handwriting-recognition machine-learning python tensorflow

smart-manuscript's Introduction

SMART MANUSCRIPT

This software recognizes (digitizes) handwritten manuscripts and digitizer-pen input.

Note, it can only transcribe online handwriting (generated e.g. by smart pens) but no scanned pages.

Getting Started

Setup

(Create Virtual Environment)

virtualenv env -p /usr/bin/python3
source env/bin/activate

Install requirements:
```
pip install -r requirements.txt
```
Install this software
```
pip install .
```
Optional: train a new model (or use the default one)

Usage

Recognize handwritten notes

Recognize a handwritten file (of type either PDF or SVG, generated e.g. by a smart pen) and generate a PDF which is searchable and where text can be copied from.

python -m smartmanuscript input.pdf output.pdf

Test, e.g., as input "smart-manuscript/smartmanuscript/data/sample_text/The_Zen_of_Python.pdf"

Handwritten input

A simple application that transcribes handwritten input (e.g. from a digitizer pen):

python -m smartmanuscript.application

The input will be copied into the clipboard.

Train new model

Download and extract data:

a. You can use the IAMonDo-db-1.0 database to train and validate the model. Place the unzipped folder "IAMonDo-db-1.0" in the folder data.

b. You can use the IBM_UB_1 database to train and validate the model. Place the folder "IBM_UB_1" in the folder data.

c. You may use also your personal handwritten notes, analogously to the one in the directory smart_manuscript/data/sample_text. Place the files in the folder "data/my_handwriting".

Create the preprocessed records:

python -m smartmanuscript.records \
   --path="records" \
   --iam_on_do_path="./data/IAMonDo-db-1.0/" \
   --ibm_ub_path="./data/IBM_UB_1/query" \
   --my_handwriting_train_path="./data/my_handwriting/train/" \
   --my_handwriting_test_path="./data/my_handwriting/test/"

The new folder "records" should contain a "test" and a "train" folder

Train new model:

python -m smartmanuscript.train --path=graph

Example

Transcription:

The zen of Python.
by tim Peters
Beantiful is better than ugly.
Explicit is better than implicit.
Simple is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the mles.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of anbiguity, refuse the temptation to guess.
There should be one - and preferable onty dne
- obvious way to do it.
Although that way may not be obvious at
first unless youire Dutch.
Now is better than never.
Although never is often better than right now.
if the implementation is hard to explain, it's a bad idea.
It the implementation is easy to explain, it may be
a good idea.
Namespaces are one hanking great idea -
let's do more of thosel

Author

Daniel Vorberg

License

This project is licensed under the GPL License - see the LICENSE file for details

Contact

Please send bug reports, patches, and other feedback to: dv(at)pks.mpg.de

smart-manuscript's People

Contributors

Stargazers

Watchers

Forkers

paullerner fakhraddin bfinc naveen1215316111 davidrusu

smart-manuscript's Issues

smartmanuscript.application crash on fresh install develop branch

Hi, application.py still seem to crash when doing some normalisation in preprocessing. Here is my traceback. I have installed the packages on the requirements.txt + tensor2tensor 1.5.3 which is required but not on the file.

  File "/Users/marco/smart-manuscript/smartmanuscript/application.py", line 192, in on_button_release
    self.recognize()
  File "/Users/marco/smart-manuscript/smartmanuscript/application.py", line 238, in recognize
    top_prediction, predictions, probabilities = self.recognizer.recognize_line(strokes, resort=True)
  File "/Users/marco/smart-manuscript/smartmanuscript/reader.py", line 107, in recognize_line
    features = strokes_to_features(ink, resort=resort)
  File "/Users/marco/smart-manuscript/smartmanuscript/writing.py", line 1040, in strokes_to_features
    ink, skew_is_horizontal=skew_is_horizontal)
  File "/Users/marco/smart-manuscript/smartmanuscript/writing.py", line 285, in __call__
    ink, self._normaliztion_steps(skew_is_horizontal))
  File "/Users/marco/smart-manuscript/smartmanuscript/writing.py", line 275, in _apply_normalizations
    ink, new_transformation = normalization(ink)
  File "/Users/marco/smart-manuscript/smartmanuscript/writing.py", line 420, in normalized_slant
    angle = minimize(lambda a: tilted_ink_length(ink, a), 0).x
  File "/Users/marco/.local/share/virtualenvs/pygobject-3YKUhTy8/lib/python3.7/site-packages/scipy/optimize/_minimize.py", line 594, in minimize
    return _minimize_bfgs(fun, x0, args, jac, callback, **options)
  File "/Users/marco/.local/share/virtualenvs/pygobject-3YKUhTy8/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 996, in _minimize_bfgs
    gfk = myfprime(x0)
  File "/Users/marco/.local/share/virtualenvs/pygobject-3YKUhTy8/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 326, in function_wrapper
    return function(*(wrapper_args + args))
  File "/Users/marco/.local/share/virtualenvs/pygobject-3YKUhTy8/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 756, in approx_fprime
    return _approx_fprime_helper(xk, f, epsilon, args=args)
  File "/Users/marco/.local/share/virtualenvs/pygobject-3YKUhTy8/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 690, in _approx_fprime_helper
    f0 = f(*((xk,) + args))
  File "/Users/marco/.local/share/virtualenvs/pygobject-3YKUhTy8/lib/python3.7/site-packages/scipy/optimize/optimize.py", line 326, in function_wrapper
    return function(*(wrapper_args + args))
  File "/Users/marco/smart-manuscript/smartmanuscript/writing.py", line 420, in <lambda>
    angle = minimize(lambda a: tilted_ink_length(ink, a), 0).x
  File "/Users/marco/smart-manuscript/smartmanuscript/writing.py", line 417, in tilted_ink_length
    tilted_ink = Transformation.shear(y_angle=angle) @ ink
  File "/Users/marco/smart-manuscript/smartmanuscript/writing.py", line 198, in __rmatmul__
    new_concatenated_strokes = other @ self.concatenated_strokes
  File "/Users/marco/smart-manuscript/smartmanuscript/utils.py", line 77, in __matmul__
    return other @ self.matrix[:2, :2].transpose() + self.matrix[:2, 2]
TypeError: ufunc 'matmul' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Cannot install gi using pip install -r requirements.txt

Hi, I cannot install the gi package in requirements.txt, any suggestions on how to install that package? I cannot find online.

Many thanks

Read pressure (aka force) and timestamp from the data

Hi,
It's quite unclear what you do with the pressure and timestamp features (see Indermühle et al.).
In writing.py it seems that you extract features like pressure from the data but when using the Corpus or Inkml class I'm only able to get a list of strokes, strokes being a 2-d array of spatial coordinates.
Can you please add some documentation ?

Best regards,

References:
Indermühle, E., Frinken, V., & Bunke, H. (2012, September). Mode detection in online handwritten documents using BLSTM neural networks. In 2012 International Conference on Frontiers in Handwriting Recognition (pp. 302-307). IEEE.