googlefonts / compreffor Goto Github PK

View Code? Open in Web Editor NEW

23.0 19.0 22.0 550 KB

A CFF table subroutinizer for FontTools

License: Apache License 2.0

Python 71.68% Makefile 0.64% C++ 27.28% Cython 0.41%

compreffor's Introduction

A CFF table subroutinizer for FontTools.

compreffor's People

Contributors

Stargazers

Watchers

Forkers

behdad moyogo anthrotype wskplho bihicheng bitforks uikit0 mashabow shea690901 fullstackenviormentss firefoxxy8 hugovk jayvdb manish364824 dthadi3 isabella232 musicinmybrain tjni

compreffor's Issues

How to install?

ufo2ft depends on compreffor now, but there is no setup.py file or any install instructions that I can find.

enable auto deployment on tags to Github Releases

Someone with push access could generate Github authentication tokens and un-comment the deploy section in .travis.yml and .appveyor.yml.
Then, the next time a tag is pushed, wheels will be automatically uploaded to Github Releases page.

--generate-cff option is broken

$ compreffor LibertinusMath-Regular.hint.otf o.otf --generate-cff
Traceback (most recent call last):
  File "/env/bin/compreffor", line 10, in <module>
    sys.exit(main())
  File "/env/lib/python3.7/site-packages/compreffor/__main__.py", line 119, in main
    font['CFF '].cff.compile(f, None)
  File "/env/lib/python3.7/site-packages/fontTools/cffLib/__init__.py", line 105, in compile
    if otFont.recalcBBoxes and not isCFF2:
AttributeError: 'NoneType' object has no attribute 'recalcBBoxes'

Better C interface

The current C interface lacks some import parts, we still need Python to deal with it.

Fails to build with Cython 3

Cython 3.0.0 was recently released, and compreffor is not compatible.

python3 -m build

[...]
running build_ext
building 'compreffor._compreffor' extension
error: unknown file type '.pyx' (from 'src/cython/_compreffor.pyx')

ERROR Backend subprocess exited when trying to invoke build_wheel

I will open a PR to pin cython<3 for now, but it would be better to fix this properly.

Many warnings "Difference found in glyph"

I'm getting many warnings when compressing HelveticaNeueLTStd-UltLt.otf with --check, even though the compressed font looks fine.

$ compreffor -vv --check HelveticaNeueLTStd-UltLt.otf
Compressing font through C++ Compreffor
Took 0.002s to produce data for C++ library
Took 0.009s to run 'lib.compreff()'
Took 0.000s to extract results
Took 0.047s to decompile charstrings
0 substrings unused or negative saving subrs
0 substrings nested too deep
0 substrings being flattened
Took 0.000s to post-process subroutines
Took 0.004s to apply subroutines
Took 0.066s to compress the font
Took 0.150s to compile and save compressed font
Checking compression integrity and call depth
No subroutines found; skip decompress
No subroutines found; skip decompress
WARNING: Difference found in glyph 'plusminus'
WARNING: Difference found in glyph 'aring'
WARNING: Difference found in glyph 'ordfeminine'
WARNING: Difference found in glyph 'less'
WARNING: Difference found in glyph 'ograve'
    [... warnings for all glyphs in this font ...]
WARNING: Fonts have differences :(
Took 0.114s to check compression integrity
Subroutine nesting depth ok! [max nesting depth of 6]
Took 0.055s to check subroutine nesting depth

The version of the font is OTF 1.029;PS 001.003;Core 1.0.33;makeotf.lib1.4.1585.

`callsubr` operator in global subroutines

Currently, compreffor may generate some global subroutines which contain callsubr operator. But according to fonttools/fonttools#856 (comment), it is invalid:

I asked my colleagues and confirmed that it's invalid to use the callsubr operator in global subroutines.

In fact, I found the above fonttools' bug when trying to subset a font subroutinized by compreffor.

testPyCompressor.py fails on Python 3

Some test cases fail when running under Python 3.5.1 on my Mac. They pass when running under Python 2.7.

test_get_lcp
test_get_substrings_all
test_get_suffixes

In the next days, I'll work on adding Travis and Appveyor support to compreffor as well.
For that I would need PR #14 to be merged first.

$ py.test -v compreffor/testPyCompressor.py
============================================================================== test session starts ==============================================================================
platform darwin -- Python 3.5.1, pytest-2.9.2, py-1.4.31, pluggy-0.3.1 -- /usr/local/var/pyenv/versions/3.5.1/envs/fontmake-py35/bin/python3.5
cachedir: .cache
rootdir: /Users/cosimolupo/Documents/Github/compreffor, inifile: 
collected 20 items 

compreffor/testPyCompressor.py::TestCffCompressor::test_candidatesubr_len PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_candidatesubr_value PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_collapse_hintmask_multi_end PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_collapse_hintmask_multi_middle PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_collapse_hintmask_single_middle PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_expand_hintmask_multi_end PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_expand_hintmask_multi_middle PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_expand_hintmask_single_middle PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_get_lcp FAILED
compreffor/testPyCompressor.py::TestCffCompressor::test_get_substrings_all FAILED
compreffor/testPyCompressor.py::TestCffCompressor::test_get_substrings_standard PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_get_suffixes FAILED
compreffor/testPyCompressor.py::TestCffCompressor::test_get_suffixes_random PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_human_size PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_iterative_encode PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_multiple_nested_subr_calls PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_tokenCost PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_update_program_global PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_update_program_local PASSED
compreffor/testPyCompressor.py::TestCffCompressor::test_update_program_multiple PASSED

=================================================================================== FAILURES ====================================================================================
________________________________________________________________________ TestCffCompressor.test_get_lcp _________________________________________________________________________

self = <compreffor.testPyCompressor.TestCffCompressor testMethod=test_get_lcp>

    def test_get_lcp(self):
        """Test the lcp array generation"""

        expected = [0, 6, 5, 0, 5, 4, 0, 4, 3, 0, 3, 2, 0, 2, 1, 0, 1, 0, 0, 0, 0]

>       self.assertEqual(self.sf.get_lcp(), expected)
E       AssertionError: Lists differ: [0, 5, 6, 0, 4, 5, 0, 3, 4, 0, 2, 3, 0, 1, 2, 0, 0, 0, 0, 0, 1] != [0, 6, 5, 0, 5, 4, 0, 4, 3, 0, 3, 2, 0, 2, 1, 0, 1, 0, 0, 0, 0]
E       
E       First differing element 1:
E       5
E       6
E       
E       - [0, 5, 6, 0, 4, 5, 0, 3, 4, 0, 2, 3, 0, 1, 2, 0, 0, 0, 0, 0, 1]
E       + [0, 6, 5, 0, 5, 4, 0, 4, 3, 0, 3, 2, 0, 2, 1, 0, 1, 0, 0, 0, 0]

compreffor/testPyCompressor.py:106: AssertionError
----------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------
Warning: non-TTFont given to Compreffor
___________________________________________________________________ TestCffCompressor.test_get_substrings_all ___________________________________________________________________

self = <compreffor.testPyCompressor.TestCffCompressor testMethod=test_get_substrings_all>

    def test_get_substrings_all(self):
        """Test get_substrings without restrictions"""

        ans = [s.value() for s in self.sf.get_substrings(0, False)]

        expected_values = [(0, 1, 2, 3, 4, 5), (0, 1, 2, 3, 4), (1, 2, 3, 4, 5), (1, 2, 3, 4), \
                            (2, 3, 4, 5), (2, 3, 4), (3, 4, 5), (3, 4), (4, 5), (4,), (5,)]

>       self.assertEqual(ans, expected_values)
E       AssertionError: Lists differ: [(0, 1, 2, 3, 4, 9), (0, 1, 2, 3, 4), (1, 2, 3, 4, 9), (2,[63 chars](4,)] != [(0, 1, 2, 3, 4, 5), (0, 1, 2, 3, 4), (1, 2, 3, 4, 5), (1,[69 chars](5,)]
E       
E       First differing element 0:
E       (0, 1, 2, 3, 4, 9)
E       (0, 1, 2, 3, 4, 5)
E       
E       Second list contains 1 additional elements.
E       First extra element 10:
E       (5,)
E       
E       - [(0, 1, 2, 3, 4, 9),
E       ?                  ^
E       
E       + [(0, 1, 2, 3, 4, 5),
E       ?                  ^
E       
E          (0, 1, 2, 3, 4),
E       -  (1, 2, 3, 4, 9),
E       ?               ^
E       
E       +  (1, 2, 3, 4, 5),
E       ?               ^
E       
E       -  (2, 3, 4, 9),
E          (1, 2, 3, 4),
E       -  (3, 4, 9),
E       ?         ^
E       
E       +  (2, 3, 4, 5),
E       ?   +++      ^
E       
E       -  (4, 9),
E          (2, 3, 4),
E       +  (3, 4, 5),
E          (3, 4),
E       +  (4, 5),
E       -  (4,)]
E       ?      ^
E       
E       +  (4,),
E       ?      ^
E       
E       +  (5,)]

compreffor/testPyCompressor.py:67: AssertionError
----------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------
Warning: non-TTFont given to Compreffor
______________________________________________________________________ TestCffCompressor.test_get_suffixes ______________________________________________________________________

self = <compreffor.testPyCompressor.TestCffCompressor testMethod=test_get_suffixes>

    def test_get_suffixes(self):
        """Test the results of suffix array construction."""

        ans = self.short_sf.get_suffixes()

>       self.assertEqual(ans, [(0, 0), (1, 1), (0, 1), (0, 2), (1, 0), (1, 2)])
E       AssertionError: Lists differ: [(0, 0), (0, 1), (1, 0), (0, 2), (1, 1), (1, 2)] != [(0, 0), (1, 1), (0, 1), (0, 2), (1, 0), (1, 2)]
E       
E       First differing element 1:
E       (0, 1)
E       (1, 1)
E       
E       - [(0, 0), (0, 1), (1, 0), (0, 2), (1, 1), (1, 2)]
E       + [(0, 0), (1, 1), (0, 1), (0, 2), (1, 0), (1, 2)]

compreffor/testPyCompressor.py:82: AssertionError
----------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------
Warning: non-TTFont given to Compreffor
====================================================================== 3 failed, 17 passed in 0.15 seconds ======================================================================

Segmentation fault

The attached file will segfault with the C++ compressor, at least on my machine.

$ uname -a
Darwin upo.lan 18.5.0 Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64 i386 MacBookPro9,2 Darwin
$ compreffor -v -v -v -n 2 Test.otf FallbackPlus-Regular2.otf
Compressing font through C++ Compreffor
Took 0.155s to produce data for C++ library
[1]    46636 segmentation fault  compreffor -v -v -v -n 2 master_otf/Test.otf FallbackPlus-Regular2.otf

segfault.zip

build wheels for python3.7

Renaming default branch name from "master" to "main"

The default branch in this repo is now called main. All open pull requests were automatically updated accordingly and github.com gives us this advice:

Your members will have to manually update their local environments. We'll let them know when they visit the repository, or you can share these commands:

git branch -m master main
git fetch origin
git branch -u origin/main main

Enable Appveyor CI for push and pull requests

I tried to open one from my personal appveyor account but it doesn't trigger
https://ci.appveyor.com/project/anthrotype/compreffor-16l0f

I believe it needs to be an “organizational” account registered under googlei18n
https://www.appveyor.com/docs/team-setup/

Desubroutinize if needed

For example, running it on Noto's third_party/noto_cjk/NotoSansKR-Regular.otf results in:

roozbeh@machine-name:~/compreffor/compreffor$ ./cxxCompressor.py NotoSansKR-Regular.otf 
Warning: There are subrs in NotoSansKR-Regular.otf
Traceback (most recent call last):
  File "./cxxCompressor.py", line 387, in <module>
    main(**kwargs)
  File "./cxxCompressor.py", line 342, in main
    handle_font(filename)
  File "./cxxCompressor.py", line 320, in handle_font
    font.save(out_name)
  File "/home/roozbeh/fonttools/Lib/fontTools/ttLib/__init__.py", line 214, in save
    self._writeTable(tag, writer, done)
  File "/home/roozbeh/fonttools/Lib/fontTools/ttLib/__init__.py", line 629, in _writeTable
    tabledata = self.getTableData(tag)
  File "/home/roozbeh/fonttools/Lib/fontTools/ttLib/__init__.py", line 642, in getTableData
    return self.tables[tag].compile(self)
  File "/home/roozbeh/fonttools/Lib/fontTools/ttLib/tables/C_F_F_.py", line 20, in compile
    self.cff.compile(f, otFont)
  File "/home/roozbeh/fonttools/Lib/fontTools/cffLib.py", line 72, in compile
    for child in topCompiler.getChildren(strings):
  File "/home/roozbeh/fonttools/Lib/fontTools/cffLib.py", line 243, in getChildren
    children.extend(topDict.getChildren(strings))
  File "/home/roozbeh/fonttools/Lib/fontTools/cffLib.py", line 1422, in getChildren
    charStringsComp = CharStringsCompiler(items, strings, self)
  File "/home/roozbeh/fonttools/Lib/fontTools/cffLib.py", line 180, in __init__
    self.items = self.getItems(items, strings)
  File "/home/roozbeh/fonttools/Lib/fontTools/cffLib.py", line 286, in getItems
    cs.compile()
  File "/home/roozbeh/fonttools/Lib/fontTools/misc/psCharStrings.py", line 289, in compile
    "seac"), "illegal CharString"
AssertionError: illegal CharString

Failed to compress if len(FDArray) == 256

The C++ Compreffor fails to compress FDArrayTest257.otf:

$ compreffor -d FDArrayTest257.otf
Traceback (most recent call last):
  File "/Users/mnakamura/.pyenv/versions/global-2.7.13/bin/compreffor", line 11, in <module>
    sys.exit(main())
  File "/Users/mnakamura/.pyenv/versions/2.7.13/envs/global-2.7.13/lib/python2.7/site-packages/compreffor/__main__.py", line 112, in main
    compreffor.compress(font, **options)
  File "/Users/mnakamura/.pyenv/versions/2.7.13/envs/global-2.7.13/lib/python2.7/site-packages/compreffor/__init__.py", line 83, in compress
    cxxCompressor.compreff(ttFont, **options)
  File "/Users/mnakamura/.pyenv/versions/2.7.13/envs/global-2.7.13/lib/python2.7/site-packages/fontTools/misc/loggingTools.py", line 372, in wrapper
    return func(*args, **kwds)
  File "/Users/mnakamura/.pyenv/versions/2.7.13/envs/global-2.7.13/lib/python2.7/site-packages/compreffor/cxxCompressor.py", line 210, in compreff
    input_data = write_data(td)
  File "/Users/mnakamura/.pyenv/versions/2.7.13/envs/global-2.7.13/lib/python2.7/site-packages/fontTools/misc/loggingTools.py", line 372, in wrapper
    return func(*args, **kwds)
  File "/Users/mnakamura/.pyenv/versions/2.7.13/envs/global-2.7.13/lib/python2.7/site-packages/compreffor/cxxCompressor.py", line 92, in write_data
    fdselect = struct.pack('B', len(td.FDArray)) + array.array('B', list(td.FDSelect)).tostring()
struct.error: ubyte format requires 0 <= number <= 255

This font has 256 FDArray elements, and it is the architectural limit of CID-keyed fonts.

AttributeError: '_TTGlyphCFF' object has no attribute 'decompile'

First of all, thank you very much for open-sourcing this library.

I am trying to run the python-only version, "pyCompressor.py", on some CFF-OTF fonts of ours.
I'm running python 2.7.9 on OS X. FontTools is checked out at the latest commit (c0eb70308b47b20137fe74d421d29d81e095f68f).
I have previously used pyftsubset to --desubroutinize the fonts.

When I run python compreffor on the fonts, I always get an AttributeError exception:

$ python pyCompressor.py /Users/cosimolupo/Desktop/LushHandwritten_Rg.subset.otf 
Traceback (most recent call last):
  File "pyCompressor.py", line 1127, in <module>
    main(**kwargs)
  File "pyCompressor.py", line 1071, in main
    handle_font(filename)
  File "pyCompressor.py", line 1048, in handle_font
    compreffor.compress()
  File "pyCompressor.py", line 494, in compress
    n_locals)
  File "pyCompressor.py", line 588, in iterative_encode
    sf = SubstringFinder(glyph_set, verbose=self.verbose)
  File "pyCompressor.py", line 253, in __init__
    self.process_chstrings(glyph_set)
  File "pyCompressor.py", line 270, in process_chstrings
    char_string.decompile()
AttributeError: '_TTGlyphCFF' object has no attribute 'decompile'

_TTGlyphCFF, as defined in fontTools.ttLib.__init__, does not have a decompile method. But its self._glyph attribute (i.e. the wrapped T2CharString object) does have one. So it seems like some changes have occurred in upstream fontTools since the time pyCompressor.py:270 was originally written.

However, even if I change the previous line (269) so that it accesses the _glyph attribute:

        for k in self.glyph_set_keys:
-            char_string = glyph_set[k]
+            char_string = glyph_set[k]._glyph
            char_string.decompile()
            program = []

then, the charstring can be decompiled, but the following assertion fails with:

Traceback (most recent call last):
  File "pyCompressor.py", line 1127, in <module>
    main(**kwargs)
  File "pyCompressor.py", line 1071, in main
    handle_font(filename)
  File "pyCompressor.py", line 1048, in handle_font
    compreffor.compress()
  File "pyCompressor.py", line 494, in compress
    n_locals)
  File "pyCompressor.py", line 588, in iterative_encode
    sf = SubstringFinder(glyph_set, verbose=self.verbose)
  File "pyCompressor.py", line 253, in __init__
    self.process_chstrings(glyph_set)
  File "pyCompressor.py", line 276, in process_chstrings
    i == len(char_string.program) - 1
AssertionError

I'm afraid I don't know much about how CFF works, so I don't understand the logic of this assertion. I don't think there's anything wrong with the font files, as that assertion is raised on any font I've run through pyCompressor.py. If I comment out the assertion, then the script completes without errors. However, there must have been a reason if @samfishman put an assert there.

Btw, the cxxCompressor.py (both when using subprocess and the dynamic library) seem to be working fine. It's only the python version which has the above mentioned problems.

Thank you again,
Cheers.

Cosimo

--decompress does more than just desubroutinize

I noticed that when the "-d" (or "--decompress") option is provided to compreffor, then the output file size gets much smaller.
The reason is that the fontTools.subset.Options include a lot more optimizations which are performed by default, besides the requested desubroutinization.
From a user point of view, the "decompress" option should only desubroutinize the CFF table, and not also perform the rest of the optimizations.
I think the best way to address this would be to make the desubroutinization independent from the fontTools.subset module, as @behdad already noted here fonttools/fonttools#283.

Consider SEQUITUR(Nevill-Manning) or Re-pair compression algorithm

cf.
http://www.sequitur.info/
https://arxiv.org/pdf/cs/9709102v1.pdf