samuelcolvin / xdelta3-python Goto Github PK

View Code? Open in Web Editor NEW

34.0 6.0 15.0 33 KB

Fast delta encoding in python using xdelta3

License: Other

Python 67.13% C 20.62% Makefile 8.10% Shell 4.14%

delta-encoding xdelta3 python python3 c vcdiff rfc-3284

xdelta3-python's Introduction

xdelta3-python

Fast delta encoding in python using xdelta3.

Requirements

Python 3.5 or 3.6 - it's 2017, you should be using python 3.6 by now anyway.
linux - compilation only tested on ubuntu, might work on other platform.

Installation

pip install xdelta3

Usage

import xdelta3
value_one = b'wonderful string to demonstrate xdelta3, much of these two strings is the same.'
value_two = b'different string to demonstrate xdelta3, much of these two strings is the same.'
delta = xdelta3.encode(value_one, value_two)
# delta is an unreadable byte string: b'\xd6\xc3 ... \x01different\n\x13F\x00'

print(f'New string length: {len(value_two)}, delta length: {len(delta)}')
value_two_rebuilt = xdelta3.decode(value_one, delta)
if value_two_rebuilt == value_two:
    print('Boo Ya! Delta encoding successful.')

(with xdelta3 installed this code should run "as is", just copy it into ipython or a file and run)

How fast?

xdelta3-python is a thin wrapper around xdelta 3.1.1 which is a highly optimised c library for delta calculation and compression. It can encode a delta and decode it again for 5 small changes in a 5.5 million character string (the complete works of shakespeare) in around 10ms (or 30ms with the highest compression level). Boom. See performance.py.

xdelta3-python's People

Contributors

Stargazers

Watchers

Forkers

mobuchowski trousev karamanolev sagiben nickrose smilesydney rosetechnology terencelim gc-ss jacobfeder synodriver mike8699 absoluteblack rafaelgodoyebert thygrrr

xdelta3-python's Issues

DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats

x@192-168-10-195 Inferius-master % ./bundlegen -d 'iPhone10,3' -i '14.5'                                                       
Creating Firmware Bundle for iPhone10,3, iOS 14.5
[1] Grabbing decryption keys...
[2] Patching bootchain...
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
[3] Patching kernel...
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
[4] Patching ramdisk...
    Extracting 038-44812-292.dmg...
    Decrypting 038-44812-292.dmg_rdsk.dmg...
    Mounting 038-44812-292.dmg_rdsk.dmg...
    Copying ramdisk/usr/sbin/asr to asr
    Unmounting 038-44812-292.dmg_rdsk.dmg...
    Patching asr -> asr.pwn
    Setting permissions 755 for asr.pwn
    Mounting 038-44812-292.dmg_rdsk.dmg...
    Copying asr.pwn to ramdisk/usr/sbin/asr
    Unmounting 038-44812-292.dmg_rdsk.dmg...
    Creating 038-44812-292.dmg.im4p from 038-44812-292.dmg_rdsk.dmg...
    Calculating 038-44812-292.dmg | 038-44812-292.dmg.im4p diff...
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
    Saving diff as 038-44812-292.asr.delta
[5] Making Firmware Bundle...
Finished creating Firmware Bundle for iPhone10,3, iOS 14.5: 'FirmwareBundles/iPhone10,3_14.5_18E199.bundle'.

Windows support would be nice

I'm using this on this tool:
https://github.com/i30817/zxd3/

And this not being a OS independent package or have a windows version uploaded to pypi prevents me from releasing it on pypi for windows.

Unfortunately i know less than nothing about making native code compilable for windows and how to make pypi make that happen, so this is a issue, not a pull request.

DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats

Running

with open(ramdisk['path'], "rb") as original, \
       open(f"{ramdisk['path']}.im4p", "rb") as new_file, \
       open(f"{bundle}/{ramdisk['name'][:-4]}.asr.delta", "wb") as delta:
       delta.write(xdelta3.encode(original.read(), new_file.read()))

results in

/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)
/Users/x/.pyenv/versions/3.9.0/lib/python3.9/site-packages/xdelta3/main.py:89: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  return _xdelta3.execute(new_value, original, flags, 0)

View this issue for details: python-pillow/Pillow#3750

Please update code to remove warnings

How to install package using branch fix-large-decode

I've been using the master branch installed via pip but I'm hitting the xdelta3.XDeltaError.

I tried installing via pip install -e git+https://github.com/samuelcolvin/xdelta3-python.git@fix-large-decode but getting this error:

xdelta3/_xdelta3.c:3:10: fatal error: xdelta3.h: No such file or directory                                                                                       3 | #include "xdelta3.h"
        |          ^~~~~~~~~~~
  compilation terminated.

Any tips on how to build with the original xdelta libraries via pip install?

It's 2021, you should be using python 3.10 by now anyway

xdelta3 library fails on Python 3.10

Now this library doesn't work at all

Support memoryview ranges in decode and encode

I know that some python libraries with native code have added support for this (i needs some kind of explicit reference type).

Reference: indygreg/python-zstandard#26

This can save a lot of memory depending on your buffers size. Feel free to close this if the bytes passed to the encoding and decoding functions need to operate in place on the arrays.

Disabling secondary compression

I need to disable secondary compression for delta files because does open-vcdiff does not support it. However, I fail to convince xdelta3 to skip secondary compression. The command line tool keeps adding that level of compression even with no option argument to -S, and this python wrapper seems to have a similar issue.

In my code, I currently have

delta = xdelta3.encode(other, image, xdelta3.Flags.SEC_NOALL)

which gives me the following error at run-time:

Traceback (most recent call last):
  File "./pack-image.py", line 15, in <module>
    main()
  File "./pack-image.py", line 11, in main
    delta = xdelta3.encode(other, image, xdelta3.Flags.SEC_NOALL)
  File "/usr/lib64/python3.6/site-packages/xdelta3/main.py", line 89, in encode
    return _xdelta3.execute(new_value, original, flags, 0)
xdelta3.XDeltaError: Error occur executing xdelta3: XD3_INTERNAL

Without xdelta3.Flags.SEC_NOINST, the code works, but the image features secondary compression.

Any idea?

delta size different from xdelta-cli

Environment

OS: Debian 9
xdelta3 version : 3.10
xdelta3-python version: 0.0.5

Reproduce

echo $'11111" > 1.txt

echo $'000000\n000000\n000000\n000000\n000000\n000000\n000000\n' > 2.txt

xdelta3 -e -s 1.txt 2.txt delta

in python
had data1 from 1.txt and data2 from 2.txt
writes xdelta3.encode(data1, data2) to py_delta

Question

py_delta is 20B, but delta is 39B.
use xdelta3 -d -s 1.txt delta 3.txt or xdelta3 -d -s 1.txt py_delta 3.txt all get the same result without problem.

It works fine, but I'm still unnecessarily worrying, did I miss some arguments with xdelta3?
And how to get the same result?

Does the encode method have the distributive property?

I'm trying to xdelta the contents of zip files without extracting them to disk. Therefore i need a cycle of dumping bytes from the zip extract to memory, delta-diffing those those bytes and writing the result to file, truncating the memory offset used and redoing the cycle. It's somewhat easy to create a generator that just gives all the bytes in two lists of files for python with yield and zip_longest, so that's not a problem.

My uncertainty comes from if this will work to create a equivalent xdelta file to having the complete file. Xdelta algorithm doesn't have to 'seek' in its input files right? And is your published api (a single 'encode' method for writing the diff) enough for this usecase of feeding partial contents of files to the decode method and writing that output to disk and that is not different from having the complete byte array and doing a encode?

ie:

if s == s1 ++ s2
encode(s) = encode(s1) ++ encode(s2)

If not, is the above possible if compression method is 0 ?

xdelta3.XDeltaError: Output of decoding delta longer than expected

I'm trying to use your library. Here's a simple example of how I'm trying it:

    b1 = open('b1.bin', 'rb').read()
    b2 = open('b2.bin', 'rb').read()

    d = xdelta3.encode(b1, b2)
    b3 = xdelta3.decode(b1, d)
    print(b2 == b3)

This gives me an exception: xdelta3.XDeltaError: Output of decoding delta longer than expected

I'm attaching a zip file with b1.bin and b2.bin. By the way, they're simple HTML files, I just made sure to treat them as binary so I don't trip over encodings. Also, when trying it in reverse (not b1 -> b2, but b2 -> b1) it all works well.

$ python --version
Python 3.6.1
Running on Mac OS, Python installed via pyenv. xdelta3==0.0.5.

files.zip