mostawesomedude / construct Goto Github PK
View Code? Open in Web Editor NEWPython binary parsing library
License: MIT License
Python binary parsing library
License: MIT License
It would be nice if collections included a way to print them that truncated long sequences. For now, I'm using this:
def trunc(text):
ret = []
for l in str(text).splitlines():
if len(l) > 100:
l = l[:40] + ' ... ' + l[-40:]
ret.append(l)
return "\n".join(ret)
Please separate the license into a LICENSE file. This makes for better packaging.
Hi,
I've tried to post a message to http://construct.wikispaces.com/message/list/home but I need to join the wiki and it's invite-only.
Wouldn't be easier to move everything to github or use google groups?
The upstream repository has been https://github.com/construct/construct for a long while. For disclosure, I am its new maintainer. You should probably move there. This repository probably was not looked at for years now.
This parses EXIF and JFIF files. this is my first construct, and while making it I noticed I was missing a couple things:
class FastReader(Construct):
def _parse(self, stream, context):
return stream.read()
def _build(self, obj, stream, context):
stream.write(obj)
SegBody = Struct(None,
UBInt16('size'),
Field('data', lambda ctx: ctx['size'] - 2),
)
Seg = Struct('seg',
Literal('\xff'),
Byte('kind'),
Switch('body', lambda c: c['kind'],
{
SOS: FastReader('data'),
},
default = Embed(SegBody),
)
)
JPEG = Struct('jpeg',
Literal('\xff\xd8'),
GreedyRange(Seg),
)
pi@raspberrypi:~/gr-satellites/build/construct $ make upload
./setup.py sdist upload
running sdist
running egg_info
writing requirements to construct.egg-info/requires.txt
writing construct.egg-info/PKG-INFO
writing top-level names to construct.egg-info/top_level.txt
writing dependency_links to construct.egg-info/dependency_links.txt
reading manifest file 'construct.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'construct.egg-info/SOURCES.txt'
running check
creating construct-2.9.45
creating construct-2.9.45/construct
creating construct-2.9.45/construct.egg-info
creating construct-2.9.45/construct/lib
copying files to construct-2.9.45...
copying LICENSE -> construct-2.9.45
copying MANIFEST.in -> construct-2.9.45
copying README.rst -> construct-2.9.45
copying setup.py -> construct-2.9.45
copying construct/__init__.py -> construct-2.9.45/construct
copying construct/core.py -> construct-2.9.45/construct
copying construct/debug.py -> construct-2.9.45/construct
copying construct/expr.py -> construct-2.9.45/construct
copying construct/version.py -> construct-2.9.45/construct
copying construct.egg-info/PKG-INFO -> construct-2.9.45/construct.egg-info
copying construct.egg-info/SOURCES.txt -> construct-2.9.45/construct.egg-info
copying construct.egg-info/dependency_links.txt -> construct-2.9.45/construct.egg-info
copying construct.egg-info/requires.txt -> construct-2.9.45/construct.egg-info
copying construct.egg-info/top_level.txt -> construct-2.9.45/construct.egg-info
copying construct/lib/__init__.py -> construct-2.9.45/construct/lib
copying construct/lib/binary.py -> construct-2.9.45/construct/lib
copying construct/lib/bitstream.py -> construct-2.9.45/construct/lib
copying construct/lib/containers.py -> construct-2.9.45/construct/lib
copying construct/lib/hex.py -> construct-2.9.45/construct/lib
copying construct/lib/py3compat.py -> construct-2.9.45/construct/lib
Writing construct-2.9.45/setup.cfg
Creating tar archive
removing 'construct-2.9.45' (and everything under it)
running upload
Password:
I am trying to pass a param which contains a padding byte b"\x55", but somewhere along the process it is getting converted to an int and then I get the following:
construct.core.PaddingError: pattern expected to be bytes of length 1
the next tests fail:
>>> from construct import *
from construct.text import *
TextualIntAdapter(Field("textintadapter", 3)).build(-12)
'12-'
TextualIntAdapter(Field("textintadapter", 1)).build(0)
Traceback (most recent call last):
File "", line 1, in
File "construct/core.py", line 206, in build
self.build_stream(obj, stream)
File "construct/core.py", line 214, in build_stream
self._build(obj, stream, Container())
File "construct/core.py", line 284, in _build
self.subcon._build(self._encode(obj, context), stream, context)
File "construct/core.py", line 324, in _build
_write_stream(stream, self.length, obj)
File "construct/core.py", line 306, in _write_stream
raise FieldError("expected %d, found %d" % (length, len(data)))
construct.core.FieldError: expected 1, found 0
When building a Struct which contains an Anchor, not providing the anchor's label as a key in the Container instance yields an AttributeError.
#!/usr/bin/python2
from construct import *
struct = Struct(None, Anchor('foo_anchor'))
struct.build(Container())
Running the testcase yields: AttributeError: 'Container' object has no attribute 'foo_anchor'
As an anchor's value is solely determined by it's position in the construct, it shouldn't have to be specified in the Container?
This seems to be an issue with CString being an adapted RepeatUntil, which always throws an error for sizeof calls.
>>> from construct import *
>>> str = CString("a_string")
>>> str_parsed = str.parse("foo\x00")
>>> print str_parsed
foo
>>> str.sizeof()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/construct-2.06-py2.7.egg/construct/core.py", line 242, in sizeof
raise SizeofError(e)
construct.core.SizeofError: can't calculate size
>>> str.sizeof(str_parsed)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/construct-2.06-py2.7.egg/construct/core.py", line 242, in sizeof
raise SizeofError(e)
construct.core.SizeofError: can't calculate size
There seems to be a missing .iteritems() in the pretty_str function which causes the printing of containers to fail. Changing the line to "for k, v in self.iteritems()" fixes it for me. Traceback is included below. Running on Python 2.7.2 on Windows 7.
Traceback (most recent call last):
File "replay.py", line 25, in <module>
print y
File "C:\Python27\lib\site-packages\construct-2.05-py2.7.egg\construct\lib\container.py", line 86, in __str__
return self.__pretty_str__()
File "C:\Python27\lib\site-packages\construct-2.05-py2.7.egg\construct\lib\container.py", line 14, in wrapper
return func(self, *args, **kw)
File "C:\Python27\lib\site-packages\construct-2.05-py2.7.egg\construct\lib\container.py", line 92, in __pretty_str__
for k, v in self:
ValueError: too many values to unpack
After a quick check, it appears that construct works to some degree after running the 2to3 tool... Cstring and Repeater objects can be created, can parse, and can build. However, the Bytes object cannot parse - it wants a StringIO, not a bytesIO, yet it can't accept a str object.
Anyways, it would be nice to have a python3 version that worked (and was on PyPi)!
This is more like a question rather an issue. I'd expect MEtaField's related length fiekd to be autopopulated on building a structure, however this does not happen:
>>> from construct import *
>>> foo = Struct("foo",
... Byte("length"),
... MetaField("data", lambda ctx: ctx["length"])
... )
>>> foo.build(Container(data="test"))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/pymodules/python2.6/construct/core.py", line 206, in build
self.build_stream(obj, stream)
File "/usr/lib/pymodules/python2.6/construct/core.py", line 214, in build_stream
self._build(obj, stream, Container())
File "/usr/lib/pymodules/python2.6/construct/core.py", line 662, in _build
subobj = getattr(obj, sc.name)
AttributeError: 'Container' object has no attribute 'length'
Could you explain correct behavior?
May be it is Python's bindata
which is a Ruby gem.
Hi sorry about this, the 'setup.py' is supposed to install construct isn't it?
I ran it, CMD pops up then closes immediately.
Any help? :(
Hi,
I'm trying to build a Construct for a specific protocol and am a bit stuck. I wonder whether you guys can give me a tip.
Our message contains a list of records. The first entry is uncompressed, the following entries are compressed. That means that each field of a compressed entry depends on the value of the previous entry.
Message = Struct("Message",
Embed( Uncompressed ),
OptionalGreedyRange( Compressed )
)
At the moment I've parsing the raw bits of a Compressed entry as am not sure what I should do to refer to the previous entry.
I can think of two strategies:
Ideally my struct would return a list containing all entries. Right now, with the above Struct, I'm getting the fields of the uncompressed entry plus a list of the other entries.
Thank you!! I love Construct and have been making progress with my port to Java: https://github.com/ziglionz/construct
Hello, engineer,I want to see the historical version of the document, where can I get it?
I am dealing with large Length-Value packages, so I memory-mapped the file I am reading and trying to use lazy evaluation to skip the value field, till I actually need to read it from file.
However, the parsing fails as soon as I start to use Lazy(Bytes(this.length))
instead of LazyArray(this.length))
.
I created a little example to highlight the issue:
import construct as c
data = b"\x02ab\x03abc\x04abcd"
success = c.GreedyRange(c.Struct("len" / c.Byte,
"val" / c.LazyArray(c.this.len, c.Byte))).parse(data)
print(success)
print()
failed = c.GreedyRange(c.Struct("len" / c.Byte,
"val" / c.Lazy(c.Bytes(c.this.len)))).parse(data)
print(failed)
This is the resulting output:
ListContainer:
Container:
len = 2
val = <LazyListContainer: 0 of 2 items cached>
Container:
len = 3
val = <LazyListContainer: 0 of 3 items cached>
Container:
len = 4
val = <LazyListContainer: 0 of 4 items cached>
ListContainer:
Container:
len = 2
val = <function Lazy._parse.<locals>.execute at 0x7fdecd3c3f28>
Container:
len = 97
val = <function Lazy._parse.<locals>.execute at 0x7fdecd3c39d8>
Container:
len = 98
val = <function Lazy._parse.<locals>.execute at 0x7fdecd3c3d08>
Container:
len = 3
val = <function Lazy._parse.<locals>.execute at 0x7fdecd316510>
Container:
len = 97
val = <function Lazy._parse.<locals>.execute at 0x7fdecd316598>
Container:
len = 98
val = <function Lazy._parse.<locals>.execute at 0x7fdecd316620>
Container:
len = 99
val = <function Lazy._parse.<locals>.execute at 0x7fdecd3166a8>
Container:
len = 4
val = <function Lazy._parse.<locals>.execute at 0x7fdecd316730>
Container:
len = 97
val = <function Lazy._parse.<locals>.execute at 0x7fdecd3167b8>
Container:
len = 98
val = <function Lazy._parse.<locals>.execute at 0x7fdecd316840>
Container:
len = 99
val = <function Lazy._parse.<locals>.execute at 0x7fdecd3168c8>
Container:
len = 100
val = <function Lazy._parse.<locals>.execute at 0x7fdecd316950>
What am I doing wrong or might this be a bug?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.