I haven't looked into this too deeply, but just thought I'd ask. How come the length of the binfile object is not equal to the length of the bytes returned by binfile.as_binary()?

For example, if you added this statement to the regression test code, the test will fail.

--- a/tests/test_bincopy.py
+++ b/tests/test_bincopy.py
@@ -397,6 +397,7 @@ class BinCopyTest(unittest.TestCase):
         self.assertEqual(binfile.minimum_address, 0)
         self.assertEqual(binfile.maximum_address, 184)
         self.assertEqual(len(binfile), 170)
+        self.assertEqual(len(binfile), len(binfile.as_binary()))

         # Dump with start address beyond end of binary.
         self.assertEqual(binfile.as_binary(minimum_address=512), b'')

Could you explain the difference between the two?

Thanks!

import bincopy
hexdata = bincopy("myfile.hex")
print(len(next(hexdata.segments.chunks(size=32))))

2

I find this surprising. Anyone doing len(chunk) probably wants the length of its data member, not the fact that it's a two-tuple. I propose implementing Chunk like so:

class Chunk(namedtuple("Chunk", ["address", "data"])):  # Or _Chunk, depending on outcome of #35
    def __len___(self):
        return len(self.data)

While this would technically change the API, I argue that it is extremely unlikely to break anything; anyone doing len(chunk) almost certainly means len(chunk.data).

Hello,

Due to our needs of our project, the headers could have binary values that represents Integer values over than ASCII such as 0x88, but the library does not allow it rasing an exception:

`bincopy.py in as_srec(self, number_of_data_bytes, address_length_bits)
684
685 if self._header:
--> 686 encoded_header = self._header.encode('utf-8')
687 record = pack_srec('0', 0, len(encoded_header), encoded_header)
688 header.append(record)

UnicodeDecodeError: 'ascii' codec can't decode byte 0x88 in position 0: ordinal not in range(128)
`

Could you modify the library to not enconde in UTF-8 and allow any kind of value to not limit only string headers?

Many thanks!!
Ferran.

records = """
    :02000004000AF0
    :10B8440000000000000000009630000007770000B0
"""
hexfile = bincopy.BinFile()
hexfile.add_ihex(records)
align = 8
chunks = hexfile.segments.chunks(size=16, alignment=align)
assert not any(c.address % align for c in chunks)

IMO, the first chunk should be padded such that it is also aligned.

For my use-case it would be useful for as_binary to raise an Exception if I ask for data which is not contained in the files I have added.

At the moment, if padding is None, it is converted to a default of 0xFF . I would request the following (which admittedly changes the API):

make the default parameters more explicit:

def as_binary(self,
                  minimum_address=None,
                  maximum_address=None,
                  padding=0xFF):

and then alter the behaviour of padding==None to raise a RuntimeException if there is any data requested which does not 'exist'. Would this be acceptable?

Hi there,

I am running the following script :

import bincopy 

f = bincopy.BinFile("Test_2.hex")
print(f.as_binary())

I get the following error :

I noticed that when I reduce the size of the binary file, it works. (the original binary is around 26 KB)

Thanks in advance

There is an issue with outputting type 4 addresses that are > 0xffff, they create address that have 5 digits, and are invalid in the format. i.e. have a segment with address as 0x1000e, and it will be output as two records, one the Extended Linear Address record which is correct, but then the data record has all 5 didgits of the address. example:

read and then write the following text:

:020000040001F9
:04000E00A4CFFFDD9F

you will get an error on the write due to the CRC routine getting an odd number of digits:

:020000040001F9
:041000E00A4CFFFDD9F note address is 0x1000e

the issue is in the as_ihex function:

def as_ihex(self, size=32, address_length=32):
"""Return string of Intel HEX records of all data.

    """

    data_address = []
    extmaximumed_address = -1

    for address, data in self.segments.iter(size):
        address //= self.word_size_bytes

        if address_length == 32:
            if ((address >> 16) & 0xffff) > extmaximumed_address :
                extmaximumed_address = ((address >> 16) & 0xffff)

I added this fix as a hack for my situation, but would not work for subsequent segments that are in the same address space.

                address = address & 0xffff   

end of hack

                packed = pack_ihex(4,
                                   0,
                                   2,
                                   binascii.unhexlify('%04X'
                                                      % extmaximumed_address))
                data_address.append(packed)
        else:
            raise Error('unsupported address length %d'
                             % address_length)

        data_address.append(pack_ihex(0, address, len(data), data))

But the function also assumes that all address are in sorted order, which I do not believe is the case for all Intel hex files, unless you are sorting them as you insert into the list, I may have missed that.

I'm having a weird issue that, to be honest, I can't imagine how it's a bincopy issue but it doesn't appear to affect any other packages. When our admin bootstraps a Windows machine he installs bincopy (and several other packages). Later, a user runs python and attempts to run import bincopy and gets an ImportError. Logging in as admin, I can successfully perform the import.

We suspected it was a permissions error so we tested our theory by changing the owner of the Libs directory to the present user (recursively) and ensuring proper permissions were set, but it made no difference. We then logged in as admin and removed bincopy (pip uninstall bincopy), then reinstalled as the user account (pip install bincopy) and everything worked fine. In both cases bincopy was installed into C:\Python27\Libs\site-packages which is in the search path for both users.

I looked through the package and can't find anything that should be affected by installing user. Have you seen anything like this? I have been installing and using bincopy under Linux for a while without any such issues.

SRecord reading fails on hex files targeting devices with 16-bit bytes (such as TI Piccolo), where 4 hex digits increment address by 1. Below is a beginning of such file:

S00600004844521B
S2083E80000123ABCD9D
S2223E8002761B0005ABBDA8BDA0BDC2BDC3BDFE04FF6929425616761F002C0E01761FB8
S2223E80110227FF3D1E00761F002C0E02761F0227FF3D1E10761F002C0E03761F022702

See how 0123ABCD changes address from 3E8000 to 3E8002.

Would be nice to have a way to select byte width.

after updating 17.9 -> 17.10, bincopy can no longer be used as a module entry point

python -m bincopy
now returns

NameError: name 'main' is not defined

i havent investigated the error properly myself, but in bincopy.py there are two main shields which looks fishy to me.

https://github.com/eerimoq/bincopy/blob/master/bincopy.py#L321
https://github.com/eerimoq/bincopy/blob/master/bincopy.py#L2017

thank you for your great work!

The Microchip HEX format is identical to the Intel format, except the addresses in the HEX file are twice the actual machine addresses. This is because Microchip's PIC architecture typically has wider instructions than its machine word. For example, the PIC24 uses 16-bit words but 24-bit instructions.

This can be handled by parsing the file as Intel format with a word size of one byte, and then changing the word size to two bytes after parsing is done.

I can make a PR if you think this would be a useful feature. It could be enabled with an optional microchip=False argument to BinFile.

Hi
I was checking the code in bincopy.py and I think the if condition used to check for non-overlapping segments in remove_data is incorrect:

if ((minimum_address >= self.maximum_address)
            and (maximum_address <= self.minimum_address)):

This will only be true in the rare case of trying to remove a zero-length segment from a zero-length segment at the same address which is not what was intended. I think the and condition should be changed to or i.e.

if ((minimum_address >= self.maximum_address)
            or (maximum_address <= self.minimum_address)):

The line crc = (0x100 - crc) will give 0x100 instead of 0x00 for the two's complement of zero.

Try crc = (~crc + 1) & 0xff instead.

Hello Erik,

I am not able to parse ELF file produced by CodeWarrior. I get the following error: bincopy.AddDataError: data added to a segment must be adjacent to or overlapping with the original segment data

However, I am able to parse it using the pyelftools.

Attached is the ELF file and corresponding binary created by the objcopy.
files.zip

Thanks, Marek

Hi Erik,
This is a very cool tool, but the instruction for converting a file from (for example) Intel HEX to binary is missing. How do you do that?
Cheers,

Thanks for the useful library.

Is it possible to extract part of Data by start address from S19 and convert to binary.
because in my S19 there are jumps in addresses and there is no need to convert all of them to binary and add 0xFF in such jumped places.

Hi! When trying to package 17.14.3 for Arch Linux I had two failing tests, as there are test files missing in the sdist tarball:

=================================== FAILURES ===================================
_______________________ BinCopyTest.test_add_elf_blinky ________________________

self = <tests.test_bincopy.BinCopyTest testMethod=test_add_elf_blinky>

    def test_add_elf_blinky(self):
        bf = bincopy.BinFile()
>       bf.add_elf_file('tests/files/evkbimxrt1050_iled_blinky_sdram.axf')

tests/test_bincopy.py:1777:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bincopy.BinFile object at 0x7fa750e43e80>
filename = 'tests/files/evkbimxrt1050_iled_blinky_sdram.axf', overwrite = False

    def add_elf_file(self, filename, overwrite=False):
        """Open given ELF file and add its contents. Set `overwrite` to
        ``True`` to allow already added data to be overwritten.

        """

>       with open(filename, 'rb') as fin:
E       FileNotFoundError: [Errno 2] No such file or directory: 'tests/files/evkbimxrt1050_iled_blinky_sdram.axf'

bincopy.py:1161: FileNotFoundError
_________________________ BinCopyTest.test_add_elf_gcc _________________________

self = <tests.test_bincopy.BinCopyTest testMethod=test_add_elf_gcc>

    def test_add_elf_gcc(self):
        bf = bincopy.BinFile()
>       bf.add_elf_file('tests/files/elf/gcc.elf')

tests/test_bincopy.py:1788:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <bincopy.BinFile object at 0x7fa750e43ac0>
filename = 'tests/files/elf/gcc.elf', overwrite = False

    def add_elf_file(self, filename, overwrite=False):
        """Open given ELF file and add its contents. Set `overwrite` to
        ``True`` to allow already added data to be overwritten.

        """

>       with open(filename, 'rb') as fin:
E       FileNotFoundError: [Errno 2] No such file or directory: 'tests/files/elf/gcc.elf'

bincopy.py:1161: FileNotFoundError
=========================== short test summary info ============================
FAILED tests/test_bincopy.py::BinCopyTest::test_add_elf_blinky - FileNotFound...
FAILED tests/test_bincopy.py::BinCopyTest::test_add_elf_gcc - FileNotFoundErr...
========================= 2 failed, 91 passed in 1.22s =========================

It would be nice to be able to use these types in annotations:

write_data(data: bincopy.Chunk) -> None:
    ...

instead of having to write:

write_data(data: bincopy._Segment._Chunk) -> None:
    ...

which is both harder on the eyes and makes various linters complain about accessing protected members.

It is not possible to load a srec file with python 3.2.1:

bincopyfile_obj = bincopy.BinFile()
bincopyfile_obj.add_srec_file(target_path)

The Problem:

    address = int(record[4:4+width], 16)
    data = binascii.unhexlify(record[4 + width:4 + 2 * size - 2]) # Throws Exception: 'str' does not support the buffer interface
    actual_crc = int(record[4 + 2 * size - 2:], 16)

The docs (Python 3.x) of binascii describes the behavior:
"a2b_* functions accept Unicode strings containing only ASCII characters. Other functions only accept bytes-like objects (such as bytes, bytearray and other objects that support the buffer protocol)."

Hello,
I have found an issue within parsing of ELF file generated for ARM Cortex M4 MCU generated by GCC compiler (IAR has same issue).

The issue is following:
As a ELF file has a lot of sections that should be placed in ROM area of MCU memory map, and one is BSS section (part of initial data that should be copied into RAM after MCU start) that is place after last .text section, but the BINCOPY placed it into address where will be copied after startup.

To simplify it:

The bin copy place this section on 0x1fff_0000 address instead of ~0x0000_9400.

To proof that I generate from GCC as ELF file (dev_hid_mouse_freertos.elf) as Intel HEX file(dev_hid_mouse_freertos.hex) both those file I convert by bincopy into S19 files(dev_hid_mouse_freertos_elf.s19, dev_hid_mouse_freertos_hex.s19) and compare that (See picture above). And to be sure that ELF file is correct I also convert ELF file by "arm-none-eabi-objcopy.exe" from GNU Tools and get binary file that's is using just right lower addresses (dev_hid_mouse_freertos.bin).

I guess that there is some issue with using of final address of data after MCU startup (any mark of reallocation in ELF probably) and address where is stored in ROM.
bincopy issue.zip

Thank you
Petr

I want to extract the data from S19 file.
I decided to convert it to binary in this way (as it is done in:https://stackoverflow.com/questions/24110121/how-to-manipulate-srec-file/45465864#45465864)

import bincopy
f = bincopy.BinFile()
f.add_srec_file("New.s19")
f.as_binary()

Then I want to save this object and extract data part.
I use .write to save but it has "TypeError: write() argument must be str, not BinFile" error.

with open('out.txt', 'w') as w:
    w.write(f)

How should i save it and what is your idea?
Thanks

One test fails on 64-bit Windows 10 host & 32-bit Python:

C:\GIT\bincopy>git log -1
commit 2f83b6f1570cb96bb925b416b631112c0ab4283e (HEAD -> master, tag: 14.4.0, origin/master, origin/HEAD)
Author: Erik Moqvist <[email protected]>
Date:   Mon Jul 16 08:08:12 2018 +0200

    Version 14.4.0.

C:\GIT\bincopy>git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        .project
        .pydevproject

nothing added to commit but untracked files present (use "git add" to track)

C:\GIT\bincopy>python2
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z


C:\GIT\bincopy>python3
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:06:47) [MSC v.1914 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> ^Z

Tests seem to fail:

C:\GIT\bincopy>python2 setup.py test
...
======================================================================
FAIL: test_binary (tests.test_bincopy.BinCopyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\git\bincopy\tests\test_bincopy.py", line 245, in test_binary
    padding=b'\x00'), fin.read())
AssertionError: bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:2001:20010000214601360121470136007EFE09D219012146017E17C20001FF5F16002148011979\r\n:20012000194E79234623965778239EDA3F01B2CA3F0156702B5E712B722B7321460134219F\r\n:00000001FF\r\n') != '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:2001:20010000214601360121470136007EFE09D219012146017E17C20001FF5F16002148011979\n:20012000194E79234623965778239EDA3F01B2CA3F0156702B5E712B722B7321460134219F\n:00000001FF\n'

----------------------------------------------------------------------
Ran 61 tests in 2.170s

FAILED (failures=1)

C:\GIT\bincopy>python3 setup.py test
running test
...
======================================================================
FAIL: test_binary (tests.test_bincopy.BinCopyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\git\bincopy\tests\test_bincopy.py", line 245, in test_binary
    padding=b'\x00'), fin.read())
AssertionError: bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x[205 chars]r\n') != b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00[188 chars]FF\n'

----------------------------------------------------------------------
Ran 61 tests in 1.912s

FAILED (failures=1)
Test failed: <unittest.runner.TextTestResult run=61 errors=0 failures=1>
error: Test failed: <unittest.runner.TextTestResult run=61 errors=0 failures=1>

There is fill(value=None, max_words=None)
that: Fills all empty space between segments
Is there any way to fill empty space between chunks?

how can we fill empty space in S19 file that is less than 4000 bytes and then create new S19 file?
the issue is how to write?

f = bincopy.BinFile("file1.s19")
print(f.info())
f.fill(value=b'\xFF', max_words=4000)   #4000)
print(f.info())
print(type(f))
with open("file2.s19") as s19:
    f.write(f.as_srec())

The error is:
f.write(f.as_srec())
AttributeError: 'BinFile' object has no attribute 'write'

thanks

Not sure if its "an issue" in code, naming or just maybe lack of doc, but maximum_address not maximum address of given file/segment, its behaves closer to end iterator (pointing 1 byte after last one?)

The BINCOPY create invalid section from ELF file. The ELF file is created in NXP MCUXpresso compiler.

The SREC from BINCOPY contains additional (probably irrelevant) data
Observation: the output contains some data at 0x8000000, while the image is located at 0x80002000.

Used OBJCOPY to validate result:
arm-none-eabi-objcopy.exe -O srec evkbimxrt1050_iled_blinky_sdram.axf evkbimxrt1050_iled_blinky_sdram.srec
Use BINCOPY to unify S record output to simplify comparison:
bincopy convert -i srec -o srec .\evkbimxrt1050_iled_blinky_sdram.srec .\evkbimxrt1050_iled_blinky_sdram.s19

Observation: no data at 0x8000000, the code starts at 0x80002000

Procedure to reproduce:

• Convert ELF file by bincopy:
  bincopy convert -i elf -o srec .\evkbimxrt1050_iled_blinky_sdram.axf converted.s19

• Compare result with evkbimxrt1050_iled_blinky_sdram.s19 created by "arm-none-eabi-objcopy.exe"
  bincopy.zip

Hello thanks for your great job, we already use bincopy in our projects. it works very well!
my question were do we have a function that can change the content in a specific address, and update to a new s19 file
for example
old
S32580040320FFFFFFFF0000100000010000000000000000000000000000000000000000000026
new
S32580040320FFFFFFFF00001000000FFFF000000000000000000000000000000000000000003C
hope your respond.

Hi,

you seem to only support the specialised TI Verilog hex format and I'd like to see support for the base format. I've recently had to implement my own package as I couldn't find anything outside of srec_cat that could read/write this format.

Did you have any plans to add support for Verilog hex? I think it would be fairly easy move your TI specific loading/dumping as wrappers around a Verilog hex loader/dumper.

My implementation is here: https://github.com/idex-biometrics/veriloghex/blob/main/veriloghex/veriloghex.py