Feature Request: Support for user-defined character encodings about imhex HOT 8 CLOSED

Current progress: I just implemented support for loading thingy files and display a decoded version of the data next to the ASCII pane in the hex editor. Is this something like what you were looking for?

from imhex.

It's a very elegant solution! The way previous hex editors with table supports did it was to replace the text display, but it messed with the alignment. Using a different font was a good decision. Thank you very much for your work so far on the issue. This makes me happy beyond words.

Since you asked for feedback even though it's still a work in progress, I think the main things missing for now are:

• we can't actually edit the text in the third column (either direct keyboard input, or copy pasting UTF-8 formatted text from the clipboard), which other hex editors attempted to provide. (Yet?) This is the one thing left before it becomes "usable" for its intended use case. This includes highlighting/selecting text, etc. and identifying matching raw byte values.
• it would be nice to have the ability to customize the placeholders for unknown characters (byte values that have no matching character in the Thingy file) or unsupported characters (whenever you try to paste a character not supported by this encoding/Thingy table - most hex editors use "?" (0x3F) for this)

For now, getting the basics of this implementation would be rad.

Things that can wait for later include: how to deal with enabling MTE optimization for pasted text (potential for complicating clipboard handling), non-8-bit encodings (potential for complicating future text editing support), secondary encodings (ditto), and variables (not as much trouble, but still work to do).

from imhex.

This could be of use for implementing a fully fetured custome encoding:

http://transcorp.romhacking.net/scratchpad/Table%20File%20Format.txt

from imhex.

@OmbraRD That's exactly what I implemented a while ago already coming from the suggestions in this issue :)

from imhex.

Hi!
First of all, wow never in my life have I seen an issue with so much detail! You absolutely sparked my interested.

What's the things you'd like to see in InHex with those features? Simply a way to load custom encodings into ImHex for the string finder to pick them up and convert them into properly readable strings?

As for special characters, so far I tried to avoid using a custom font as it means shipping it next to the executable but it's probably a smart thing to do anyways, even more with features like you described it.

Thanks a lot!

from imhex.

I'm gonna use this issue to propose (as a backburner/future tag) some simple/limited implementation of bin.vis type of visualization:
http://binvis.io/

The entropy graph is good and useful, but these type of graphs do differ and represent data differently.

from imhex.

What's the things you'd like to see in InHex with those features? Simply a way to load custom encodings into ImHex for the string finder to pick them up and convert them into properly readable strings?

Thanks a lot for your interest. To answer your question, yes. These would be interesting:

1. Implementing character mapping support from an array rather than the default ASCII table used in the current hex editor. They will be used in the text field of the basic hex viewer window to display, edit (either in insert or overwrite modes), copy/paste from external sources... using those mappings rather than the defaults.

2. That array can be sourced from a conventional standard encoding, out of a list of conventional encodings (UTF-8, Shift-JIS, various code pages classified by languages...) OR from an external user-defined encoding in either of those three formats that can be imported and exported (and dare I suggest, edited.)

Considering how many options there is there it might be better to make this a separate dialog. Other hex editors used menus for this and it was horrifying to look at.

• toggle between standard ascii table/other standard encodings/use user-defined encoding
• choice between encodings (if other standard encodings) or options to import/export/edit user-defined encodings (if use user-defined encoding is used)
• big endian toggle
• if user-defined encoding, toggles to view/hide normal printable characters, DTE/MTE printable characters, printable characters interpreted using a table switch (so anything not from the first sub-table), normal control codes delimited with [], table switch control codes, line breaks identified with \n, and the end control code. (I left out interpreting binary data because it's too complex to get right). Note: Keep in mind if more than just normal printable characters is enabled, then lines can have more than 16 characters and highlighting will need to be adjusted accordingly.
• if user defined encoding, toggle to enable/disable DTE/MTE editing (for direct input and copy/pasting)
• display less than 16 bytes on a line in both hex fields and text field, and define how many (X). If this behavior is on, then whenever the user jumps to an offset the line would begin at that exact same offset rather than a multiple of X.
• display the rows 0-F in reverse order, only affecting the text field and its highlighting, to account for right-to-left encodings.

3. Small emoji/icons for non-character bytes to represent them in the text field visually (can be turned off):

• whitespace (notably the line break, which will be \n in the table files)
• undefined bytes in the character encoding
• one for control codes in general (mainly multi-table control codes, and in addition to that, other control codes/DTE/MTE if they were set to be hidden)
• one for the END control code

4. An info field for each byte that starts interpreting text from that byte according to the current user encoding until it hits either an END control code, or a failsafe max amount of characters that can be user defined.

This will display a string (with line breaks if they occur? support for color markdown?)
This will not ignore table entries that include values in binary (like in the text field in the hex editor), on the condition there's an explicit entry defining in binary an END control code.

I'm gonna use this issue to propose (as a backburner/future tag) some simple/limited implementation of bin.vis type of visualization:
http://binvis.io/
The entropy graph is good and useful, but these type of graphs do differ and represent data differently.

This reminds me of other tools (built on hex editors) that offer similar visualization.
They're often meant to visualize data structures known as "tile maps", in short a way to rearrange graphical data onscreen. A tile map is an array representing the entire screen or some subsection of it, and each cell has which graphical tile ID to use.

So these tilemap viewers (Djinn Tile Mapper, YY-CHR 2020 Mapper) would take a 16*16 tile bitmap image of the graphical memory, define that as "the encoding", and each byte in the file will be visualized as the bitmap from the single tile (often a 8*8 bitmap) whose ID matches this byte.
It's not very practical to add this to a hex editor though (there has to be some flexibility changing the number of cells per line to get the tile arrangements to display correctly) but I did see some use this as a replacement to hex editors for some out of the way text. But at some point, you're just making a specialized editor.

from imhex.

Support for binary encoding... eg. Dragon Quest IV 6-bit encoding

%000000=あ
%000001=い
%000010=う
%000011=え
%000100=お
%000101=か
%000110=き
%000111=く
%001000=け
%001001=こ
%001010=さ
%001011=し
%001100=す
%001101=せ
%001110=そ
%001111=た
%010000=ち
%010001=つ
%010010=て
%010011=と
%010100=な
%010101=に
%010110=ぬ
%010111=ね
%011000=の
%011001=は
%011010=ひ
%011011=ふ
%011100=へ
%011101=ほ
%011110=ま
%011111=み
%100000=む
%100001=め
%100010=も
%100011=や
%100100=ゆ
%100101=よ
%100110=ら
%100111=り
%101000=る
%101001=れ
%101010=ろ
%101011=わ
%101100=を
%101101=ん
%101110=っ
%101111=ゃ
%110000=ゅ
%110001=ょ

from imhex.