anno-mods / filedbreader Goto Github PK

View Code? Open in Web Editor NEW

14.0 2.0 5.0 18.44 MB

A command line tool for working with a proprietary bluebyte file compression used in Anno 2205 and 1800.

C# 100.00%

compression

filedbreader's Introduction

FileDBReader

A simple command line unpacker, repacker and interpreter for proprietary Anno 1800 compression.

Decompressing: Convert BlueByte's documents to XML, representing Data as HexStrings
Interpreting: Uses Xpath to select xmlNodes in documents and converts them.

After you are done editing the decompressed and interpreted xml file, you have to reinterpret it to hex (using the same interpreter) and compress it again.

Tip: you can also create xml patches and apply changes using meow's xmltest which works like the modloader.

Credits: First version of FileDB unpacking done by @VeraAtVersus, based on reverse engineering by @lysannschlegel

TLDR Usage

decompress -f <inputfiles> -c <CompressionVersion> -i <interpreterFile>
compress -f <inputFiles> -o <outputFileExtension> -c <CompressionVersion> -i <interpreterFile>
interpret -f <inputFiles> -i <interpreterFile>
toHex -f <inputFiles> -i <interpreterFile>
check_fileversion -f <inputfiles>
fctohex -f <inputfiles> -i <interpreterFile>
hextofc -f <inputfiles> -i <interpreterFile>

Note that i is optional on decompress, compress, fctohex and hextofc verbs. If provided, the program will directly convert from compressed to interpreted / from interpreted to recompressed.

The FileFormats folder contains a few more-or-less correct interpreter documents that have been collected over time. Don't let that stop you from creating your own. For this, refer to the Wiki.

AnnoMods.BBDom

This project also includes a library for use in your own projects. Documentation

To get started, simply add the Nuget Package to your project

Supported .NET versions: .NET 6.0 and above

Compression Versions

There are three versions of this compression

Version 1 is what you find in files up to Anno 1800, GU 12 (31.08.2021) -> documentation
Version 2 is used for new or updated files after this date -> documentation
Version 3 is Version 2 with extra steps, introduced shortly after Version 2 -> documentation

The compressor autodetects versions while decompressing. You can use the check_fileversion verb to determine yourself.

Binary Data in xml-based Anno files

Some xml-based file formats, especially in Anno 1701/1404/2070, do not use BlueByte's compression, but they still have binary parts:

<binary>CDATA[<bytesize><content>]</binary>

This tool can also convert those parts into:

<binary>CDATA[<hex_representation_of_content>]</binary>

Which also can be interpreted using an interpreter file. CDATA nodes must be marked in the interpreter with

<Convert Structure = "Cdata">

Bytesizes are automatically adjusted.

The most common example of this are .fc files for visual feedback.

To use this functionality, run

fctohex -f <inputfiles> -i <interpreterFile>

Invalid XML

Anno accepts </> as an xml closing tag. While reading, any of these closing tags are autocorrected, and since Anno also understands the valid xml syntax, you can use them ingame right away.

filedbreader's People

Contributors

Stargazers

Watchers

Forkers

jakobharder taludas shad0wlife supersandro2000 moidu21

filedbreader's Issues

Rewrite the serializer to be extensible

It's a clusterfuck. Needs cleanup.

Parameter for output filename

The current version is writing the output into the working directory and changes the extension.
This behavior is different to what 2.0 did before.
On top of that, if input and output file are the same by that logic the output gets changed to _fcexport.xml.
I'm not clear on all the cases, but it's confusing ;-)

I need a stable way to get the output filename. Ideally with my prefered target name.

Wrong Bytesize (data 12) Kap Trelawney

Hi want to decompress Sunken Treasure Map (a7t (gamedata.data))

I get


Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 1. The node is ignored for that reason.
Wrong Bytesize at allowedBuyers, Bytesize according to Interpreter: 4, Found in File: 78. The node is ignored for that reason.
Wrong Bytesize at allowedBuyers, Bytesize according to Interpreter: 4, Found in File: 78. The node is ignored for that reason.

I can fix it, but i dont know what is needed at all, e.g. Interpreter: 78 and at None.

maybe you can take a look :)

thx for this awesome tool! MP makes much more sense then before!!! :)))))

Wrong path in Island_Gamedata interpreter file

In the interpreter files Island_Gamedata.xml and Island_Gamedata_V2.xml this line is wrong (capital D):

<Convert Path ="//ObjectID" Type="Int64"/>

This results in "Wrong bytesize" messages.
It should be:

<Convert Path ="//ObjectId" Type="Int64"/>

Support ITuple types

All Tuple<..> and ValueTuple<..>, with the possible exception of tuples for n>=8 since they are implemented differently.

Sparse Grid Interpreter needs more work

Tagging @Atria1234 for this as well.

The values of the sparse grid don't get handled correctly. They aren't just a byte array, but for a Sparse Grid with 16x16 Tiles, the values tag contains 256 None tags, which are either empty (self closing) or contain a data element that is 4 bytes in size. This also means that that data cannot be interpreted as AreaID data because there each coordinate only has 2 bytes of data.

[Clickable Spoiler] Here is a sparse grid from the community island (data interpreted as 2x Int16)

      <VarMapData>
        <SparseEnabled>True</SparseEnabled>
        <x>320</x>
        <y>320</y>
        <block>
          <mode>1</mode>
          <x>16</x>
          <y>16</y>
          <default>
            <None />
          </default>
        </block>
        <block>
          <x>64</x>
          <y>80</y>
          <values>
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None>
              <data>65 0</data>
            </None>
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None>
              <data>3928 -16384</data>
            </None>
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None>
              <data>3929 -15360</data>
            </None>
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None>
              <data>3930 -14336</data>
            </None>
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None>
              <data>3931 -13312</data>
            </None>
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None>
              <data>66 0</data>
            </None>
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None>
              <data>66 0</data>
            </None>
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None>
              <data>193 0</data>
            </None>
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
            <None />
          </values>
        </block>
        <block>
          <mode>0</mode>
        </block>
      </VarMapData>

So generally the info about the mode is correct, just the actual data is not what we expected it to be. I don't know if it's 4 Bytes or 2 Shorts (like above), I'm just mostly certain it's not 1 Int32 😄

ArgumentException

Hi there,

just tried using the Savegame Visualizer, which threw an error relating to FileDBReader. So I tried running FileDBReader on its own, using:

FileDBReader decompress -f "..\test\Christian Jäger_S87.a7s"

which gave me the following error:

[READER]: Autodetected FileVersion = Version1
Unhandled exception. System.ArgumentException: An item with the same key has already been added. Key: 0
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at FileDBSerializing.FileDBParser_V1.ParseDictionary(Int32 Offset)
   at FileDBSerializing.FileDBParser_V1.ReadTagSection(Int32 OffsetToOffsets)
   at FileDBSerializing.DocumentParser`1.LoadFileDBDocument(Stream s)
   at FileDBReader.src.XmlRepresentation.Reader.Read_Version1(Stream input)
   at FileDBReader.src.XmlRepresentation.Reader.Read(Stream input)
   at FileDBReader.src.ToolFunctions.Decompress(IEnumerable`1 InputFiles, String InterpreterPath, Boolean overwrite, IEnumerable`1 ReplaceOps)
   at FileDBReader.CommandLineHandler.<>c__DisplayClass19_0.<Main>b__0(DecompressOptions o)
   at CommandLine.ParserResultExtensions.MapResult[T1,T2,T3,T4,T5,T6,T7,T8,T9,TResult](ParserResult`1 result, Func`2 parsedFunc1, Func`2 parsedFunc2, Func`2 parsedFunc3, Func`2 parsedFunc4, Func`2 parsedFunc5, Func`2 parsedFunc6, Func`2 parsedFunc7, Func`2 parsedFunc8, Func`2 parsedFunc9, Func`2 notParsedFunc)
   at FileDBReader.CommandLineHandler.Main(String[] args)

For what it's worth, the program auto-detects FileVersion 1, which seems odd, as I am using the latest version of Anno 1800.

Using the command

FileDBReader decompress -c 2 -f "..\test\Christian Jäger_S87.a7s"

does not do anything (at least not to my eyes), the error is still the same, I think.

I just tried several other save games, every one gives me the same error.

Am I doing something wrong? What's going on?

Don't overwrite files by default

Right now, output files are always just overwritten.
This should by changed to not overwrite by default, but give the user the option to set a flag to force overwriting.

Implement a precheck for the output file that terminates the program if the output file already exists
Implement a command line flag to force overwriting

Add FileFormat for accountdata.a7s

I tried decompressing and decoding Anno 1800's accountdata.a7s but my current knowledge is a bit lacking. Especially the nested <None/> list like looking structure gave me a hard time.

Can not unpack savegame data.a7s/data.bin

The unpacking from data.bin to xml simply does not continue after:

PS C:\Users\WDAGUtilityAccount\Desktop\FDB TEST> .\FileDBReader.exe decompress -f savegames/Scenario04/data.bin -i FileFormats/a7s_all.xml -y
[READER]: Autodetected FileVersion = Version3
FILEDB Deserialization took 394.4068 ms
FILEDB to XML conversion took: 1254.1961 ms
Started interpreting savegames/Scenario04/data.bin
[READER]: Autodetected FileVersion = Version3
FILEDB Deserialization took 85.1243 ms
FILEDB to XML conversion took: 767.4832 ms

it does nothing for minutes on a small savegame and taskmanager shows no workload. And the xml file stays empty.
Tested with FileDBReader.exe version 2.42 and 2.43.
The other files in savegame: meta, gamesetup and header, work fine by the way.

BUT:
With FileDBReader.exe 2.3 it continues and shows some errors with
Wrong Bytesize at None, Bytesize according to Interpreter: 4, Found in File: 8. The node is ignored for that reason.
and after they are skipped the xml file is successfully filled.

Python code based on AnnoSavegameVisualizer to reprocude from a savegame:

import zlib
import pathlib
import subprocess

path = pathlib.Path("savegames/Autosave 0.a7s")
out_path = pathlib.Path("savegames/rdaunpacked")
out_path.mkdir(parents=True, exist_ok=True) # create folder if not exist
subprocess.call(f"..\RDAConsole.exe extract -f \"{path}\" -o \"{out_path}\" -y -n")
files = list(out_path.glob("*.a7s"))
for file in files:
  content = open(file, 'rb').read()
  content = zlib.decompress(content)
  with open(file.with_suffix(".bin"), 'wb') as f:
      f.write(content)
  subprocess.call(f".\FileDBReader.exe decompress -i \"FileFormats/a7s_all.xml\" -f \"{str(file.with_suffix('.bin'))}\" -y")

QoL Improvement for UseEnum

Just specifying an enum node in a convert in an interpreter file should activate it's use, making the UseEnum argument irrelevant.

error with gamedata

when i try to decompress gamedata (data25.rda dlc12 map) i get this error

http://quentfr21.free.fr/error.png

thanks in advance

Edit: i'm on pre-release 2.3.99

`-o` option for all commands

E.g. hextofc doesn't have it yet.

ctt problem

hello i try to use the ctt.xml fileformat but it always fail with this

C:\anno 1800 modding\FileDBReadernew>FileDBReader.exe decompress -f 2x18.ctt -i fileformats/ctt.xml -y
[READER]: Autodetected FileVersion = Version1
Terminated conversion of 2x18.ctt because of an unknown Error.

how can i solve this ?

Support Encoding Attribute for Strings

EncodingAwareStrings are cool but Attributes are cooler.

Only do this after the serializaton rewrite