nickelc / hpk Goto Github PK
View Code? Open in Web Editor NEWHPK archiver for Haemimont Engine game files (Tropico 3-5, Omerta, Victor Vran, Surviving Mars, Jagged Alliance 3 etc.)
License: GNU General Public License v3.0
HPK archiver for Haemimont Engine game files (Tropico 3-5, Omerta, Victor Vran, Surviving Mars, Jagged Alliance 3 etc.)
License: GNU General Public License v3.0
Hello. I've decided to localize the new game Stranded: Alien Dawn.
But I ran into a problem (maybe I'm too dumb).
Unpacking proceeds normally, everything is decoded correctly.
However, I can't pack it back.
An access denied error pops up. What am I doing wrong?
d:\Works\Tools\hpk-v0.3.9-x86_64-pc-windows-msvc>hpk print "E:\SteamLibrary\steamapps\common\Stranded Alien Dawn\Local\English.hpk"
reading file: E:\SteamLibrary\steamapps\common\Stranded Alien Dawn\Local\English.hpk
header:
data_offset: 0x24
fragments_residual_offset: 0x0
fragments_residual_count: 0
fragments_per_file: 1
fragments_filesystem_offset: 0x5E138
fragments_filesystem_length: 24
filesystem entries: 3
filesystem fragments:
0x5E11F len: 25
0x5E10D len: 18
0x24 len: 385257
dir: index=1 depth=0 ""
fragment: 0x5E11F len: 25
dir: index=2 depth=1 "CurrentLanguage"
fragment: 0x5E10D len: 18
file: index=3 depth=2 "CurrentLanguage\\Game.csv"
fragment: 0x24 len: 385257
compressed: ZSTD inflated_length=1989815 chunk_size=131072 chunks=16
chunks: 0x4C len: 29351
0x72F3 len: 27035
0xDC8E len: 26581
0x14463 len: 25720
0x1A8DB len: 26760
0x21163 len: 23628
0x26DAF len: 20872
0x2BF37 len: 28986
0x33071 len: 29952
0x3A571 len: 26444
0x40CBD len: 17184
0x44FDD len: 21604
0x4A441 len: 25649
0x50872 len: 25496
0x56C0A len: 25503
0x5CFA9 len: 4416
Error message
d:\Works\Tools\hpk-v0.3.9-x86_64-pc-windows-msvc>hpk create D:\Works\Tools\hpk-v0.3.9-x86_64-pc-windows-msvc\pack D:\Works\Tools\hpk-v0.3.9-x86_64-pc-windows-msvc\extracted
Error: Hpk(Io(Os { code: 5, kind: PermissionDenied, message: "Access denied." }))
Here is the archive with localization English.zip
Trying to unpack savegame gives
Error: Hpk(Io(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }))
file itself attached
Hi! Thank you for providing hpk unpacker/archiver for Haemimont Engine game files.
I'm currently having a problem when archiving hpk files for localization (Korean language, edited from unpacked Tchinese.hpk file) fonts are missing in-game.
I first assumed it was font issue, but when I archived the unpacked Tchinese.hpk file without editing, and tested it in-game, same issue occurred.
Following is the link to 1)original Tchinese.hpk file, 2)rearchived Tchinese.hpk file from unpacked files
Could it be that I'm somehow doing the archiving process not properly? I've made sure when archiving, the directory order remained same (Tchinese/CurrentLanguage/Game.csv) and used bat file with command line .\hpk.exe create Tchinese Tchinese.hpk
Is there a command line sequence to allow me to inject files in a hpk package giving the path within the package?
Example Jagged Alliance 3 -
Say we have a folder C:\something\UI\Mercs unpacked from UI.hpk. I modify the files in \Mercs. I then have a couple of files, and I know their paths. Can I skip repacking the folder?
Down the line I want to create a bat to inject a selection of files into UI.hpk and distribute it with the modified files, but only the modified files. Is there a way to inject files with the .bat or do I have to run commands to unpack, overwrite and repack.
I looked through your documentation but I'm stuck.
Hello. Could you add support for the first templar game? So that we can extract models, textures, sounds, etc. Thanks!
I'm completely new to rust language and sorry for my weak English, so here is the last output:
$ cargo build --release
Compiling hpk v0.3.4 (/opt/git/hpk)
error: Could not compile `hpk`.
Caused by:
process didn't exit successfully: `rustc --edition=2018 --crate-name hpk src/main.rs
--color always --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C
metadata=efda9f278588f912 -C extra-filename=-efda9f278588f912 --out-dir
/opt/git/hpk/target/release/deps -L dependency=/opt/git/hpk/target/release/deps --extern
byteorder=/opt/git/hpk/target/release/deps/libbyteorder-8396a9ff3ced34b0.rlib --extern
clap=/opt/git/hpk/target/release/deps/libclap-6282fbebe1e3647f.rlib --extern
filetime=/opt/git/hpk/target/release/deps/libfiletime-0206700f28040b46.rlib --extern
flate2=/opt/git/hpk/target/release/deps/libflate2-cc230f0d2bf00371.rlib --extern
glob=/opt/git/hpk/target/release/deps/libglob-e086e0d80c192087.rlib --extern
hpk=/opt/git/hpk/target/release/deps/libhpk-edf78b3620d574a2.rlib --extern
lz4_compress=/opt/git/hpk/target/release/deps/liblz4_compress-cb84362fc1ac9e99.rlib
--extern nom=/opt/git/hpk/target/release/deps/libnom-2fb83814f174885f.rlib --extern
tempfile=/opt/git/hpk/target/release/deps/libtempfile-97e1e08034b9fee1.rlib --extern
walkdir=/opt/git/hpk/target/release/deps/libwalkdir-f5079254e823656c.rlib --extern
zstd=/opt/git/hpk/target/release/deps/libzstd-782b431503605148.rlib -L
native=/opt/git/hpk/target/release/build/miniz-sys-71c32bff269ebcb6/out -L
native=/opt/git/hpk/target/release/build/zstd-sys-ab6ec32720866ff2/out`
(signal: 11, SIGSEGV: invalid memory reference)
$
rustc --version
: 1.32.0
Linux AMD64 under Debian 7.3.0
I had to modify the cargo for zstd to 4.22+zstd.1.3.8 to get it to build.
when extracting the Surviving Mars .hpk files it appears to work correctly, but if you open any of the resulting files they aren't valid. For example if you open a .Lua file the text is unreadable. If you attempt to open a .tga it is not recognized as a valid picture file.
I downloaded this and ran it fresh today just to verify.
Steam version doesn't like fragmented files compressed with lz4 block format but has no problems with zlib.
what can i do?
pls check my terminal output:
Compiling hpk v0.3.2 (/home/grinsel/Downloads/hpk)
error: Edition 2018 is unstable and only available for nightly builds of rustc.
error: Could not compile hpk
.
To learn more, run the command again with --verbose.
grinsel@ding:~/Downloads/hpk$ cargo build --verbose
Compiling hpk v0.3.2 (/home/grinsel/Downloads/hpk)
Running rustc --edition=2018 --crate-name hpk src/hpk/mod.rs --color always --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=d9d205de6e274196 -C extra-filename=-d9d205de6e274196 --out-dir /home/grinsel/Downloads/hpk/target/debug/deps -C incremental=/home/grinsel/Downloads/hpk/target/debug/incremental -L dependency=/home/grinsel/Downloads/hpk/target/debug/deps --extern byteorder=/home/grinsel/Downloads/hpk/target/debug/deps/libbyteorder-988a048c1905d79a.rlib --extern clap=/home/grinsel/Downloads/hpk/target/debug/deps/libclap-1dd52620fc331773.rlib --extern filetime=/home/grinsel/Downloads/hpk/target/debug/deps/libfiletime-3cd8e69c8d1e50d4.rlib --extern flate2=/home/grinsel/Downloads/hpk/target/debug/deps/libflate2-a4dcf890805f6092.rlib --extern glob=/home/grinsel/Downloads/hpk/target/debug/deps/libglob-dfc6ffbad95a9e84.rlib --extern lz4_compress=/home/grinsel/Downloads/hpk/target/debug/deps/liblz4_compress-431422beb68ecad0.rlib --extern nom=/home/grinsel/Downloads/hpk/target/debug/deps/libnom-6b160a556ef528ca.rlib --extern tempfile=/home/grinsel/Downloads/hpk/target/debug/deps/libtempfile-5a6b6bc7620967b1.rlib --extern walkdir=/home/grinsel/Downloads/hpk/target/debug/deps/libwalkdir-4ad7e55a77bda48b.rlib --extern zstd=/home/grinsel/Downloads/hpk/target/debug/deps/libzstd-0fef2881ee8dcbd5.rlib -L native=/home/grinsel/Downloads/hpk/target/debug/build/miniz-sys-24ade8073a23fac7/out -L native=/home/grinsel/Downloads/hpk/target/debug/build/zstd-sys-e586e6dfbfa0abf4/out
error: Edition 2018 is unstable and only available for nightly builds of rustc.
error: Could not compile hpk
.
Caused by:
process didn't exit successfully: rustc --edition=2018 --crate-name hpk src/hpk/mod.rs --color always --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=d9d205de6e274196 -C extra-filename=-d9d205de6e274196 --out-dir /home/grinsel/Downloads/hpk/target/debug/deps -C incremental=/home/grinsel/Downloads/hpk/target/debug/incremental -L dependency=/home/grinsel/Downloads/hpk/target/debug/deps --extern byteorder=/home/grinsel/Downloads/hpk/target/debug/deps/libbyteorder-988a048c1905d79a.rlib --extern clap=/home/grinsel/Downloads/hpk/target/debug/deps/libclap-1dd52620fc331773.rlib --extern filetime=/home/grinsel/Downloads/hpk/target/debug/deps/libfiletime-3cd8e69c8d1e50d4.rlib --extern flate2=/home/grinsel/Downloads/hpk/target/debug/deps/libflate2-a4dcf890805f6092.rlib --extern glob=/home/grinsel/Downloads/hpk/target/debug/deps/libglob-dfc6ffbad95a9e84.rlib --extern lz4_compress=/home/grinsel/Downloads/hpk/target/debug/deps/liblz4_compress-431422beb68ecad0.rlib --extern nom=/home/grinsel/Downloads/hpk/target/debug/deps/libnom-6b160a556ef528ca.rlib --extern tempfile=/home/grinsel/Downloads/hpk/target/debug/deps/libtempfile-5a6b6bc7620967b1.rlib --extern walkdir=/home/grinsel/Downloads/hpk/target/debug/deps/libwalkdir-4ad7e55a77bda48b.rlib --extern zstd=/home/grinsel/Downloads/hpk/target/debug/deps/libzstd-0fef2881ee8dcbd5.rlib -L native=/home/grinsel/Downloads/hpk/target/debug/build/miniz-sys-24ade8073a23fac7/out -L native=/home/grinsel/Downloads/hpk/target/debug/build/zstd-sys-e586e6dfbfa0abf4/out
(exit code: 1)
I am trying to use hpk to mod Imperium Civitas (Glory of the Roman Empire). So far it has been easy to unpack the files, however I can't manage to repack them again and make it work.
This is how the "original" Packs/data.hpk
prints
file is compressed
header:
data_offset: 0x24
fragments_residual_offset: 0x2A299
fragments_residual_count: 49
fragments_per_file: 1
fragments_filesystem_offset: 0x20374
fragments_filesystem_length: 32824
filesystem entries: 4103
and this is what I get when printing the re-packaged data.hpk
file is compressed
header:
data_offset: 0x24
fragments_residual_offset: 0x0
fragments_residual_count: 0
fragments_per_file: 1
fragments_filesystem_offset: 0x1EE164
fragments_filesystem_length: 32824
filesystem entries: 4103
Is there an option I am missing that might be useful to solve this? I have tried playing with the --chunk-size
but only changed the fragments_filesystem_offset
Thanks for everything
Garbo
When extracting armstrong.hpk
and packing it again, the game refuses to load into the menu.
Looking at your file format specifications, it seems like something weird is going on:
The original file does have the data of the first file right after the one BPUL
header starting at 0x24.
The newly created file though does contain two BPUL
headers and then the data for the first file, which then starts later (0x48 I guess?)
here the first 200 bytes of each file:
original
4250554C2400000001000000FFFFFFFF000000000000000001000000CA267A0448050000444B4946000020005650383080073804905F0100010000000000000000000080C80D0000000000000000000030B7019D012A80073804004708858588858488020200061604F70681649F6BDB9B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B27387B
repacked
4250554C2400000001000000FF0000000000000000000000010000006EB07204480500004250554C2400000001000000FFFFFFFF0000000000000000010000008E763000580000004C5A34200E000000000002001000000072657475726E20313031303738324C5A34208E250A000000020024000000D36200003EB60000BA130100C06F010045D40100FF16734200000014000000456E7469746965732F41636164656D792E656E740D0A4D35000000181F000042446F6F7223006FF7310000002023000050466C
Also I noticed that the first header in the repacked archive does have 0x000000FF
instead of 0xFFFFFFFF
.
Any idea where this difference comes from and if that could be the cause of the problem?
EDIT:
when running it from terminal, the game seems to crash at loading the DLCs, which this file is part of.
.local\share\Steam\steamapps\common\Surviving Mars\CommonLua\Dlc.lua(702): field LoadDlcs
[string ""](-1): <>
--- end of stack
EDIT2:
I also noticed, that there is a size difference between the original and the repacked file (the bak is the original one):
Trying to use this tool to extract the lua files from Surviving Mars, however the output file seems to be corrupted. Attached is one of the extracted lua files.
Hi,
Thanks a lot for your work on the tool and especially putting format details into README, saved me a bunch of time figuring it out from the code.
I've noticed that mods uploaded to Steam Workshop via Surviving Mars Mod Editor are packed into .hpk, which can be extracted by the tool, but unpacked files are still compressed with ZSTD signature at the start.
Which are same fragmented files as lz4/zlib-compressed ones, just using zstd instead of these two.
Decompression script example:
https://github.com/mk-fg/games/blob/master/surviving-mars/zstd-dec.py#L39-L64
Not sure if it's also the case for other compression, but had to add "len(buff) != dst_len" check there as for e.g. .png files, despite ZSTD header, fragments are not compressed.
Attached ModContent.hpk.zip has ModContent.hpk for my simple mod, which has .lua and .png files with ZSTD header in it, both should be decompressible by python script above, but not currently handled by hpk tool itself, unfortunately.
Hi,
Thank you for writing this, it is very handy but:
I recently used the latest version of HPK with an extraction script to decompress JA3 but it seems there are a LOT of textures missing, just in the files I was looking at there were 175 missing textures.
Some of the .mtl files had all of their textures missing, there were no issues noticed during the decompress and I have tried it twice with no luck.
I am presently attempting to list and debug my way through the files doing hand extracts to see if I can find any thing that is blindly obvious as to why its happening but thought I should let you know while Im doing that.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.