bzip2 sublibrary (in src/bzip2 folder) isn't compiling on Linux when running make. I'm use StormLib as a static library in my project and that gives me a compiling error when I'm trying to build my project with static libstorm.a
/usr/local/lib/libstorm.a(SCompression.cpp.o): In function Compress_BZIP2(void*, int*, void*, int, int*, int)': SCompression.cpp:(.text+0x7cb): undefined reference toBZ2_bzCompressInit'
SCompression.cpp:(.text+0x814): undefined reference to BZ2_bzCompress' SCompression.cpp:(.text+0x831): undefined reference toBZ2_bzCompressEnd'
/usr/local/lib/libstorm.a(SCompression.cpp.o): In function Decompress_BZIP2(void*, int*, void*, int)': SCompression.cpp:(.text+0x888): undefined reference toBZ2_bzDecompressInit'
SCompression.cpp:(.text+0x8be): undefined reference to BZ2_bzDecompress' SCompression.cpp:(.text+0x8db): undefined reference toBZ2_bzDecompressEnd'
I made a workaround by adding ${ZLIB_BZIP2_FILES} in CMakeLists.txt, line 286.

I try build library with this command mingw32-make -f makefile.w32 and have several errors from linker ((.
Because you forgot add bzip2 library for building in makefile. And I stop trying. Please check makefile.w32. And OBJS_TEST = test/test.o not exist you rename that file. Thank you.

I already asked about it in Kanma/MPQExtractor#9.

Which method used for hashing the filenames?

but it is always great than sfmpq

With the current Makefile.linux compilation stops with the _ltc_mp symbol, defined in src/libtomcrypt/src/misc/crypt_ltc_mp_descriptor.c and used in a variety of files, being undefined. This can be solved by adjusting the use of ar rcs to ar rcS and adding in a ranlib -c libStorm.a command, as was suggested in this rather old mailing list archive. I made the slight change in this commit. I am fairly new to C, so I apologize if this is an isolated error on my own part.

Here's the info on my setup in case it's just a versioning problem.

Command

clang -Wall -lc++ -lbz2 -lz -stdlib=libc++ [path to libStorm.a] [main file with StormLib include] -o [output file]

clang -v

Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.3.0
Thread model: posix

ld -v

@(#)PROGRAM:ld  PROJECT:ld64-253.9
configured to support archs: armv6 armv7 armv7s arm64 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em (tvOS)
LTO support using: Apple LLVM 7.0.2 (clang-700.1.81)

Happy to supply any more info that's needed.

I was trying to use SFileAddFileEx(...) to add a file to my map. Even though SFileAddFileEx returns true (which should imply that GetLastError() == ERROR_SUCCESS), GetLastError() gives an error code of 2 (which on Mac/Linux is ERROR_FILE_NOT_FOUND).

• I opened the archive as BASE_PROVIDER_FILE
• I tried different flags for SFileAddFileEx. I'm currently using MPQ_FILE_REPLACEEXISTING, but I tested it with 0 and it still gives that error code.
• The file I'm adding exists on my hard drive, and it is successfully added to the mpq. It is listed if I use SFileFind.

I'm pretty lost. I can always just use SetLastError(0) after adding the file (and things seem to work okay), but that probably isn't the way to approach it. Does anyone else get this?

Here is the relevant code if anyone is interested:

-(BOOL)addFile:(NSString*)file archivePath:(NSString*)path flags:(unsigned int)flags compression:(unsigned int)compression error:(NSError **)errorPtr {
    const char *fileName = [file UTF8String];
    const char *archivedName = [path UTF8String];

    if (!SFileAddFileEx(_archive, fileName, archivedName, flags, compression, compression)) {
        *errorPtr = [MPQ getError];
        return NO;
    }
    //if (GetLastError() == ERROR_FILE_NOT_FOUND) {
    //    SetLastError(0);
    //}
    *errorPtr = nil;
    return YES;
}

NSString *fileToAdd = @"/Users/j/Downloads/Warcraft III Stuff/test.j";
    unsigned int flags = MPQ_FILE_REPLACEEXISTING;
    [mpq addFile:fileToAdd archivePath:@"test.j" flags:flags compression:MPQ_COMPRESSION_ZLIB error:&error];
    if (error != nil) { // error is nil
        NSLog(@"Add file fail.");
        [ViewController alert:error];
        [mpq close:&error];
        return;
    }
    NSLog(@"%d", GetLastError()); // prints '2'

Thanks for any and all help. Chances are that this is some silly mistake on my end, but I can't seem to resolve it.

Hi, your project is very great.
and can it support write Encrypted MPQ file that use a custom password? I think ti is a very necessary function

Раннее Караулов выложил проблему связанную с Somj защитой, написанной корейцами. На неё ему пожаловался я, а он поступил умнее и связался с вами, о чем я не подумал :D Если вам не сложно взгляните на систему защиты. Ссылка на файл http://dropmefiles.com/Kcgzp . По факту она дает проге открыть файл, но из-за искажения хекса не покажет файлы.

Previously Karaulow posted the issue about Somj protection, that was created by Koreans. I told him about it, and he asked you here, which I didn't think of :D. If it won't be problematic for you, please also take a look on another protection system. File URL http://dropmefiles.com/Kcgzp . It acts nearly in same way as somj, by damaging the header, but instead of just blocking the access it allows you to open it with the programm, but you can't see any files in it.

Спасибо за внимание!
Thanks for Attention!

This happens both with ladiks mpq editor and stormlib, the library only reads 870 bytes and the actual filesize is 97312, the mpq has update in the name I am not sure if that is affecting something, it also has the Incremental Patch File flag set.

Ok so if I assume that MPQ is a patch MPQ then I can test it with SFileIsPatchedArchive ?
Well testing it, it always returns false for all of the MPQ's. So something is clearly wrong here.

The game I am talking about here is Diablo 3.

How should I go about working with this?

Hello.

Is it possible to add support to read all information about MPQ's (.w3x on my situation), which protected by some modern (not really, software to crypt available since a middle of 2012) ways?

Here's a link to such of this 'highly-protected' map. http://mvpro.net/share/github/Life_v2_7d.w3x

MPQ Editor crashes when tries to read this archive (even with listfiles). I guess It can not find 'war3map.w3i' (and also war3map.j and scripts\war3map.j in some cases) in MPQ archive, because some other MPQ explorers said this :)

From this .w3x StormLib can read only map_size and map_info. Other options is undetected.

Unfortunely i am not sure with which of utility this map protected. Very very stupid suggestion is modified Spuzzler :)

Sincerely,

Alex.

Hi :)

int nResultError = ERROR_SUCCESS;
int nError;

1. Why you have additional nError var? Anyway nResultError is overwritten every time, when nError is not ERROR_SUCCESS, so why instead of:
   ...
   nError = SSignFileCreate(ha);
   if(nError != ERROR_SUCCESS)
   nResultError = nError;
   ...
   to not use just this:
   ...
   nResultError = SSignFileCreate
   ...
2. Why you do not check for nResultError == ERROR_SUCCESS in if's like:
   if(ha->dwFlags & MPQ_FLAG_LISTFILE_NEW)
   ..
   if(ha->dwFlags & MPQ_FLAG_ATTRIBUTES_NEW)
   ..
   if(ha->dwFlags & MPQ_FLAG_CHANGED)
   ..
   so return control to caller ASAP, so caller will have most first error code. In current case, caller will always have most last error.

Hello again, yes it's me with another mpq that is being processed way too slow. The reason for it is, due to 3 heades and a spazzler making MPQ Ediotr loop in file processing. So basically it can be opened, but it consumes up to 4 gb of Ram in order to continue checking files and nearly kills CPU with work.

To download the file click: http://dropmefiles.com/nCawx

With my 1.8 Ghz processor it took 30 minutes to process it half-way. ><''

I've factored BLIZZARD_WEAK_KEY modules as follows:

static const char * szBlizzardWeakPrivateKey =
    "-----BEGIN PRIVATE KEY-----"
    "MIIBOQIBAAJBAJJidwS/uILMBSO5DLGsBFknIXWWjQJe2kfdfEk3G/j66w4KkhZ1"
    "V61Rt4zLaMVCYpDun7FLwRjkMDSepO1q2DcCAwEAAQJANtiztVDMJh2hE1hjPDKy"
    "UmEJ9U/aN3gomuKOjbQbQ/bWWcM/WfhSVHmPqtqh/bQI2UXFr0rnXngeteZHLr/b"
    "8QIhAMuWriSKGMACw18/rVVfUrThs915odKBH1Alr3vMVVzZAiEAuBHPSQkgwcb6"
    "L4MWaiKuOzq08mSyNqPeN8oSy18q848CIHeMn+3s+eOmu7su1UYQl6yH7OrdBd1q"
    "3UxfFNEJiAbhAiAqxdCyOxHGlbM7aS3DOg3cq5ayoN2cvtV7h1R4t8OmVwIgF+5z"
    "/6vkzBUsZhd8Nwyis+MeQYH0rpFpMKdTlqmPF2Q="
    "-----END PRIVATE KEY-----";

And proved effective with Diablo II.
I made my modification to StormLib to support this weak signing feature but in a so ugly way that i would rather not submit this pull request. Did not have time to make it looks elegant.

So can we support this feature in a proper way?

Sorry for disturbing you again, but may I ask you a question?
Are you ever planning to share the source files of your MPQ Editor too?

Hi,

first of all, thanks for this great library, really usefull to collect information from Blizzard games :)

As Blizzard recently announced that tehy will move from MPQ to CASC file format for next wow version, I am wondering if you plan to also update you library to handle CASC files ?

Tahnks for your answer.

For packaging purposes, git tags should always be annotated, otherwise they cannot be used in git describe. Example:

[2:10:45] adys@azura ~/tmp/StormLib % git tag                 master:fd430b9 - StormLib 
v8.20
v8.21
v8.22
v9.00
[2:10:47] adys@azura ~/tmp/StormLib % git describe --always   master:fd430b9 - StormLib 
v8.22-43-gfd430b9

Basically reopening issue from #11

On windows there is no conflict because Stormlib uses direct call to WINAPI function but on Linux/BSD this is defined as a wrapper function to get errno (and the call comes from stormlib itself). If you have another class or library doing the same wrapping you get into a conflict described in the issue above.

The solution is to simply introduce StormGetLastError() which calls GetLastError() on windows and errno on linux/bsd.

Two questions coming to mind:

1. Is this an actual bug or should we simply fix things in other projects?
2. Is it worth breaking the "API" to fix this?

I guess I am seeking for an opinion rather than asking for a fix or doing a pull request.

Hello!

The issue that I have is, when I open MPQ it has filename9249119921, so when I want for example to export a model from MPQ, I either need to use programs such as:

MPQ Recover
RMPQeX
FileListGrabber

In order to get the listfile for it. Yes, they are used for warcraft 3 maps. So I thought of asking you to add a MPQ scanner utility like in one of those programs.

Здравствуйте!

Моя так сказать проблема заключается в том, что порой я открываю MPQ файлы и вместо их настоящего имени получаю file192491294129. Так как внутри MPQ был удален встроенный листфайл. Поэтому порой приходится использовать программы, что указаны выше.

Программы созданы для поиска имен файлов в картах варкрафта 3. Было бы круто если бы и в этой программе появилась функция сканирования файлов :)

Спасибо за внимание!
Thanks for attention!

MPQEditor 3.5.1.766

1. Open Hearthstone Base MPQ (base-win.mpq) with all updates in patch mode
2. Extract for example Hearthstone_Data\Managed\Assembly-Csharp.dll

Sorry if this the wrong platform for mpq editor bugs.

Sorry, still new to Github so I was afraid to do a pull request. Just letting you know that the file Makefile.mac showed some errors when I ran it through terminal:

cd $HOME/Downloads/StormLib
make -f Makefile.mac 

It gave the error:
Makefile.mac:87: *** missing separator (did you mean TAB instead of 8 spaces?). Stop.

I found out that those lines are spaced out (excerpt):

src/libtomcrypt/src/pk/asn1/der_length_object_identifier.obj \
src/libtomcrypt/src/pk/asn1/der_length_octet_string.obj \
src/libtomcrypt/src/pk/asn1/der_length_printable_string.obj \
src/libtomcrypt/src/pk/asn1/der_length_sequence.obj \
src/libtomcrypt/src/pk/asn1/der_length_short_integer.obj \

src/libtomcrypt/src/pk/asn1/der_length_utctime.obj \

src/libtomcrypt/src/pk/asn1/der_length_utf8_string.obj \

src/libtomcrypt/src/pk/asn1/der_sequence_free.obj \

src/libtomcrypt/src/pk/ecc/ltc_ecc_map.obj \

I got around the errors by escaping those lines, like so:

src/libtomcrypt/src/pk/asn1/der_length_object_identifier.obj \
src/libtomcrypt/src/pk/asn1/der_length_octet_string.obj \
src/libtomcrypt/src/pk/asn1/der_length_printable_string.obj \
src/libtomcrypt/src/pk/asn1/der_length_sequence.obj \
src/libtomcrypt/src/pk/asn1/der_length_short_integer.obj \
\
src/libtomcrypt/src/pk/asn1/der_length_utctime.obj \
\
src/libtomcrypt/src/pk/asn1/der_length_utf8_string.obj \
\
src/libtomcrypt/src/pk/asn1/der_sequence_free.obj \
\
src/libtomcrypt/src/pk/ecc/ltc_ecc_map.obj \

If there was another solution or if you don't get the error at all, then I apologize.

Thanks for the wonderful library.

I used CMake to Generatethe Project.
After CMake Generated,I built it With VS2013
But the "stromlib.dll" has no Export funtion.
Should i configure something Extra?

Hello.

Note: in text 'you' = 'any StormLib developer'.

I am trying to use StormLib 9.0 in application called Ghost++ [ official repo: https://code.google.com/p/ghostplusplus/ ] to better handle MPQ archives (maps for Warcraft III), primary to open protected maps with this application (it`s a Bot for Warcraft III) [ so i need at least 8.0 by this changelog https://github.com/stormlib/StormLib/blob/master/doc/History.txt ]. Previous version 6.25 and it works like a charm for a years. I don't know why i don't see '6.25' in history, maybe it was dev version.

But i have an error in compile process of Ghost++ application [i compiled StormLib and Ghost++ on this machine; i am using Linux Debian, used Makefile.linux file for compilation and i am using gcc version 4.7.2 (Debian 4.7.2-5) compiler]. I have such errors (same) either with 9.0 and current dev version.

Not sure how to fix. Maybe you can make backward compability to this function (if possible)? Or just suggest what should i change, please.

Errors:

In file included from ghost.cpp:27:0:
socket.h:83:13: error: previous declaration of ‘int GetLastError()’ with ‘C++’ linkage
In file included from ghost.cpp:50:0:
/usr/local/include/StormLib/StormLib.h:1104:20: error: conflicts with new declaration with ‘C’ linkage
ghost.cpp: In member function ‘void CGHost::ExtractScripts()’:
ghost.cpp:1564:69: error: too few arguments to function ‘bool SFileReadFile(HANDLE, void*, DWORD, LPDWORD, LPOVERLAPPED)’
In file included from ghost.cpp:50:0:
/usr/local/include/StormLib/StormLib.h:1031:15: note: declared here
ghost.cpp:1591:69: error: too few arguments to function ‘bool SFileReadFile(HANDLE, void*, DWORD, LPDWORD, LPOVERLAPPED)’
In file included from ghost.cpp:50:0:
/usr/local/include/StormLib/StormLib.h:1031:15: note: declared here

About first error:

socket.h:83:13: error: previous declaration of ‘int GetLastError()’ with ‘C++’ linkage
In file included from ghost.cpp:50:0:
/usr/local/include/StormLib/StormLib.h:1104:20: error: conflicts with new declaration with ‘C’ linkage

It looks like, that C++ have such function already and it can not be reused by StormLib... Maybe do something like

void  StormLib_SetLastError(int err);
int   Stormlib_GetLastError();

as a replace of

void  SetLastError(int err);
int   GetLastError();

Just guessing.

About second error:

ghost.cpp: In member function ‘void CGHost::ExtractScripts()’:
ghost.cpp:1564:69: error: too few arguments to function ‘bool SFileReadFile(HANDLE, void*, DWORD, LPDWORD, LPOVERLAPPED)’
In file included from ghost.cpp:50:0:
/usr/local/include/StormLib/StormLib.h:1031:15: note: declared here
ghost.cpp:1591:69: error: too few arguments to function ‘bool SFileReadFile(HANDLE, void*, DWORD, LPDWORD, LPOVERLAPPED)’
In file included from ghost.cpp:50:0:
/usr/local/include/StormLib/StormLib.h:1031:15: note: declared here

I compared lines from StormLib.h:

9.0:

bool   WINAPI SFileReadFile(HANDLE hFile, void * lpBuffer, DWORD dwToRead, LPDWORD pdwRead, LPOVERLAPPED lpOverlapped);

6.25:

BOOL  WINAPI SFileReadFile(HANDLE hFile, VOID * lpBuffer, DWORD dwToRead, DWORD * pdwRead = NULL, LPOVERLAPPED lpOverlapped = NULL);

As of my very tiny knowledges of C++ it looks like that you make two options 'pdwRead' and 'LPOVERLAPPED' mandatory. It it really must be added for function to work? Maybe change back to something like this?

bool   WINAPI SFileReadFile(HANDLE hFile, void * lpBuffer, DWORD dwToRead, LPDWORD pdwRead = NULL, LPOVERLAPPED lpOverlapped = NULL);

Lines from files mentioned in error output:
socket.h:83

extern int GetLastError( );

ghost.cpp:1564

if( SFileReadFile( SubFile, SubFileData, FileLength, &BytesRead ) )

ghost.cpp:1591

if( SFileReadFile( SubFile, SubFileData, FileLength, &BytesRead ) )

Full function ExtractScripts() with these two lines:

void CGHost :: ExtractScripts( )
{
    string PatchMPQFileName = m_Warcraft3Path + "War3Patch.mpq";
    HANDLE PatchMPQ;

    if( SFileOpenArchive( PatchMPQFileName.c_str( ), 0, MPQ_OPEN_FORCE_MPQ_V1, &PatchMPQ ) )
    {
        CONSOLE_Print( "[GHOST] loading MPQ file [" + PatchMPQFileName + "]" );
        HANDLE SubFile;

        // common.j

        if( SFileOpenFileEx( PatchMPQ, "Scripts\\common.j", 0, &SubFile ) )
        {
            uint32_t FileLength = SFileGetFileSize( SubFile, NULL );

            if( FileLength > 0 && FileLength != 0xFFFFFFFF )
            {
                char *SubFileData = new char[FileLength];
                DWORD BytesRead = 0;

                if( SFileReadFile( SubFile, SubFileData, FileLength, &BytesRead ) )
                {
                    CONSOLE_Print( "[GHOST] extracting Scripts\\common.j from MPQ file to [" + m_MapCFGPath + "common.j]" );
                    UTIL_FileWrite( m_MapCFGPath + "common.j", (unsigned char *)SubFileData, BytesRead );
                }
                else
                    CONSOLE_Print( "[GHOST] warning - unable to extract Scripts\\common.j from MPQ file" );

                delete [] SubFileData;
            }

            SFileCloseFile( SubFile );
        }
        else
            CONSOLE_Print( "[GHOST] couldn't find Scripts\\common.j in MPQ file" );

        // blizzard.j

        if( SFileOpenFileEx( PatchMPQ, "Scripts\\blizzard.j", 0, &SubFile ) )
        {
            uint32_t FileLength = SFileGetFileSize( SubFile, NULL );

            if( FileLength > 0 && FileLength != 0xFFFFFFFF )
            {
                char *SubFileData = new char[FileLength];
                DWORD BytesRead = 0;

                if( SFileReadFile( SubFile, SubFileData, FileLength, &BytesRead ) )
                {
                    CONSOLE_Print( "[GHOST] extracting Scripts\\blizzard.j from MPQ file to [" + m_MapCFGPath + "blizzard.j]" );
                    UTIL_FileWrite( m_MapCFGPath + "blizzard.j", (unsigned char *)SubFileData, BytesRead );
                }
                else
                    CONSOLE_Print( "[GHOST] warning - unable to extract Scripts\\blizzard.j from MPQ file" );

                delete [] SubFileData;
            }

            SFileCloseFile( SubFile );
        }
        else
            CONSOLE_Print( "[GHOST] couldn't find Scripts\\blizzard.j in MPQ file" );

        SFileCloseArchive( PatchMPQ );
    }
    else
        CONSOLE_Print( "[GHOST] warning - unable to load MPQ file [" + PatchMPQFileName + "] - error code " + UTIL_ToString( GetLastError( ) ) );
}

Thanks in advance,

Alex.

Currently function SFileOpenArchive search for MPQ header through the entire file. It would be useful for some scenarios to be able to set maximum offset for the search.

Currently If arbitrary large file is supplied to the function (e.g. DVD iso image or similar) it takes a great amount of time for function to complete. I've hardcoded limit for my app but I think it would be nice to have the ability to set such limit in code so each user can set personal value if needed and not hack library sources :).

Subj.

Back in November I sent you an email regarding this linker error and it appears that the fixed version from the website was removed with 9.00 release and the bug re-introduced. I assume because it was never patched on github and you forgot about it.

If you could find a way to finally solve this in git that would be great.

StormLib crashes when trying to close an archive after adding some files.
Backtrace:

Program received signal SIGSEGV, Segmentation fault.
#0  0x00007ffff7b6e423 in AllocateFileEntry (ha=0x602130, szFileName=0x7ffff7bc21c9 "(listfile)", lcLocale=0) at src/SBaseFileTable.cpp:1832
#1  0x00007ffff7b735e2 in SFileAddFile_Init (ha=0x602130, szFileName=0x7ffff7bc21c9 "(listfile)", FileTime=0, dwFileSize=72, lcLocale=0, dwFlags=2164261376, phf=0x7fffffffdeb8) at src/SFileAddFile.cpp:431
#2  0x00007ffff7b7a943 in SListFileSaveToMpq (ha=0x602130) at src/SFileListFile.cpp:464
#3  0x00007ffff7b7bc1c in SFileFlushArchive (hMpq=0x602130) at src/SFileOpenArchive.cpp:567
#4  0x00007ffff7b7bd44 in SFileCloseArchive (hMpq=0x602130) at src/SFileOpenArchive.cpp:628
#5  0x0000000000400f14 in main () at test.cc:150

test.cc: http://0wx.cat/53e798b22407d13dcf5ff96bd567dddc/
The image files I used for this test can be arbitrary files.

Issue exists since 68e2c25

SFileVerifyFile fail when specifying an empty file(in my case, a .txt file with no text inside it).

They only build via the Makefile, I'm assuming (based on the git history) that the Xcode project file has been neglected, while the rest of the project had advanced.

This PR (#49) fixes it.

Hi.
...
// Close the stream provider.
if(pStream->StreamClose != NULL)
pStream->StreamClose(pStream);

    // Also close base stream, if any
    else if(pStream->BaseClose != NULL)
        pStream->BaseClose(pStream);

...

Just curious about this part:
// Also close base stream, if any
else if(pStream->BaseClose != NULL)
Why here is "else if", not just "if"? There is only StreamClose or BaseClose are possible, not both (confused about "ALSO close..." part of comment, no question if it was "OR close...")?

Add parameter for pointer to a handle when calling SFileAddFileEx()

Something changed with your downloadable tar ball (the SHA1 changed) resulting in the package to compile no longer.
I'm running OSX and git HEAD compiles just fine while the downloadable 9.0 release doesn't.

The errors are like

StormLib-9.00/src/FileStream.cpp:202:35: error: no member named 'FilePos' in 'TFileStream'
        if(ByteOffset != pStream->FilePos)

indicating that this commit is not (completely) present in the downloadable archive:
6cd009b#diff-7c8a5705e396c3b5ae4b4fcb15344391L205
(It in fact, isn't)

To be able to locate libstorm at runtime on a Linux system, the dynamic linker cache needs to be updated using ldconfig after libstorm is installed. This is done by Makefile.linux, but when using CMake this step is not performed for some reason.

MPQEditor.exe /extract file.mpq will extract all files into the folder it was run in. How to make it preserve the internal directory structure? (it should do that by default)

Running MPQEditor /console then hitting enter for an empty command crashes. Apparently on some versions of windows it doesn't, but confirmed it on W10 x64

Blizzard has created a classic games division for the purpose of reviving its older games that use the mpq data format, have you considered applying and vying for linux support?
http://us.blizzard.com/en-us/company/careers/posting.html?id=15000XZ

Apart from the copyright notice, is there any license set for StormLib?

If not, could you add one? http://choosealicense.com/ offers quite a few good ones.

Not having a license makes it kind of hard to re-use StormLib in other Open Source projects.

For packaging purposes, it would be nice if we could run the tests with "make test" on any machine. Currently, paths are hardcoded to specific MPQs on a specific machine.

you have so many project file and makefile:

StormLib.xcodeproj\
Makefile.linux
Makefile.mac
makefile.w32
StormLib.vcproj
StormLib.vcxproj
StormLib.vcxproj.filters
StormLib_dll.vcproj
StormLib_dll.vcxproj
StormLib_dll.vcxproj.filters
StormLib_test.vcproj
StormLib_test.vcxproj
StormLib_test.vcxproj.filters
StormLib_v09.sln
StormLib_v11.sln

why not write a premake-4 or premake-5 file to generate these file automatic?

If you need, I'll write one to submit a PR.

Blizzard use it too.

Not really issue.

define MPQ_FILE_SIGNATURE 0x10000000 // Present on STANDARD.SNP(signature). The only occurence ever observed

Not present in any file except StormLib.h. May be it should be removed?

Also:

define MPQ_FILE_EXISTS 0x80000000 // Set if file exists, reset when the file was deleted

define MPQ_FILE_REPLACEEXISTING 0x80000000 // Replace when the file exist (SFileAddFile)

Both have same value. Is it ok?

I would like to ask for permission to submit stormlib into Debian repository.

I already have the CPack configuration to produce a working .deb in my tree. I just need to go through all the validation steps and then try to get a Debian mentor to accept it in. I can maintain the package myself but if the project owner would like to take care of it personally that's fine with me. Since the lib is stable I will just use 9.00a code.

If this works out I would also like to do the same thing for other distros and maybe FreeBSD.

Hello,

I tried to open an old map Escape Gay Space.w3x using the newest StormLib and it fails with:

aura++: /home/josko/projects/aura-bot/StormLib/src/SFileAttributes.cpp:129: DWORD CheckSizeOfAttributesFile(DWORD, DWORD, DWORD): Assertion `false' failed.
Aborted (core dumped)

It used to work with the old StormLib of 6.xx. I am unsure if it's caused by map protection or something else. The map can be found here:

http://www.epicwar.com/maps/2626/

Is it a cunning plan to use NULL instead of INVALID_HANDLE_VALUE?

This would make use of the handle parameter of the SFileSetXXXCallback, and move towards allowing multiple threads of the same process to use StormLib.

Hi.

I have an issue with some maps for Warcraft III, which as you know, is MPQ archives :)

The issue is that for some map's map_crc and map_sha1 calculated incorrectly, but there are no error reported! I have no idea why. But i am sure it's related to StormLib. Because with StormLib 6.25 it calculated values correctly.

I had few map's in past, but now i can provide only one map. Maybe i can find other ones later, if needed.

Map link: http://mvpro.net/share/github/CAT%20catch%20MOUSE%201.41v.w3x
(please don't ponder on the name, i am not an author of this map)

Here's a values calculated by StormLib 8.0, 9.0 and current Github dev (i am not sure, maybe issue stay also in 6.20-6.24 or something like it):

[MAP] calculated map_crc = 55 253 222 169
[MAP] calculated map_sha1 = 101 122 149 64 214 142 228 32 142 133 223 200 64 19 67 220 189 60 110 126

Full output of application (GHost++):

[MAP] loading MPQ file [../maps/CAT catch MOUSE 1.41v.w3x]
[MAP] calculated map_size = 197 178 21 0
[MAP] calculated map_info = 154 142 2 148
[MAP] calculated map_crc = 55 253 222 169
[MAP] calculated map_sha1 = 101 122 149 64 214 142 228 32 142 133 223 200 64 19 67 220 189 60 110 126
[MAP] calculated map_options = 96
[MAP] calculated map_width = 56 0
[MAP] calculated map_height = 62 0
[MAP] calculated map_numplayers = 12
[MAP] calculated map_numteams = 2
[MAP] calculated map_slot1 = 0 255 0 0 0 0 8 1 100
[MAP] calculated map_slot2 = 0 255 0 0 0 1 8 1 100
[MAP] calculated map_slot3 = 0 255 0 0 0 2 8 1 100
[MAP] calculated map_slot4 = 0 255 0 0 0 3 8 1 100
[MAP] calculated map_slot5 = 0 255 0 0 0 4 8 1 100
[MAP] calculated map_slot6 = 0 255 0 0 0 5 8 1 100
[MAP] calculated map_slot7 = 0 255 0 0 0 6 8 1 100
[MAP] calculated map_slot8 = 0 255 0 0 0 7 8 1 100
[MAP] calculated map_slot9 = 0 255 0 0 1 8 8 1 100
[MAP] calculated map_slot10 = 0 255 0 0 1 9 8 1 100
[MAP] calculated map_slot11 = 0 255 0 0 1 10 8 1 100
[MAP] calculated map_slot12 = 0 255 0 0 1 11 8 1 100

But the correct values is with StormLib 6.25. Nothing changed expect StormLib version:

[MAP] calculated map_crc = 60 51 205 13
[MAP] calculated map_sha1 = 207 81 172 186 6 32 239 51 249 60 101 79 140 41 148 90 90 8 49 176

Full:

[MAP] loading MPQ file [../maps/CAT catch MOUSE 1.41v.w3x]
[MAP] calculated map_size = 197 178 21 0
[MAP] calculated map_info = 154 142 2 148
[MAP] calculated map_crc = 60 51 205 13
[MAP] calculated map_sha1 = 207 81 172 186 6 32 239 51 249 60 101 79 140 41 148 90 90 8 49 176
[MAP] calculated map_options = 96
[MAP] calculated map_width = 56 0
[MAP] calculated map_height = 62 0
[MAP] calculated map_numplayers = 12
[MAP] calculated map_numteams = 2
[MAP] calculated map_slot1 = 0 255 0 0 0 0 8 1 100
[MAP] calculated map_slot2 = 0 255 0 0 0 1 8 1 100
[MAP] calculated map_slot3 = 0 255 0 0 0 2 8 1 100
[MAP] calculated map_slot4 = 0 255 0 0 0 3 8 1 100
[MAP] calculated map_slot5 = 0 255 0 0 0 4 8 1 100
[MAP] calculated map_slot6 = 0 255 0 0 0 5 8 1 100
[MAP] calculated map_slot7 = 0 255 0 0 0 6 8 1 100
[MAP] calculated map_slot8 = 0 255 0 0 0 7 8 1 100
[MAP] calculated map_slot9 = 0 255 0 0 1 8 8 1 100
[MAP] calculated map_slot10 = 0 255 0 0 1 9 8 1 100
[MAP] calculated map_slot11 = 0 255 0 0 1 10 8 1 100
[MAP] calculated map_slot12 = 0 255 0 0 1 11 8 1 100

I understand, is value correct or not, only in game. With the StormLib 6.25 here's no problem. With other version, i tested, issue (can't play), because GHost++ || Game kicks me (player) because of incorrect map_crc / map_sha1 values.

Code in application, which calculates map_crc and map_sha1:

    // try to calculate map_size, map_info, map_crc, map_sha1

    BYTEARRAY MapSize;
    BYTEARRAY MapInfo;
    BYTEARRAY MapCRC;
    BYTEARRAY MapSHA1;

    if( !m_MapData.empty( ) )
    {
        m_GHost->m_SHA->Reset( );

        // calculate map_size

        MapSize = UTIL_CreateByteArray( (uint32_t)m_MapData.size( ), false );
        CONSOLE_Print( "[MAP] calculated map_size = " + UTIL_ByteArrayToDecString( MapSize ) );

        // calculate map_info (this is actually the CRC)

        MapInfo = UTIL_CreateByteArray( (uint32_t)m_GHost->m_CRC->FullCRC( (unsigned char *)m_MapData.c_str( ), m_MapData.size( ) ), false );
        CONSOLE_Print( "[MAP] calculated map_info = " + UTIL_ByteArrayToDecString( MapInfo ) );

        // calculate map_crc (this is not the CRC) and map_sha1
        // a big thank you to Strilanc for figuring the map_crc algorithm out

        string CommonJ = UTIL_FileRead( m_GHost->m_MapCFGPath + "common.j" );

        if( CommonJ.empty( ) )
            CONSOLE_Print( "[MAP] unable to calculate map_crc/sha1 - unable to read file [" + m_GHost->m_MapCFGPath + "common.j]" );
        else
        {
            string BlizzardJ = UTIL_FileRead( m_GHost->m_MapCFGPath + "blizzard.j" );

            if( BlizzardJ.empty( ) )
                CONSOLE_Print( "[MAP] unable to calculate map_crc/sha1 - unable to read file [" + m_GHost->m_MapCFGPath + "blizzard.j]" );
            else
            {
                uint32_t Val = 0;

                // update: it's possible for maps to include their own copies of common.j and/or blizzard.j
                // this code now overrides the default copies if required

                bool OverrodeCommonJ = false;
                bool OverrodeBlizzardJ = false;

                if( MapMPQReady )
                {
                    HANDLE SubFile;

                    // override common.j

                    if( SFileOpenFileEx( MapMPQ, "Scripts\\common.j", 0, &SubFile ) )
                    {
                        uint32_t FileLength = SFileGetFileSize( SubFile, NULL );

                        if( FileLength > 0 && FileLength != 0xFFFFFFFF )
                        {
                            char *SubFileData = new char[FileLength];
                            DWORD BytesRead = 0;

                            if( SFileReadFile( SubFile, SubFileData, FileLength, &BytesRead, NULL ) )
                            {
                                CONSOLE_Print( "[MAP] overriding default common.j with map copy while calculating map_crc/sha1" );
                                OverrodeCommonJ = true;
                                Val = Val ^ XORRotateLeft( (unsigned char *)SubFileData, BytesRead );
                                m_GHost->m_SHA->Update( (unsigned char *)SubFileData, BytesRead );
                            }

                            delete [] SubFileData;
                        }

                        SFileCloseFile( SubFile );
                    }
                }

                if( !OverrodeCommonJ )
                {
                    Val = Val ^ XORRotateLeft( (unsigned char *)CommonJ.c_str( ), CommonJ.size( ) );
                    m_GHost->m_SHA->Update( (unsigned char *)CommonJ.c_str( ), CommonJ.size( ) );
                }

                if( MapMPQReady )
                {
                    HANDLE SubFile;

                    // override blizzard.j

                    if( SFileOpenFileEx( MapMPQ, "Scripts\\blizzard.j", 0, &SubFile ) )
                    {
                        uint32_t FileLength = SFileGetFileSize( SubFile, NULL );

                        if( FileLength > 0 && FileLength != 0xFFFFFFFF )
                        {
                            char *SubFileData = new char[FileLength];
                            DWORD BytesRead = 0;

                            if( SFileReadFile( SubFile, SubFileData, FileLength, &BytesRead, NULL ) )
                            {
                                CONSOLE_Print( "[MAP] overriding default blizzard.j with map copy while calculating map_crc/sha1" );
                                OverrodeBlizzardJ = true;
                                Val = Val ^ XORRotateLeft( (unsigned char *)SubFileData, BytesRead );
                                m_GHost->m_SHA->Update( (unsigned char *)SubFileData, BytesRead );
                            }

                            delete [] SubFileData;
                        }

                        SFileCloseFile( SubFile );
                    }
                }

                if( !OverrodeBlizzardJ )
                {
                    Val = Val ^ XORRotateLeft( (unsigned char *)BlizzardJ.c_str( ), BlizzardJ.size( ) );
                    m_GHost->m_SHA->Update( (unsigned char *)BlizzardJ.c_str( ), BlizzardJ.size( ) );
                }

                Val = ROTL( Val, 3 );
                Val = ROTL( Val ^ 0x03F1379E, 3 );
                m_GHost->m_SHA->Update( (unsigned char *)"\x9E\x37\xF1\x03", 4 );

                if( MapMPQReady )
                {
                    vector<string> FileList;
                    FileList.push_back( "war3map.j" );
                    FileList.push_back( "scripts\\war3map.j" );
                    FileList.push_back( "war3map.w3e" );
                    FileList.push_back( "war3map.wpm" );
                    FileList.push_back( "war3map.doo" );
                    FileList.push_back( "war3map.w3u" );
                    FileList.push_back( "war3map.w3b" );
                    FileList.push_back( "war3map.w3d" );
                    FileList.push_back( "war3map.w3a" );
                    FileList.push_back( "war3map.w3q" );
                    bool FoundScript = false;

                    for( vector<string> :: iterator i = FileList.begin( ); i != FileList.end( ); ++i )
                    {
                        // don't use scripts\war3map.j if we've already used war3map.j (yes, some maps have both but only war3map.j is used)

                        if( FoundScript && *i == "scripts\\war3map.j" )
                            continue;

                        HANDLE SubFile;

                        if( SFileOpenFileEx( MapMPQ, (*i).c_str( ), 0, &SubFile ) )
                        {
                            uint32_t FileLength = SFileGetFileSize( SubFile, NULL );

                            if( FileLength > 0 && FileLength != 0xFFFFFFFF )
                            {
                                char *SubFileData = new char[FileLength];
                                DWORD BytesRead = 0;

                                if( SFileReadFile( SubFile, SubFileData, FileLength, &BytesRead, NULL ) )
                                {
                                    if( *i == "war3map.j" || *i == "scripts\\war3map.j" )
                                        FoundScript = true;

                                    Val = ROTL( Val ^ XORRotateLeft( (unsigned char *)SubFileData, BytesRead ), 3 );
                                    m_GHost->m_SHA->Update( (unsigned char *)SubFileData, BytesRead );
                                    // DEBUG_Print( "*** found: " + *i );
                                }

                                delete [] SubFileData;
                            }

                            SFileCloseFile( SubFile );
                        }
                        else
                        {
                            // DEBUG_Print( "*** not found: " + *i );
                        }
                    }

                    if( !FoundScript )
                        CONSOLE_Print( "[MAP] couldn't find war3map.j or scripts\\war3map.j in MPQ file, calculated map_crc/sha1 is probably wrong" );

                    MapCRC = UTIL_CreateByteArray( Val, false );
                    CONSOLE_Print( "[MAP] calculated map_crc = " + UTIL_ByteArrayToDecString( MapCRC ) );

                    m_GHost->m_SHA->Final( );
                    unsigned char SHA1[20];
                    memset( SHA1, 0, sizeof( unsigned char ) * 20 );
                    m_GHost->m_SHA->GetHash( SHA1 );
                    MapSHA1 = UTIL_CreateByteArray( SHA1, 20 );
                    CONSOLE_Print( "[MAP] calculated map_sha1 = " + UTIL_ByteArrayToDecString( MapSHA1 ) );
                }
                else
                    CONSOLE_Print( "[MAP] unable to calculate map_crc/sha1 - map MPQ file not loaded" );
            }
        }
    }
    else
        CONSOLE_Print( "[MAP] no map data available, using config file for map_size, map_info, map_crc, map_sha1" );

I don't understand why is it happens and is it possible to fix it by StormLib developers in new versions.

Please try to investigate to the issue.

Here's zipped StormLib which shows correct values (6.25): http://mvpro.net/share/github/StormLib-6.25.zip

Thanks in advance,

Alex.

When extracting the files inside this patched MPQ, stormlib crashes:

adys@azura ~/HSB % sha256sum 3388.direct/base-Win.MPQ 3388.direct/Updates/hs-0-3604-Win-final.MPQ
3e194fd8b4642887c8ef043bcbe220f6375a3a1f5720bcb38fa9be6c77308ff9  3388.direct/base-Win.MPQ
bb24c23cda87f6bedd2fda22b37378decbcb78983f41aa2d3028ffcce47d2561  3388.direct/Updates/hs-0-3604-Win-final.MPQ

I think the patch chain is wrong (I can't figure out on top of what build that patch mpq is supposed to be), but stormlib shouldnt crash.

Blz has been using the new CASC file system in WoD Beta and Heroes of the Storms, its format can be found here (http://www.ownedcore.com/forums/world-of-warcraft/world-of-warcraft-model-editing/471104-analysis-of-casc-filesystem.html) .

i compile stormlib use msvc. compile cpp file too slow.
other compiler already supports pch.
and primary cpp file write this:

#define __STORMLIB_SELF__
#include "StormLib.h"
#include "StormCommon.h"

it's good! not other header file. you already use "pch".
how about to add a file, named like: StormPrehead.h

#include "StormPrehead.h"

replace this code.

#define __STORMLIB_SELF__
#include "StormLib.h"
#include "StormCommon.h"

close pch compile option, it's not influence.

but problem is other library cannot add this file.
I create third party static library to solve it.

there seem to be around 50 issues that have been detected by a free online coverity scan.
if you dont want to set it up i can send you an invite i guess so you can look at the issues
im still very new to the project so i doubt any fixes coming from me would be that good

http://www.coverity.com/products/coverity-scan/

Hi,
I'm using your lib for creating a tool with it and Qt.
My UI use the Qt types like the qstring for storing file names and i use this method for converting a Qstring to TCHAR* :
TCHAR* tname = (TCHAR*)name.toLatin1().data();

Stangely this way works 1/2 :(
And when this is'nt working the lib create an archive with a name like "a" and use it normally or sometimes the lib throw the error 87 (bad arguments) or the error 5 (access denied) or the error 123 (bad volume name)

The macro for deprecation creates a variable once for each translation unit when included from C. This results in multiply defined symbols, which in turn means errors during linkage on my compiler (gcc 4.8.1, mingw on windows). I think the easiest way to get consistent behavior would be to explicitly mark the const variable that STORMLIB_DEPRECATED generates as static to give it internal linkage. In C++, const variables have internal linkage by default, but in C they do not. This fixes the errors.

Minimal test case:
A.c: #include <StormLib.h>
B.c: #include <StormLib.h>
$ gcc A.c B.c -shared
B.o:B.c:(.rdata+0x0): multiple definition of 'STREAM_PROVIDER_LINEAR'
A.o:A.c:(.rdata+0x0): first defined here
B.o:B.c:(.rdata+0x4): multiple definition of 'STREAM_PROVIDER_ENCRYPTED'
A.o:A.c:(.rdata+0x4): first defined here
etc.