GLOBAL.BSA Layout about opentesarena HOT 6 CLOSED

Yep, I have the VFS parsing GLOBAL.BSA and can pull individual entries from it, as if they were files on disk. :) That's where the XMI files are for the MIDI music, so it'll be part of the pull request for music once I get it working (which is close).

from opentesarena.

You might just be able to mmap individual sections of the data files as well. At least for the act of any necessary parsing or conversion operations that would produce a result memory buffer from an input buffer anyway. Depends on how much of the original data is readily usable through modern API's, and whether or not its worth it to bind virtual address space to this data (even if its temporary as an alternative to reading the entire buffer in and doing whatever to produce a result) -- it would be pretty circumstantial.

from opentesarena.

Great. I'll be committing some small changes in a while, such as changing all <SDL2/SDL.h> includes to just "SDL.h" as that's what the CMake website suggests here (in the last paragraph).

mmap isn't compatible with Windows if I remember correctly.

from opentesarena.

Windows has an equivalent to mmap: https://github.com/DraconicEnt/KGE/blob/develop/components/support/platform/windows/file.cpp#L19

That code doesn't quite work (I haven't gotten the chance to fix it, but that should generally be right), but Windows does support the act of memory mapping files: https://msdn.microsoft.com/en-us/library/windows/desktop/aa366556(v=vs.85).aspx

In contrast: https://github.com/DraconicEnt/KGE/blob/develop/components/support/platform/unix/file.cpp#L19 (which does work)

I don't know if there's any restrictions on mapping multiple locations in a file, but I don't imagine there is any problems with mapping locations read only (as we don't want to write to the BSA's anyway).

I mention the is-it-worth-it because in plenty of cases it probably isn't much different than just reading the entire buffer into RAM and performing some operations on it, or even just reading portions at a time. It's just a thing to think about on a case by case basis.

from opentesarena.

I'm not too keen on mmap'ing it (through mmap or the Windows equivalent), personally. Although the individual files in the default BSA aren't very big, when mods can use their own BSAs it can potentially have pretty large files which would be wasteful to map into memory (e.g. FLACs, high quality video files).

Besides, the standard istream interface doesn't have a method to access the underlying memory map, so it'll still need to be read in through the read method anyway. I have a wrapper istream implementation that uses a normal ifstream on the real file, but constrains it to a portion of that file with a corrected offset (so 0 is the beginning of the given resource, and it won't read past the end of the resource).

from opentesarena.

I wasn't talking about mapping the entire BSA. Once we know where everything is we can mmap individual portions. The initial reading for this data is probably just reading header data. At that point we can decide what we want to mmap on a per file basis. Stuff like audio files we'd probably have taking up memory anyway, at least initially until decoded unless it's a streamable format. As I mentioned in my initial post, it really depends on specific circumstance whether or not this is worth it versus just reading the data normally.

For memory mapping, you'd have to have a straight file handle either way, so I don't really see the istream issue being in the way (obtain a read handle on the BSA and mmap whatever designated by our location data).

You'd just have to write a wrapper around Linux/Windows memory mapping if it was something that could be useful akin to what I was doing in my links above.

from opentesarena.