graylinkim / sc2reader Goto Github PK

This project forked from ggtracker/sc2reader

A python library that extracts data from various Starcraft II resources to power tools and services for the SC2 community. Who doesn't want to hack on the games they play?

Home Page: http://sc2reader.readthedocs.org

License: MIT License

Python 97.50% C++ 2.50%

sc2reader's Introduction

What is sc2reader?

sc2reader is a python library for extracting information from various different Starcraft II resources. These resources currently include Replays, Maps, and Game Summaries; we have plans to add support for Battle.net profiles and would gladly accept adapters to the more entrenched SCII sites such as sc2ranks.

There is a pressing need in the SC2 community for better statistics, better analytics, better tools for organizing and searching replays. Better websites for sharing replays and hosting tournaments. These tools can't be created without first being able to open up Starcraft II game files and analyze the data within. Our goal is to give anyone and everyone the power to construct their own tools, do their own analysis, and hack on their own Starcraft II projects under the open MIT license.

Who Uses sc2reader?

sc2reader is currently powering:

Websites: ggtracker.com, gamereplays.org, sc2companion.com
Tools: The Core
Experiments: Midi Conversion

If you use sc2reader and you would like your tool, site, project, or implementation listed above, drop us a line on our mailing list or stop by our #sc2reader IRC channel and say hi!

Current Status

sc2reader is production ready at release and under active development on Github. It is currently capable of parsing varying levels of information out of the three primary resource types listed below. For a more detailed and exact description of the information that can be extracted please consult the documentation hosted on ReadTheDocs.

Replays

Replays can be parsed for the following general types of information:

Replay details (map, length, version, expansion, datetime, game type/speed, ...)
Player details (name, race, team, color, bnet url, win/loss, ...)
Message details (text, time, player, target, pings, ...)
Unit Selection and Hotkey (Control Group) events.
Resource Transfers and Requests (but not collection rate or unspent totals!)
Unfiltered Unit commands (attack, move, train, build, psi storm, etc)
Camera Movements for all players and observers.

Replays from release 2.0.8 on ward make additional state information available:

Unit states - creation time, positions, and deaths times
Player resource stats - collection rates/unspent totals
Player spending stats - resources spent and lost

Further game state information can be extracted from this raw information:

All unit selections and hotkey values for every frame of the game.
APM/EPM and its untold variations.
Supply counts, expansion timings, build orders, etc

We have data dictionaries in place for standard games that make unit meta data available. Unit meta data is currently limited to:

Costs - mineral, vespene, supply
Classification - army, building, worker

Additionally, abilities that create units/buildings have the built unit linked with the build time in game seconds.

Unfortunately this information IS NOT currently versioned and is only accurate for the latest builds of Starcraft. Versioned meta data support will be added in future releases.

Maps

Maps can be parsed for the following information:

Minimap and Icon images (tga format)
enUS localized Name, Author, Description, Website (if available)
Map Dimensions, Camera Bounds, Tileset
Player Slot data and Advanced Teams alliance/enemy data.

There is a lot more in here to be had for the adventurous.

Game Summaries

Game Summary files are downloaded by the client in order to allow you to view the game summary from your match history. Prior to 2.0.8 they were the only way to get the information from the summary screen. Since the 2.0.8 release you have been able to compute this information yourself from the replay files.

Thank you Prillan and Team Liquid for helping to decode this file.

Lobby Properties (game speed, game type, ...)
Player Information (Race, Team, Result, bnet info, ...)
Player Graphs & Stats (Army Graph, Income Graph, Avg Unspent Resources, ...)
URLs to map localization files and images
Player build orders up to 64 (real) actions

Parsing on these files is now production ready for those that can get them. See the Team Liquid thread for details on how to go about getting them.

Again, these files are generally unnecessary after the 2.0.8 release.

Basic Usage

..note:

For example scripts, checkout the docs and the `sc2reader.scripts`_ package on Github or in the source.

Loading Replays

For many users, the most basic commands will handle all of their needs:

import sc2reader
replay = sc2reader.load_replay('MyReplay', load_map=true)

This will load all replay data and fix GameHeart games. In some cases, you don't need the full extent of the replay data. You can use the load level option to limit replay loading and improve load times:

# Release version and game length info. Nothing else
sc2reader.load_replay('MyReplay.SC2Replay', load_level=0)

# Also loads game details: map, speed, time played, etc
sc2reader.load_replay('MyReplay.SC2Replay', load_level=1)

# Also loads players and chat events:
sc2reader.load_replay('MyReplay.SC2Replay', load_level=2)

# Also loads tracker events:
sc2reader.load_replay('MyReplay.SC2Replay', load_level=3)

# Also loads game events:
sc2reader.load_replay('MyReplay.SC2Replay', load_level=4)

If you want to load a collection of replays, you can use the plural form. Loading resources in this way returns a replay generator:

replays = sc2reader.load_replays('path/to/replay/directory')

Loading Maps

If you have a replay and want the map file as well, sc2reader can download the corresponding map file and load it in one of two ways:

replay = sc2reader.load_replay('MyReplay.SC2Replay', load_map=true)
replay.load_map()

If you are looking to only handle maps you can use the map specific load methods:

map = sc2reader.load_map('MyMap.SC2Map')
map = sc2reader.load_maps('path/to/maps/directory')

Using the Cache

If you are loading a lot of remote resources, you'll want to enable caching for sc2reader. Caching can be configured with the following environment variables:

SC2READER_CACHE_DIR - Enables caching to file at the specified directory.
SC2READER_CACHE_MAX_SIZE - Enables memory caching of resources with a maximum number of entries; not based on memory imprint!

You can set these from inside your script with the following code BEFORE importing the sc2reader module:

os.environ['SC2READER_CACHE_DIR'] = "path/to/local/cache"
os.environ['SC2READER_CACHE_MAX_SIZE'] = 100

# if you have imported sc2reader anywhere already this won't work
import sc2reader

Using Plugins

There are a growing number of community generated plugins that you can take advantage of in your project. See the article on Creating GameEngine Plugins for details on creating your own. To use these plugins you need to customize the game engine:

from sc2reader.engine.plugins import SelectionTracker, APMTracker
sc2reader.engine.register_plugin(SelectionTracker())
sc2reader.engine.register_plugin(APMTracker())

The new GameHeartNormalizerplugin is registered by default.

Installation

sc2reader runs on any system with Python 2.6+, 3.2+, or PyPy installed.

From PyPI (stable)

Install from the latest release on PyPI with pip:

pip install sc2reader

or easy_install:

easy_install sc2reader

or with setuptools (specify a valid x.x.x):

wget http://pypi.python.org/packages/source/s/sc2reader/sc2reader-x.x.x.tar.gz
tar -xzf sc2reader-x.x.x.tar.gz
cd sc2reader-x.x.x
python setup.py install

Releases to PyPi can be very delayed (sorry!), for the latest and greatest you are encouraged to install from Github master.

From Github

Github master is generally stable with development branches more unstable.

We use travis-ci to provide a record of our continuous testing and coveralls.io provides a record of our test coverage. Please verify that tests are passing before installing development versions.

Install from the latest source on Github with pip:

pip install -e git+git://github.com/GraylinKim/sc2reader#egg=sc2reader

or with setuptools:

wget -O sc2reader-master.tar.gz https://github.com/GraylinKim/sc2reader/tarball/master
tar -xzf sc2reader-master.tar.gz
cd sc2reader-master
python setup.py install

For Contributors

Contributors should install from an active git repository using setuptools in develop mode. This will install links to the live code so that local edits are available to external modules automatically:

git clone https://github.com/GraylinKim/sc2reader.git
cd sc2reader
python setup.py develop

Please review the CONTRIBUTING.md file and get in touch with us before doing too much work. It'll make everyone happier in the long run.

Testing

We use the built in unittest module for testing. If you are still on Python 2.6 you will need to install unittest2 because our test suite requires newer features than are included in the main distribution.

To run the tests just use:

python test_replays/test_all.py
python test_s2gs/test_all.py

When repeatedly running tests it can be very helpful to make sure you've set a local cache directory to prevent long fetch times from battle.net:

export SC2READER_CACHE_DIR=local_cache
# or
SC2READER_CACHE_DIR=local_cache python test_replays/test_all.py

Good luck, have fun!

Community

sc2reader has a small but growing community of people looking to make tools and websites with Starcraft II data. If that sounds like something you'd like to be a part of please join our underused mailing list and start a conversation or stop by #sc2reader on FreeNode and say 'Hi'. We have members from all over Europe, Australia, and the United States currently, so regardless of the time, you can probably find someone to talk to.

Issues and Support

We have an issue tracker on Github that all bug reports and feature requests should be directed to. We have a mailing list with Google Groups that you can use to reach out for support. We are generally on FreeNode in the #sc2reader and can generally provide live support and address issues there as well.

Acknowledgements

Thanks to all the awesome developers in the SC2 community that helped out and kept this project going.

Special thanks to the people of the awesome (but abandoned!) phpsc2replay project whose public documentation and source code made starting this library possible.
Thanks to sc2replay-csharp for setting us straight on game events parsing and assisting with our v1.5 upgrade.
Thanks to ggtracker.com for sponsoring further development and providing the thousands of test files used while adding s2gs and HotS support.
Thanks to Blizzard for supporting development of 3rd party tools and releasing their s2protocol full reference implementation.

sc2reader's People

Contributors

Stargazers

Watchers

sc2reader's Issues

parse error from linux

$ sc2printer '149414-nSkENNY-VS-aflubin.sc2replay'

Version 1.3.6.19269 replay:
149414-nSkENNY-VS-aflubin.sc2replay
Error parsing event Type=0x1, Code=0xb
Previous Event: selection
2C22AC00000115030101018F000001
Following Bytes:
0C220B04400120000C22AC000300011C00050602900000019100000192000001930000019400

I have tried with many replays but here is an example - http://www.sc-replay.com/download.php?id=149414

Any help will be appreciated

s2gs parsing always hits battle.net

The option to supply some kind of caching object would be important for large-scale usage.

Is there any info about replay.smartcam.events

I wanna to implement an application that make it easier to record and play observer's movement and voice. I think this part will be helpful. Is there any new result about smartcam?

1.3.3 Patch Issues

The patch breaks replay parsing. Thanks @grom358 for some sample replays:

http://drop.sc/packs/112?pass=d7ec0e29-4076-4eb7-930a-40e9b146e506

bad interpreter

$ sc2printer ~/Downloads/33777.SC2Replay
bash: /usr/local/bin/sc2printer: c:\Python27\python.exe^M: bad interpreter: No such file or directory

$ cat /usr/local/bin/sc2printer

#!c:\\Python27\python.exe

import os,sys
from sc2reader import Replay
from sc2reader.exceptions import ParseError

def doFile(filename):

    try:
        replay = Replay(filename)
        print "\nStarcraft II Version %s" % replay.releaseString
        print "%s on %s played on %s" % (replay.type,replay.map,replay.date)

        #Player[0] is None so that players can be indexed by ID
        for team,players in replay.teams.iteritems():
            print "\n\tTeam %s: %s" % (team,replay.results[team])
            for player in players:
                print "\t\t%s" % player

    except ParseError as e:
        prev = e.replay.events[-1]
        event = e.event
        bytes = e.event.bytes+e.bytes.peek(30)
        print "\nVersion %s replay:\n\t%s" % (e.replay.releaseString,e.replay.file)
        print "\tError parsing event Type=%s, Code=%s" % (hex(e.event.type),hex(e.event.code))
        print "\tPrevious Event: %s" % prev.name
        print "\t\t"+prev.bytes
        print "\tFollowing Bytes:"
        print "\t\t"+bytes


def doDir(dirname):
    for name in os.listdir(dirname):
        replaypath = os.path.join(dirname,name)

        if os.path.isfile(replaypath):
            doFile(replaypath)
        elif os.path.isdir(replaypath):
            doDir(replaypath)

assert len(sys.argv)>1, "A path to at least 1 sc2replay file is required"
for filename in sys.argv[1:]:
    if os.path.isfile(filename):
        doFile(filename)
    elif os.path.isdir(filename):
        doDir(filename)```

Parsing shouldn't die when there are deselect problems

There are at least two cases where we will unavoidably lose track of the Unit IDs in a selection group, and perhaps also the number of units in the selection group:

when a selected egg turns into unit(s)
when a drone turns into a building-under-construction

Because of these cases, we will sometimes receive deselection events that appear to be out-of-bounds, for example deselecting unit #15 when we thought there were only 11 units in the group.

When this happens, rather than freaking out, sc2reader should calmly note that there are selection problems with the given hotkey group, and create some kind of flag or count of the # of times a problem occurred with a given group.

This way, it is left up to the end user to decide how to react to such problems -- throw away the game entirely, ignore the problems, or something else.

sc2printer wrong magic

the magic in sc2reader is as follows:

#!c:\Python27\python.exe

this is not good, it should be:

#!/usr/bin/env python

possibly followed by:

coding: utf-8

or else it only works if you have python there installed :) which is not an issue on windows, but an issue on unix, where this directory does not exist. :)

Ability = None issue

{'is_local': True, 'code': 11, 'ability': None, 'is_init': False, 'is_camera_movement': False, 'frame': 37612, 'pid': 1, 'player': Player 1 - xxxxx (Terran), 'second': 2350, 'is_unknown': False, 'type': 1, 'is_player_action': True}

Replay ability set to None.

The replay testes can be downloaded here: http://www.sc2replayed.com/replays/176215-1v1-terran-taldarim-altar-le

Same is also the case for this:
http://www.sc2replayed.com/replays/176213-1v1-terran-protoss-backwater-gulch
which is another build.

winner_known flag not set.

The winner_known flag is never set any more.

Event Type 7

I have several that are failing because of an unknown event type being found -- please find me for one:

[for my reference later ] Desert Strike (4) -- 824

AttributeError: 'Player' object has no attribute 'chosen_race'

graylin@graylin-laptop:/home/sc2reader$ sc2printer 37114.SC2Replay

--------------------------------------
37114.SC2Replay

   Map:      The Shattered Temple
   Date:     2011-09-23 02:29:46
Traceback (most recent call last):
  File "/usr/local/bin/sc2printer", line 9, in <module>
    load_entry_point('sc2reader==0.2.0', 'console_scripts', 'sc2printer')()
  File "/home/sc2reader/sc2reader/scripts/sc2printer.py", line 92, in main
    doFile(file, arguments)
  File "/home/sc2reader/sc2reader/scripts/sc2printer.py", line 45, in doFile
    races.append(''.join([player.chosen_race[0] for player in team.players]))
AttributeError: 'Player' object has no attribute 'chosen_race'

A fix could be:

chosen_race => pick_race
actual_race => play_race

Dev Readme out of date

Branch new_data has significant changes to the top level functions of sc2reader. An updated readme needs to reflect the changes. May require new scripts?

Replay from latest SC2 fails to parse

Parsing this http://dl.dropbox.com/u/10271756/Season%202/Metalopolis.SC2Replay replay results in a "Unknown event: 0x0 - 0x9a at 0x15" exception being throw.

That replay was recently recorded with the latest build of StarCraft 2.

Unknown event: 0x4 - 0x1c at 0x1725d

I receive the following error:
TypeError Unknown event: 0x4 - 0x1c at 0x1725d
at sc2reader/parsers.py in getParser, line 221.

This replay I'm encountering the error on can be found here: http://www.sc2replayed.com/replays/127299-1v1-terran-protoss-metalopolis

I'm using the latest version from git.

s2gs processing can infinite loop sometimes

For example on this US one: 104eef5d50e73f17523225dd262e0b88b4949b108b3d79bba469fff670228770.s2gs

resources.py load_lobby_properties() has a while len(left_lobby) > 0 loop.
Nothing prevents it from looping infinitely, and for the given s2gs file, it does.

I don't understand exactly which assumption made by the author turned out not to be true.

'Wrap in Stalker'

A small typo: you have 'Wrap in Stalker (warp gate)' instead of 'Warp in Stalker (warp gate)' as an ability name.

s2gs parsing trouble with unknown data structures 22, 24, 26, 28

the s2gs hashes are private, will send them in separate message.

replay.date is wrong

I ran the PrettyPrinter example on two different replays. Both give the same date in 1600:

$ python2 test.py 2011-05-17\ 01-10-24\ ZvZ\ batrick\ (Z)\ vs\ guitarizt\ (Z)\ The\ Shattered\ Temple.SC2Replay

2011-05-17 01-10-24 ZvZ batrick (Z) vs guitarizt (Z) The Shattered Temple.SC2Replay

SC2 Version 1.3.3.18574
Ladder Game, 1600-12-31 20:00:00
1v1 on
Length: 20.50

Team 1: (Z) guitarizt

Team 2: (Z) batrick

$ python2 test.py Ulaan\ Deeps.SC2Replay

Ulaan Deeps.SC2Replay

SC2 Version 1.4.3.21029
Ladder Game, 1600-12-31 20:00:00
3v3 on
Length: 17.03

Team 1: (Z) batrick
(Z) irishGUARD
(T) Fuzzzy

Team 2: (T) Lonely
(P) silent
(P) Devin

BuildOrder event classification

build order ability events seem to be coming through as UnknownLocationAbilityEvents. They should be coming through as command card events.

Some research is required here.

map url problems (SEA?) 8.5% affected

This one is important: it kills parsing for 8.5% of our replay sample set (47 out of 549 attempted thus far), and if it is region-specific then it'd be a shame to lose all users in a particular region.

example: http://ggtracker.com/replays/204633/getreplay

error message:
URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>

traceback:
File "/Users/david/Dropbox/Programming/ggtracker_django/ggtracker/replays/views.py", line 25, in json_uploader
if replayPersister.upload_from_ruby(id, sender_subdomain):
File "/Users/david/Dropbox/Programming/ggtracker_django/ggtracker/replays/replay_persister.py", line 54, in upload_from_ruby
replay = sc2reader.load_replay(replaystringio, load_map=True)
File "/Users/david/Dropbox/Programming/ggtracker_django/venv/src/sc2reader/sc2reader/factories.py", line 65, in load_replay
return self.load(Replay, source, options, *_new_options)
File "/Users/david/Dropbox/Programming/ggtracker_django/venv/src/sc2reader/sc2reader/factories.py", line 113, in load
return self._load(cls, resource, filename=filename, options=options)
File "/Users/david/Dropbox/Programming/ggtracker_django/venv/src/sc2reader/sc2reader/factories.py", line 123, in _load
obj = cls(resource, filename=filename, *_options)
File "/Users/david/Dropbox/Programming/ggtracker_django/venv/src/sc2reader/sc2reader/resources.py", line 233, in init
self.load_map()
File "/Users/david/Dropbox/Programming/ggtracker_django/venv/src/sc2reader/sc2reader/resources.py", line 304, in load_map
map_file = StringIO(urllib2.urlopen(map_url).read())
File "/Users/david/local/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/Users/david/local/lib/python2.7/urllib2.py", line 394, in open
response = self._open(req, data)
File "/Users/david/local/lib/python2.7/urllib2.py", line 412, in _open
'_open', req)
File "/Users/david/local/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/Users/david/local/lib/python2.7/urllib2.py", line 1199, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/Users/david/local/lib/python2.7/urllib2.py", line 1174, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>

SC2Gears can parse the replay without any problem.

Possible typo in reader.py

phpsc2replay skips only one byte. Is this a typo? :

class GameEventsReader_19595
...
def right_click_target
...
buffer.skip(11)

Support non-Ladder maps

For instance footman wars. These maps seems to use a different set of events that haven't been accounted for yet.

parse failure when empty slots in match

I just had the scripts fail when making a game against ai on Slag Pits (2v2 map) and not setting it to 2v2 (so there was an empty slot on each team.

Callstack was:

utils.py: 397
processors.py: 43
init.py:159

Issue can be fixed by changing processors.py:39 from

if pid != 16:

if pid != 16 and pid in replay.person.keys():

This way there is no out of bounds call to replay.person

Add replay.winner field

Right now to find the winning players for a replay you need to

winning_team = filter(lambda t: t.result='Win', replay.teams)[0]
winning_players = winning_team.players

I'd be nice to have a short cut that looks like:

winning_players = replay.winner.players

Install problem

I recently installed Python 2.7, setuptools, sc2reader, mpyq but I get an error when running sc2reader. Any ideas what’s wrong?

sc2reader flourish$ sc2printer 'replays/1.SC2Replay'
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/bin/sc2printer", line 5, in
pkg_resources.run_script('sc2reader==0.2.0', 'sc2printer')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 489, in run_script
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1214, in run_script
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sc2reader-0.2.0-py2.7.egg/EGG-INFO/scripts/sc2printer", line 6, in

ImportError: cannot import name Replay

parse HotS beta replays

I can provide some if necessary

Two winners or two losers in 1v1

In the MLG replays, which unfortunately I can't share, there were two replays in which both players were deemed to have lost, and one replay where both players were deemed to have won.

The next step in resolving this issue is for me to look through ggtracker replays and see if I can identify the same issue in any of those. And those replays can be shared.

Two MissingAbility issues

27693 Tactictime Missing ability 0x21c00 from MetaData
27235 Tactictime Missing ability 0x12a40 from MetaData

For this replay

https://r-play.s3.amazonaws.com/replay_files/2012/04/30/Antiga_Shipyard_18.SC2Replay

retrieve the map from s2gs

Graylin says "I did find the information in one of the localization files. We should be able to pull it out and expose it"

Very exciting! This will make s2gs-only analysis that much more powerful.

Extract Build orders

I would really like to be able to extract the build orders of just buildings from the replays.

This would be similar to the description here:

https://groups.google.com/d/msg/sc2gears/FgtR6Qv8wug/xpmRWgOwrhEJ

The API you have created is excellent, clearly you have put a lot of work into it.
Though, most of Ability events are either still Unkown or incorrect like the "research_graviton_catapult".

I would assume mapping the abilities to the actions is a difficult process. Though sc2gears has done this.
Is this information public, to be reused?
I would like to help with this effort.

Cheers
Graham

Max Nickname Length

How long is the longest acceptable bnet handle?

Missing Abilities

I don't know what data files or exactly where to put this stuff

[2012-05-05 22:08:32,314: ERROR/PoolWorker-2] /Users/issackelly/Projects/python/replayswithfriends/replayswithfriends/replayswithfriends/packaged/media/replay_files/1/Cinder Fortress (6).SC2Replay
[2012-05-05 22:08:32,314: ERROR/PoolWorker-2] Release String: **1.4.3.21029**
[2012-05-05 22:08:32,314: ERROR/PoolWorker-2]   Player 1 - CapnCrunch (Terran)
[2012-05-05 22:08:32,315: ERROR/PoolWorker-2]   Player 2 - FrootRoop (Protoss)
[2012-05-05 22:08:32,315: ERROR/PoolWorker-2]   Player 3 - Tree (Terran)
[2012-05-05 22:08:32,315: ERROR/PoolWorker-2]   Player 4 - izak (Terran)
[2012-05-05 22:08:32,315: ERROR/PoolWorker-2]   Player 5 - Spaz (Protoss)
[2012-05-05 22:08:32,316: ERROR/PoolWorker-2]   Player 6 - Revyver (Terran)

Relevent Info Below

[2012-05-05 22:08:32,316: ERROR/PoolWorker-2] 18045 CapnCrunch  Missing ability 0x6f61 from MetaData

at 18:47 CapnCrunch removed one SCV from the queue of his PF

[2012-05-05 22:08:32,346: ERROR/PoolWorker-2] 23463 FrootRoop   Missing ability 0x3702 from MetaData

At 24:26 FrootRoop researchd Anion Pulse Crystals from his Fleet beacon

1.5 seems to still not work

Hey I installed sc2reader and was playing around with it, i couldn't get it to work with the 1.5 replay i tried

$ sc2printer test-replays/1v1-1.5.0-us.SC2Replay

--------------------------------------
test-replays/1v1-1.5.0-us.SC2Replay

Traceback (most recent call last):
  File "/usr/local/bin/sc2printer", line 8, in <module>
    load_entry_point('sc2reader==0.3.3', 'console_scripts', 'sc2printer')()
  File "build/bdist.macosx-10.7-intel/egg/sc2reader/scripts/sc2printer.py", line 93, in main
  File "build/bdist.macosx-10.7-intel/egg/sc2reader/scripts/sc2printer.py", line 15, in doFile
  File "build/bdist.macosx-10.7-intel/egg/sc2reader/factories.py", line 269, in load_replay
  File "build/bdist.macosx-10.7-intel/egg/sc2reader/resources.py", line 177, in read_data
  File "build/bdist.macosx-10.7-intel/egg/sc2reader/readers.py", line 265, in __call__
sc2reader.exceptions.ReadError: Unknown event code, Type: 0, Code: 28

replay in question can be found at http://drop.sc/234236

replay.details windows datetime magic number

The 116444735995904000 number in the replay.details article is the magic number used to convert windows datetime to epoch datetime.

Support for file objects

When I use Replay object (in sc2reader 0.2.0) or read_file (in current master), it supposes I give the file name. In some environments (like Google AppEngine) I might not have access to filesystem with a stored replay file, but have a file object (like BlobReader) instead

mpyq actually can accept file objects or file names.

Fix Format Strings

It seems like Python2.6 doesn't let you be lazy with:

"{}, {}, {}, {}".format(a,b,c,d)

We need to fill in numbers all over the place, or use a keyword format:

"{0},{1},{2},{3}".format(a,b,c,d)
"{a},{b},{c},{d}".format(a='a',b='b',c='c',d='d')

Ugh.

Proper time format

When time seconds is between 0 and 9, there should be a 0 before first integer

example:

2:2 should be 2:02

Match Lobby Properties to Attributes

The lobby properties from the s2gs files have id's that match id's found in the replay.attributes.events file from the replay. This link should be explored a bit more. The attribute id mappings for the replay files will probably be useful in making more sense of the lobby properties.

Perhaps something you could take a look at as well when you have time @Prillan, @dsjoerg.

Fix resource transfer events

The multiplier probably needs to be shifted down 4 bits. Thanks @tec27.

def read_resource(buffer):
    block = buffer.read_int(BIG_ENDIAN)
    base, multiplier, extension = block >> 8, (block & 0xF0) >> 4, block & 0x0F 
    return base*multiplier+extension

See: https://github.com/GraylinKim/sc2reader/blob/new_data/sc2reader/readers.py#L382

Update sc2printer

The sc2printer script hasn't been updated to reflect the attribute name changes of recent commits.

graylin@graylin-laptop:/home/sc2reader$ sc2printer 37114.SC2Replay

--------------------------------------
37114.SC2Replay

   Map:      The Shattered Temple
   Date:     2011-09-23 02:29:46
Traceback (most recent call last):
  File "/usr/local/bin/sc2printer", line 9, in <module>
    load_entry_point('sc2reader==0.2.0', 'console_scripts', 'sc2printer')()
  File "/home/sc2reader/sc2reader/scripts/sc2printer.py", line 92, in main
    doFile(file, arguments)
  File "/home/sc2reader/sc2reader/scripts/sc2printer.py", line 45, in doFile
    races.append(''.join([player.chosen_race[0] for player in team.players]))
AttributeError: 'Player' object has no attribute 'chosen_race'

Make it easier to report errors

Currently when sc2reader fails it throws an exception and people cry. sc2reader still needs to throw exceptions, but we can't be having people crying.

One way to fix this would be adding a sc2formatissue script that properly handles the exception and logs all relevant information so people can see what happend on a non-technical way. These logs could be posted with their issue to make support easier.

Other suggestions?

IndexError: list index out of range

replay file: http://drop.sc/41598/d

sc2printer test_replays/41598.SC2Replay 
--------------------------------------
test_replays/41598.SC2Replay

Traceback (most recent call last):
  File "/home/xxx/.virtualenvs/py27/bin/sc2printer", line 8, in <module>
    load_entry_point('sc2reader==0.2.0', 'console_scripts', 'sc2printer')()
  File "/home/xxx/projects/sc2reader/sc2reader/scripts/sc2printer.py", line 90, in main
    doFile(file, arguments)
  File "/home/xxx/projects/sc2reader/sc2reader/scripts/sc2printer.py", line 17, in doFile
    prev = e.game_events[-1]
IndexError: list index out of range

sc2reader.exceptions.MPQError

$ sc2printer ~/Downloads/37114.SC2Replay

replay: http://drop.sc/37114/d

--------------------------------------
/home/nopz/Downloads/37114.SC2Replay

Traceback (most recent call last):
  File "/usr/local/bin/sc2printer", line 9, in <module>
    load_entry_point('sc2reader==0.2.0', 'console_scripts', 'sc2printer')()
  File "/home/nopz/projects/sc2reader/sc2reader/scripts/sc2printer.py", line 92, in main
    doFile(file, arguments)
  File "/home/nopz/projects/sc2reader/sc2reader/scripts/sc2printer.py", line 15, in doFile
    replay = sc2reader.read_file(filename, debug=True)
  File "/home/nopz/projects/sc2reader/sc2reader/__init__.py", line 210, in read_file
    return __defaultReader.read_file(location, **user_options)
  File "/home/nopz/projects/sc2reader/sc2reader/__init__.py", line 192, in read_file
    replays = self.read(file_in, **user_options)
  File "/home/nopz/projects/sc2reader/sc2reader/__init__.py", line 90, in read
    replays.append(self.make_replay(replay_file, **options))
  File "/home/nopz/projects/sc2reader/sc2reader/__init__.py", line 139, in make_replay
    raise exceptions.MPQError("Unable to extract file: {0}".format(file))
sc2reader.exceptions.MPQError: Unable to extract file: replay.message.events

(github master)

Parse Player Data from custom maps.

low priority because it's a custom map, Marine Arena 1.0.0

Example: http://ggtracker.com/replays/205060/getreplay

Happened in 8 out of 145 replays, killing the parsing in those cases.

Six times it was this: ReadError: Event type 0x7 unknown at position 0x0., Type: 7
Once it was this: ReadError: Event type 0x7 unknown at position 0xe5., Type: 7
Once it was this: ReadError: Event type 0x2e unknown at position 0x235., Type: 46

sc2gears can't parse it either (http://ggtracker.com/replays/205060/getreplay)

Observer Support

Since observer messages and actions may be logged, we need to identify observers and create Player objects for them in the replay.observer(s) fields.

Clarify Timestamp documentation

The timestamp documentation isn't as clear as it could be and couple probably benefit from a renaming, timestamp is somewhat of a misnomer, and some better examples.

See this support thread for details.

'NoneType' object has no attribute 'split' on certain maps

Happens on several of my replays, including:
ggtracker.com/replays/205113/getreplay

guessing that

    game_strings = self.archive.read_file('enUS.SC2Data\LocalizedData\GameStrings.txt')

sometimes sets game_strings to None.

This causes sc2reader to die when parsing the replay.

sc2gears can't parse this one either.

2.2% of our replay sample can't parse because of this (12 out of 549).

rep.end_time
datetime.datetime(1601, 1, 1, 4, 0)

However, this other approach happens to work for this file:

datetime.utcfromtimestamp(utils.windows_to_unix(rep.raw_data['replay.details'].file_time))
datetime.datetime(2012, 5, 23, 3, 36, 19)

I don't want to just change the code that's in there now because I assume it's that way for a reason?