fjia30 / markovsoccergame Goto Github PK

This project doesn't seem to have an associated license. This makes it dangerous for people that want to use it (like me), because we don't know whether we can't do it freely or not.

Is this project open-source and free? I assume that you made it public here because you wanted to make it open-source (it's already open) and probably free too.

If yes, I would recommend the MIT license, because it's quite permissive but still requires people to give credit to you.

I am currently trying to play with your code (I only realised now that this project has no license and I assumed that it was open-source), so, if you want, I can add the MIT license to the project and then make a pull request with all my changes.

It seems that you assume that the state space has 8 * 8 * 2 states. We have a 2 * 4 = 8 grid, so one might think that there are 8 ways to place A, 8 ways to place B, and then there are 2 ways to place the ball. However, if you use this approach, you assume that A, B and the ball can be placed in the same cell. In the original Littman's paper (minimax Q-learning), this is not the case. A, B and the ball must always be in different cells - so, in that case, the correct number of states is 8 * 7 * 2 = 112.

I'm trying to understand why you defined the state space like self.state_space = (8, 8, 2), which seems to suggest that

1. you calculated the state space wrongly, or
2. maybe I am misinterpreting what the variable self.state_space is supposed to represent.
3. you allow players to be in the same cell

You write in the comments like this self.state_space: <num of variable1, num of variable2, num of variable3>, but this is unclear to me. You use state_space to define the Q-functions in the agent. Clearly, these should be represented as multi-dimensional arrays, such that each entry in the array corresponds to a tuple (a1, a2, state), so I think that makes sense.

Could you please clarify what is your approach to define the state space, and how does that affect e.g. the definition of the Q-function and its shape?