Why

As a

User of pyCMO

I want

to be able to easily train RL agents in pyCMO

So that

I can develop RL agents that can solve pyCMO scenarios

Acceptance Criteria

Given

that it is currently unclear how we can train RL agents in pyCMO

When

we convert pyCMO into an OpenAI Gym environment (or at least make an interface that will make pyCMO behave like a Gym environment

Then

we open up pyCMO to more collaboration and we can leverage a vast repository of RL agents that are compatible with Gym environments

Notes

• OpenAI Baselines is a set of high-quality implementations of reinforcement learning algorithms.
• Gymnasium is an open source Python library for developing and comparing reinforcement learning algorithms by providing a standard API to communicate between learning algorithms and environments

Why

As a

user of pyCMO

I want

to be able to use more actions in the game

So that

I can train agents that use a wider variety of actions

Acceptance Criteria

Given

we currently only support 8 actions (i.e. no action, launch aircraft, attack contact, refuel, rtb, etc.) in AvailableFunctions

When

we add more actions to AvailableFunctions

Then

we can have more actions to play with

Notes

Reference Command Lua Docs

We should

clean up and optimize the floridistan scenario

Because

the scenario currently takes a long time to complete due to the distance between the target and the attacker. In addition, we have no rewards baked into the scenario

Complexity

Easy

Definition of Done

• floridistan now has a reward system (i.e. for destroying the target, or penalization when an aircraft is lost)
• floridistan takes a shorter amount of time to complete (reduce transit times)
• floridistan has a time limit that is shorter than 24 hours

We should

make the data types that are exported in the features object consistent (from the Lua library that outputs the observations to xml to the FeaturesFromSteam class that parses it)

Because

currently we do not have a consistent agreement of the data types of each field (e.g. an aircraft's heading is a string rather than a float)

Complexity

Easy to medium.

Definition of Done

1. Change the namedtuple objects in features.py to NamedTuple from typing to add typing.
2. Make data types consistent for fields in features.py and pycmo_lib.lua and fix any tests or demos that might be broke.
3. Potentially change SteamClientProps from a data class to a NamedTuple class.

We should

refactor actions.py to handle the Steam edition of Command

Because

currently actions.py is written to be only compatible with the premium edition of Command, which requires an expensive license to which many users do not have access

Complexity

Easy to medium. We just need to re-write some of the Lua functions that gets returned.

Definition of Done

Users can get a list of valid actions per scenario.

Hi,

First of all, thanks for sharing the code in Github. Before I purchase the game and try out your code, I would like to ask a question.

In the config file, you used the Professional edition. Is this required? I wonder if Steam edition is also supported.

Thanks.

Why

As a

I want

So that

Acceptance Criteria

Given

When

Then

Notes

We should

improve the reliability of protocol.py when it executes scripts that are meant to click on and close windows (e.g. closeScenarioEndMessage.bat and MoveMouseEnterScenario.ps1)

Because

we currently rely on timing (i.e. sleep) in the scripts themselves to ensure correct timing. This is not always going to be reliable per the discussion in this comment. We need to create a module in protocol.py that will check the existence of the window to be closed, and restart the script to close the window as necessary.

Complexity

Easy to medium

Definition of Done

We have more certainty that when we run scripts that close windows they actually finish and can verify that the windows have been closed

Why

As a

user of pyCMO

I want

to be able to work with the Steam/commercial version of Command

So that

I save tens of thousands of dollars in license fees

Acceptance Criteria

Given

pyCMO currently only supports the Premium edition of Command

When

we research alternative ways to send agent commands to the Steam/commercial version of Command and implement this functionality

Then

pyCMO will be compatible with the Steam/commercial version of Command

Notes

Notes from past related issues

• You need the Lua function ScenEdit_ExportScenarioToXML() to output observations. The only alternative is to write your own Lua script and output data using the io module, but this might be cumbersome.
• I actually do not remember if you can use the io module in the non-Premium version. You should try it out in the console, but I believe the answer is no. Without io or some of the features from the Premium version, you cannot send commands (actions) into the game to take actions.
• Currently we send actions to the environment/game using a TCP/IP port, a feature only available in the Premium version.
• In order to output data (observations) to your local system, a Profession Edition of Command is required. I will do further experimentation with the commercial Steam version but it is my understanding that you cannot output files. I also do not think that you can send commands to the game because pycmo sends it through an open TCP/IP socket and you can only open that with a Professional version I think.

Why

As a

user of Command: Professional Edition (CPE)

I want

the latest versions of pyCMO to still work with CPE

So that

I can use pyCMO with CPE

Acceptance Criteria

Given

we made a lot of changes from pyCMO v1,1.0 onwards to support the Steam version of Command

When

we validate that pyCMO still works with CPE

Then

we can ensure that CPE users can also use pyCMO

Notes

Blocked until we have access to CPE

Describe the bug
MoveMouseEnterScenario.ps1 does not work on computers with a screen resolution smaller than 1920x1080 (e.g. a laptop). Thus, we cannot enter the scenario when we restart

To Reproduce
Steps to reproduce the behavior:

1. Run PyCMO on a laptop with a screen size smaller than 1920x1080.
2. See error

Expected behavior
We can use the MoveMouseEnterScenario.ps1 script on any system (regardless of screen size) to enter the scenario.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
As a temporary fix, users can edit lines 78-79 of the MoveMouseEnterScenario.ps1 script:

$MOVEMENTSIZEY = 600
$MOVEMENTSIZEX = 500

These values are the xy distance from the top of the "Side selection and briefing" to the "Enter scenario" button.

We should

simplify the code needed to create demos

Because

we currently have three demos and they share a lot of repeated code

Complexity

Easy to medium. We just need to move all the repeated code into a module like run_loop. We are basically simplifying the process of creating a training loop. Currently this is only used for demos but in the future this is useful for actually training agents.

Definition of Done

We have a module that can be called to set up and run demos

We should

develop a CMO scenario that can be used to train and demonstrate a reinforcement learning agent

Because

we currently do not have a demonstration of how pyCMO can be used to train an RL agent

Complexity

Creating the scenario should not be hard, but developing one that would be tractable to train an agent in while also demonstrating the capabilities of the framework might be challenging

Definition of Done

We have a scenario that we can use to train and test RL agents

Notes

• One idea is to replicate AFSIM's floridistan scenario
• A/G scenario with bomber defeating SAMs

When the Client sends data to the game, program execution will be frozen if the game raises an error due to poorly formatted Lua code. Thus, the scenario will not be able to move forward.

See CMOENV.step()

We should

refactor pycmo to handle both Premium and Steam editions of Command

Because

currently pycmo are written to be only compatible with the premium edition of Command, which requires an expensive license to which many users do not have access

Complexity

Medium to hard. We can proceed in two phases of work. First, we can refactor the current codebase and note pieces of code that are exclusive to the premium edition of the game. We can then add the codebase to support the Steam edition of the game.

Definition of Done

Users are able to specify and use pycmo for both the premium and Steam editions of Command. More emphasis will be placed on the latter because we do not have access to the premium version of the game (and this codebase has already been developed previously).

I am playing with reference to your pycmo.

About the random agent, the number of steps cannot exceed 50, Have you tried solving this problem?
(and the reason for this is believed to be an issue with storing up to 200 variables on the main function in Lua.)

Do you have experience performing more than 50 steps?

In the case of Q-learning agent, it appears to be a Q-learning algorithm rather than a code for CPE, so I would like to ask if you are interested in developing ML Agent.

Why

As a

user of pyCMO

I want

to be able to check when a scenario is running or paused

So that

I can make decisions to either resume or pause the scenario. In addition, the pyCMO environment would also benefit from knowing this information more explicitly.

Acceptance Criteria

Given

we currently do not check when a scenario is running or paused (we make certain assumptions about the state). The closest we get is the script to check if the "Incoming message" window exists, which implies the scenario is paused.

When

we implement a check for whether the scenario is running or paused

Then

we will improve the reliability of the pyCMO environment

Notes

We need to know more about

whether we can replace the batch and Powershell scripts that are currently being used to restart a scenario with the Python module ahk

So that

we can potentially improve the performance of pyCMO

Questions to answer

1. Can we replace every batch and Powershell script with functions from ahk?
2. What are the tradeoffs of using ahk?

Why

As a

user of pyCMO

I want

to be able to export a side's contacts from the Steam version of Command

So that

I can use contacts information to target enemy units

Acceptance Criteria

Given

ScenEdit_ExportScenarioToXML currently does not support a side's contacts

When

we add the ability to export contacts to the ExportSidesToXML function

Then

we can export contacts from the game and use it for our RL agents

Why

As a

user of PyCMO

I want

to train an RL agent to solve the floridistan scenario using the PyCMO Gym environment provided in update 1.4.0

So that

I finally have RL agents in CMO

Acceptance Criteria

Given

we currently only have a scripted agent solving floridistan

When

we refine gym_demo.py and train an RL agent to solve floridistan

Then

we will prove that PyCMO can be used to train RL agents for CMO

Notes

Why

As a

user of pyCMO

I want

to see release notes

So that

I can keep track of new features that are added to the main branch of pyCMO

Acceptance Criteria

Given

we currently do not use release notes to track new features that are added

When

we add release notes and update them whenever we merge develop into main

Then

we can track features as they are added

Notes

Why

As a

user of PyCMO

I want

to visually see PyCMO"s capabilities and how to use the tool

So that

I can use PyCMO

Acceptance Criteria

Given

we currently rely on written documentation for tutorials

When

we create some demo videos for PyCMO

Then

we can better communicate the purpose and capability of the tool

Notes

1. Make a video to show a scripted agent solving the floridistan scenario as a baseline demo video
2. Make a tutorial video or gif that shows how to use PyCMO

We should

look into upgrading to the latest Python version (maybe 3.12)

Because

the project is using an old version of Python

Complexity

Easy - Medium. The only difficulty would come from dependencies that break after the upgrade.

Definition of Done

We have upgraded to a newer version of Python than 3.9

Hello, Thanks a lot for your great work! Do you know if there is any way of recording the game while an agent is playing to be able to visualize it later?

Why

As a

user of pyCMO

I want

to be able to restart a scenario, end a scenario, and check when a scenario has ended in the Steam version of Command

So that

I can develop training procedures for RL agents (which depends on ending a scenario and restarting it)

Acceptance Criteria

Given

we currently do not have a way for pyCMO to check that a scenario has ended, end a scenario, and restart a scenario

When

we implement these features

Then

we can start to write training procedures for RL agents

Notes

Hello,

First of all I would like to thank you for sharing your very interesting work! I would like to ask whether development on the NeuralNetworkAgent is still in progress or its abandonned.

Thanks!

We should

refactor features.py to handle the Steam edition of Command

Because

currently features.py is written to be only compatible with the premium edition of Command, which requires an expensive license to which many users do not have access

Complexity

Medium to hard. The Steam edition of the game does not have built-in data export capabilities, which means we will have to do a lot of Lua scripting.

Definition of Done

Users can export data from the Steam edition of Command using features.py, and the types of data being exported should match with what is possible to export from the Premium edition of Command.

Notes

• Related to this bigger issue
• Good notes

Why

As a

user of pyCMO

I want

to be able to specify different reward models for my scenarios

So that

I can train RL agents

Acceptance Criteria

Given

we currently only export the player's side's total score as the reward

When

we implement a way for users to specify a reward model

Then

we get closer to being able to train RL agents

Notes

One idea is to create a custom RewardHandler class that gets passed into CMOEnv that can calculate the reward based on the current observation

Describe the bug
The restartScenario.bat script does not work when the game first loads because the "Load recent" option in the game's "File" menu is not initialized correctly. We have to hover our mouse and click on "Load recent" before the list of recent game files (which restartScenario.bat relies on) can show up. It is not a critical bug though because we just need to remember to click on "Load recent" when we first open the game, but it is annoying.

To Reproduce
Steps to reproduce the behavior:

1. Boot up Command and load a scenario (do not click or use "Load recent")
2. Run the restartScenario.bat file or the scripts/floridistan/restart_demo.py demo.

Expected behavior
restartScenario.bat should restart the scenario when Command first loads without needing the user to click on the "Load recent" button once.

Describe the bug
The "Scenario End" message can appear repeatedly when a scenario is run on turbo mode and the observation export event trigger is set to a low duration.

To Reproduce
Steps to reproduce the behavior:

1. Load the floridistan scenario.
2. Change the speed to double turbo mode and the time trigger for the Export observation every seconds trigger to 5 seconds.
3. Run the scripts/floridistan/restart_demo.py demo and see the error during iteration 25.

Expected behavior
The Scenario End message should not popup repeatedly, or we should be able to catch this and address it as needed.

Description

As developers, we should set up an environment that controls the required packages and dependencies so that we minimize the time required to start developing. Poetry is one popular environment and dependency manager

Acceptance criteria

Developers can set up an environment on their local machine and run the repository

We need to know more about

how the in-game time compression interacts with the regular time triggers that are set

So that

we can increase the speed of the game safely (ensuring that important events are still fired correctly) and minimize training times

Questions to answer

1. Are time compression and regular time triggers independent of each other?
2. If a user sets the game to use the maximum speed, will triggers that fire every X seconds use the time passed in-game or the CPU clock?
3. If we set a trigger to fire every X seconds and use the max time compression, will that trigger still fire every X seconds reliably?

Summary

Sometimes when we restart the scenario through pyCMO, the time compression is not correctly set, or it resets to x1 time

Steps to reproduce

1. Run the floridistan/restart_demo.py script and notice that sometimes when the scenario restarts and loads, the time compression is reset to x1 instead of the correct x15.

What is the current bug behavior?

The time compression is reset after the scenario restarts and loads

What is the expected correct behavior?

The time compression is correctly set to whatever was specified in the Export observation initially event in the game when the scenario restarts and loads.

Relevant logs and/or screenshots

I was wondering whether the library strictly requires a professional edition or it can also be used with commercial version on Steam. If yes, do you have a setup guide to interact with CMO ?

Thank you in advance for your help!

Describe the bug
Related to issue #25, when a scenario ends in-game, the Scenario has Ended event might not fire properly and the environment might not know that the scenario has ended and get stuck.

To Reproduce
Steps to reproduce the behavior:

1. Load the scen/steam_demo_restart.scen scenario.
2. Disable the Scenario has Ended event.
3. Let the scenario run.
4. See error

Expected behavior
We need to know when the Scenario has Ended event has not fired properly and handle it accordingly. Currently, we can also use the CMOEnv::end_game() function to end the game.

Summary

Sometimes the MoveMouseEnterScenario.ps1 script is flaky and we miss the "Enter scenario" button

Steps to reproduce

1. Run the floridistan/restart_demo.py several times and you might see the mouse gets moved beyond the "Enter scenario" button.

What is the current bug behavior?

MoveMouseEnterScenario.ps1 does not always guarantee the movement of the mouse to the "Enter scenario" button and click on it

What is the expected correct behavior?

MoveMouseEnterScenario.ps1 will always move the mouse to the "Enter scenario" button and click on it

Relevant logs and/or screenshots