To improve portability, we should introduce a mechanism to inspect the underlying hardware (e.g. GPU, TPU, CPU-only) and set values based on that. We can create a factory pattern to return a set of pre-defined constants for hardware we recognize, as well as a default to use for non-CPU hardware which we do not recognize.

We need an implementation of PPO. Start with using the Keras API.

We need a simulation environment to test with. Something like an environment from AI Gym. Choose one to explore the capabilities of PPO combined with CQL.

Once we have our high level PPO implementation, we should verify that it is working. Design a test plan to confirm this, and carry it out.

Now that we have a working version of PPO in notebook form over in #1, it would be nice to convert it into a python module which can be imported and called.

We need to implement the pieces of CQL for offline RL. See https://bair.berkeley.edu/blog/2020/12/07/offline/ for details.

Play around with our PPO implementation both with and without CQL. Evaluate what impact it has.