Each notebook contains the content and code-along of each session. We recommend that you run the notebooks from Google Colaboratory for minimal setup requirements. Edit the Fill in The Code section for coding assigments and check with our way of solving them in solutions.

Markov Decision Processes / Discrete States and Actions

• What is Reinforcement Learning: Pavlov's kitties
• How Useful is Reinforcement Learning: games, robotics, ads biddings, stock trading, etc.
• Why is Reinforcement Learning Different: level of workflow automation in classes of machine learning algorithm
  • Use cases for reinforcement learning
• Reinforcement Learning Framework and Markov Decision Processes
• GridWorld example to explain:
  • Problems: Markov decision processes, states, actions, and rewards
  • Solutions: policies, state values, (state-)action values, discount factor, optimality equations
• Words of Caution: a few reasons Deep Reinforcement Learning Doesn't Work Yet
• Challenges:
  • Read up on Bellman's equations and find out where they hid in our workshop today.
  • What are you ideas about how we can find the policy policy?
  • Play around with Gridworld. Tweak these variables and see what happens to state and action values:
    • Expand the grid and/or add some more traps
    • Wind probability
    • Move rewards
    • Discount factor
    • Epsilon and how to decay it (or not)

Discrete States and Actions

• Blackjack-v0 environment, human play and computer play
• Optimal Strategy for Blackjack
• What is Monte Carlo Method
• Monte Carlo Prediction
• Monte Carlo Control: All-visit, First-visit, and GLIE
• Challanges:
  • What are some other ways of solving reinforcement learning problems? How are they better or worse than Monte Carlo methods e.g. performance, data requirements, etc.?
  • Solve at least one of the following OpenAI gym environments with discrete states and actions:
    • FrozenLake-v0
    • Taxi-v2
    • Blackjack-v0
    • Any other environments with discrete states and actions at OpenAI Gym
  • Check session2b.ipynb if you are interested in using Monte Carlo method to solve Grid World. This will give you more insight into difference between all-visit and first-visit Monte Carlo.

Discrete States and Actions

• OpenAI Gym toy environment to explain temporal difference learning: sarsa, q-learning, expected sarsa
• Homework: solve an environment with discrete states and actions such as:
  • FrozenLake-v0
  • Taxi-v2
  • Blackjack-v0
• Take-home Challenges: Solve an environment with continuous states: discretization, tile codings, etc. such as
  • Acrobat-v1
  • MountainCar-v0
  • CartPole-v0
  • LunarLander-v2
• Points to consider:
  • What are the state space, action space, and rewards of the environment?
  • What algorithms did you use to solve the environment and why?
  • How many episodes did you solve it in? Can you improve the performance? (Tweaking discount factor, learning rate, using Monte Carlo instead of TD)

• Tensor operations
• Feedforward
• Activation functions
• Losses
• Backpropagation
• Why is deeper usually better? Spiral example

Continuous States and Discrete Actions

• Some approaches to continuous states: discretization, tile coding, other encoding, linear approximations
• Vanilla DQN: experience replay and target functions
• Take-home Challenges: Work on an Atari game and detail the process of hyperparameter tuning

Continuous States and Discrete Actions

• Rainbow
  • Vanilla DQN (experience replay + target network)
  • Double DQN
  • Prioritized experience replay
  • Dueling networks
  • Multi-step learning
  • Distributional RL
  • Noisy networks
• Take-home Challenges: Implement Rainbow and compare it to your last project

Continuous States and Actions

• Policy gradient methods: a2c, a3c, ddpg, REINFORCE

• Monte Carlo tree search

• Explore vs exploit: epsilon greedy, ucb, thompson sampling
• Reward function setting
• Hackathon nights to play Blackjack, Poker, Pommerman, boardgames and self-driving cars

• Sutton and Barto
• Stanford CS234
• Udacity RL Nanodegree
• David Silver Lectures
• UC Berley Lectures
• Siraj's Move 37
• Denny Britz Repo
• Intro to RL in Trading

• OpenAI Gym
• Unity ML-Agent Toolkit
• Pommerman
• MetaCar
• Boardgame.io

• Unity ML-Agent Toolkit
• Dopamine