• Review related papers for the topic: Meta Reinforcement Learning for sequential decision problems.
• 1-5 score how confident I am in understanding the paper (5: I think I got all the key concepts. 1: I just skim through the paper)
• Meta-RL: leverage varied experiences from previous tasks to adapt quickly to the new task at hand.

• We assume a distribution of tasks p(𝓣), where each task is a Markov decision process (MDP).
• 𝓣 = {p(s0), p(st+1|st, at), r(st, at)}
• p(s0): initial state distribution
• p(st+1|st, at): transition distribution
• r(st, at): reward function .
• p(𝓣) encompasses tasks with varying transition functions (e.g., robots with different dynamics) and varying reward functions (e.g., navigating to different locations).
• The transition and reward functions are unknown, but can be sampled by taking actions in the environment
• => Objective: maximize cumulative reward (or minimize regret)

Note: At the moment, I focus more on MetaRL for Bandit problem.

Note2: Brief summary slides: https://docs.google.com/presentation/d/1W_qSnl3KwAoLb0yGoETh4grZNTcUocEr4Vlm8CNQh2g/edit?usp=sharing

• Augmented DQN basically teach the model to do Meta-Exploration => How do we do this to extend to other problems and method. => MAESN paper mentioned that following multiple (non-optimal) policies can inform the agent about the task structure (meta-exploration)
• Why does eps = 0.1 help improve the performance ? (training with eps is understanable, but it helps even when inference)
• Understanding PEARL: investigating the prior that they estimate. Does it overlap with the (bandit) environment actual prior? Can we improve upon this method? => They extract the latent context vector, probably contain the arms' prior for bandit problem (but I'm not gonna do an elaborate research on that).
• Gradient based method used gradient update(s) at test time to quickly adapt, while context based just extract context information from some approximate function. Can we combine them to increase efficient?
• When facing out-out-distribution tasks, Gradient Based methods (MAESN, MAML) revert back to normal Policy Gradient, while Context Based methods will most likely fail. Can we exploit this characteristic to increase the robustness of PEARL ?

• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning(4) (VariBAD)

• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables (PEARL) (4.5 Read Soft Actor-Critic paper to fully understand final details)

• A Simple Neural Attentive Meta-Learner(5) (SNAIL)
• RL2: Fast Reinforcement Learning via Slow Reinforcement Learning(5) (RL2)
• Learning to Reinforcement Learn(4)
• Learning to Learn without Gradient Descent by Gradient Descent (3.5)

• With probabilistic latent context
  • Meta-Reinforcement Learning of Structured Exploration Strategies (MAESN) (3)
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (MAML) (3)
• Learning to Explore with Meta-Policy Gradient
• ProMP: Proximal Meta-Policy Search
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning.

• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Evolved Policy Gradients

• Meta-Gradient Reinforcement Learning

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (SAC)
• Meta-learning of Sequential Strategies(1) (Review stuff)

• Meta-Learning for Contextual Bandit Exploration(3)
• A Bayesian Approach to Unsupervised One-Shot Learning of Object Categories (Hierarchical Bayesian approach)
• Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Control (3) (PCC)

Table 1: Results on multi-arm bandit problems. OTS: Optimistic Thompson Sampling. Greedy: with the best empirical mean. Horizon = N, Number of arms = K

Table 2: Results on tabular MDPs. Check SNAIL paper for original source.

Table 3: Results on multi-arm bandit problems. Horizon = 300, number of arms = 2, gamma = 0.9.

NOTE:

• There is some bias in the number: DQN (12 trajectories) method received more trials than others.
• Augmented DQN: generate training samples with known good latent features (average reward, number_of_chosen**-0.5, current timestep).
• Augmented DQN on average required ~7 times less data to converge than vanilla DQN.