There is an effort to emulate some physics, such as gravity effects by pushing center of mass downwards. But there is no seperation of agent and environment.

Tried networks.

• Chain: ls, qs, goal input -> delta_qs.
• Chain: xis, yis, goal input -> delta_qs.

Local com control.

• Implement COMx control.
  • delta_q1 = 2 * x_c * dx/dq1; not = dx/dqq; i.e. min x_c^2 not x_c.
  • Discounted com control for q_i by 1 / i.
  • Sending com to origin vs origin + goal / 2. Can actually send anywhere.
  • Optimized calculation.
• COMy control. push com_y downward.
• Local maxima problem ys = 0. (very rare problem since other controls are generally involved.).
• Heuristics to model powering through (adrenaline).
  • gaussian randomized end control (sometimes the weight is > 1 modelling overpower).

Center of mass realism; duct-taping.

• Local com control.
  • Implement COMx control.
    • delta_q1 = 2 * x_c * dx/dq1; not = dx/dqq; i.e. min x_c^2 not x_c.
    • Discounted com control for q_i by 1 / i.
    • Sending com to origin vs origin + goal / 2. Can actually send anywhere.
    • Optimized calculation.
  • COMy control. push com_y downward.
  • Local maxima problem ys = 0. (very rare problem since other controls are generally involved.).
  • Heuristics to model powering through (adrenaline).
    • gaussian randomized end control (sometimes the weight is > 1 modelling overpower).

Arbitrarily global optimal control (Random sample solve and interpolate control).

• From the spirit of RANSAC.
• Given end effector goal, randomly sample q vector (in q clamps range) and keep the q* which achieves closest approach.
• This at limit should not be stuck at local minima. Therefore is bit different from gradient descent.
• These iterations can be stopped after a fixed number of samples or if closest approach is less than a threshold.
• Given q* just interpolate from current q to q*.
• Parallelizable.

Arbitrarily global optimal control (Cross-entropy solve and interpolate control).

• From the spirit of CEO.
• Improvement. Instead of sampling randomly in whole q clamp, sample in small region around q, take the best q*, then sample in vicinity of q* and so on.
• More prone to local minima but given enough big sampling region local minima can be avoided.

Global optimal planning (Solving local planning minumum problem. Agents get stuck due to them even for cases where there is a solution).

• 1. Heuristics to reduce local minima.
  • relaxation time (theoretically guaranteed local minima problem solve given enough relaxation time).
• 2. View it as a two link chain (decrease degree of freedom) (Don't want to implement now).
• 3. Random global optimal solve.
• 4. Cross entropy global optimal solve.

Reaching a hold.

• Local planners.
• Global planners. How to snap to hold once close enough (Give responsibility to local planner).
• Weights of both planners as a function of ticks.
• Restrict q0 sampling.
• Optimize sorting in genetic planners.

Switching pivot.

• q and q clamp assignment on switching (refer to code for math and why q1 clamp has to be (-inf, inf)).

Matching hands. If your right hand is free and next hold is on your left; switch hands.

• using goal_reached_slack in deciding to match hands;

  let have_to_match = match pivoting_side {
      Side::Left => given_goal[0] - origin[0] < -Chain::GOAL_REACHED_SLACK,
      Side::Right => given_goal[0] - origin[0] > Chain::GOAL_REACHED_SLACK,
  };

• But now the end of the hand can be atmost 2 * GOAL_REACHED_SLACK from the hold.
• And to visualize this we need sqaures of size 4 * GOAL_REACHED_SLACK from the hold.

Learning based neck position predictor + (Monte-carlo + gradient descent) based inverse kinematic controller for chains.

2 limb as 2 switching Chain.

• Enforcing constraints - (no more constraints; uses previous constraints).
• Formulating as RL problem.
  • Very nice visualization of holding goal w.r.t non-holding goal.
  • Useful for debugging, reward function design and testing.
    • since the output of network does not depend on qs (initial state) if every point is roguhly tested it is enough, O(N) no need to test every pair of points (src, dest) O(N^2).
  • Encoding input.
  • Decoding output.
  • Reward function design.
    • Explain why end of episode comy reward is a bad one (If the goal is high up and the agent reaches it; its com y will be high resulting in a lower reward for good behavior).
    • This is also a problem with in episode comy reward but (giving less weight to it and having a lot of episodes per batch asymptotically mitigates it).
  • FCN design.
  • CEO parameters tuning.
  • Left, right holding seperate networks.
  • Ensure and showcase mostly working, holding origin invariance, scale invariance.
  • Auto scale goal region based on holding ls.
  • Discuss how non-trivial the deciding holding goal and how RL is a good tool here (different positions, non-trivial gaits, discontinuity at ends).
• Switcing and matching (transfers).
• Can even make a full network visualization ((x,y)_non_holding_goal vs dist(x,y)_holding_goal_from_origin).
• Left and right holding single network (since the current task can be seperated into mutually exclusive and exhaustive problems; left holding and right holding, seperate networks for each would work).
• Improve comy behavior (loss function itself and tuning).
• Improve visuals (stich sprites).
• Improve the transfers demo.
• Uneven climbing agents (ohc reach).