Comments (1)
Sorry for delay :).
The main difference is that ExperienceSource
produces all traces of given length, but ExperienceSourceFirstLast
returns only first and last states with calculated discounted reward between. It could be illustrated on example.
Suppose we have single episode with states 0 -> 1 -> 2 -> 3 -> 4. On the last state episode is terminated.
Suppose we have ExperienceSource(steps_count=3)
, then it will produce the following data on iteration:
- [Experience(state=0), Experience(state=1), Experience(state=2)]
- [Experience(state=1), Experience(state=2), Experience(state=3)]
- [Experience(state=2), Experience(state=3), Experience(state=4)]
- [Experience(state=3), Experience(state=4)]
- [Experience(state=4)]
But ExperienceSourceFirstLast(steps_count=3)
will return the following:
- ExperienceFirstLast(state=0, last_state=2)
- ExperienceFirstLast(state=1, last_state=3)
- ExperienceFirstLast(state=2, last_state=None)
- ExperienceFirstLast(state=3, last_state=None)
Reward returned by ExperienceSourceFirstLast
is aggregated using gamma passed on constructor.
Most of the time, ExperienceSourceFirstLast
is more convenient, as we're not normally need intermediate states. But sometimes, we need more control, so, ExperienceSource
could be handy.
In terms of implementation, ExperienceSourceFirstLast
is a wrapper around ExperienceSource
.
from ptan.
Related Issues (20)
- Slower speed than book indicates HOT 3
- Where to find and how to restore dqn_speedup model? HOT 2
- ExperienceReplayBuffer stores second-last transition twice HOT 2
- Re-open "ExperienceReplayBuffer stores second-last transition twice" HOT 2
- Code run time HOT 1
- It requires pytorch 1.3.0, but I can't download it HOT 8
- Which PyTorch version? HOT 2
- unable to install ptan HOT 1
- [FR] Possibility of rendering and testing the models after training.
- Weights should not affect probabilities in PrioReplayBuffer
- Unable to install ptan via pip (old versions of dependencies not available) HOT 2
- Is it possible to support PyTorch 1.5? HOT 3
- Can not install ptan HOT 1
- Examples in chapters 8 and 9 don't run HOT 5
- Not able to run ptan.ignite when custom events inherit from enum.Enum HOT 4
- Can not install ptan because of Pytorch version HOT 4
- Pytorch version 1.7.1
- Multi agent support
- atari-py is no longer supported
- Outdated Experience Source
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ptan.