Comments (5)
Hi @FeynmanDNA thanks for checking out gym-lattice!
The default setup is that each generated sequence is an episode with sparse rewards (i.e., you only get the reward at the end). One or more agents can solve it:
- If it's just one agent, then it will create a sequence for each episode. Ideally, you'd run multiple episodes across the whole training time (trajectory).
- You can also do multiple agents using Batch PPO, IMPALA, or A2C. Each agent will then have its own trajectory consisting of multiple episodes (and consecutively, a generated sequence per episode).
It seems that this project's been two years ago so I'm not sure how updated it is in its domain. Unfortunately, I wasn't able to continue my research on this (but still keeping tabs on what's happening in this topic). The current SoTA in my opinion is DeepMind's AlphaFold. They do some level of feature engineering (distance and angle predictions) before running it on a classifier.
If you wish to know more information, I've written some posts regarding this project (some may be a bit dated):
- On gym-lattice itself: https://ljvmiranda921.github.io/projects/2018/05/13/gym-lattice/
- On multiagent setups: https://ljvmiranda921.github.io/projects/2018/09/14/pfn-internship/
- On episodes and training steps: https://ljvmiranda921.github.io/notebook/2017/11/26/a-brief-soiree-with-reinforcement-learning/
from gym-lattice.
hi @ljvmiranda921 , I have read through your related writings on RL on your blog before. They are very educational and inspiring! I have always wondered were there any followup research from your Waseda lab-mates in this direction :)
Thank you so much for your sharing and pointers to other resources. I will check them out!
Regarding the "agent" and "generated sequence", the main concern I have is that since your observation_space is kind of tied to the sequence length:
self.grid_length = 2 * len(seq) + 1
...
self.observation_space = spaces.Box(low=-2, high=1,
shape=(self.grid_length, self.grid_length),
dtype=int)
So when the training is done for one sequence HHHPHH
, the same RL q-table or DQN weights might not generalize to other sequences like HPPPHH
?
My first impression on the gym-lattice design is that the sequence is in a way like the frozenLake map. The agent will learn to traverse through a particular frozenLake map, but will need to retrain for a different frozenLake map.
I am very new to RL, so please correct me if my understanding about the gym-lattice design is wrong.
Once again, awesome open-source project with solid tests and documentations. Thank you for your work!
from gym-lattice.
Hi @FeynmanDNA , thanks for this!
So when the training is done for one sequence HHHPHH, the same RL q-table or DQN weights might not generalize to other sequences like HPPPHH?
I don't have enough information to confirm this, but back then I'm expecting that the same DQN weights should be able to generalize on sequences of the same length. I'm trying to remember why the design is like that, but my vague recollection is that I'm planning to compare it on the HP Lattice Protein Folding baselines (linked here is a recent paper related to this).
My first impression on the gym-lattice design is that the sequence is in a way like the frozenLake map. The agent will learn to traverse through a particular frozenLake map, but will need to retrain for a different frozenLake map.
You may be on to something, my hypothesis is that it should generalize if they're on the same length. Thus you can use the same agent to solve HPHH
and HHPP
(length=4), but not HHHPPPPHHPHPH
(length!=4). Would be nice to confirm this experimentally!
from gym-lattice.
Thank you very much for your generous sharing!
I will try out some experiments with the current observation space design, and let you know if anything interesting comes up!
from gym-lattice.
@ljvmiranda921 Hi Miranda, thank you very much for open-sourcing this awesome repo! We built on your gym-lattice environment and implemented a prototypical DQN network with LSTM core that is able to achieve better results than previous RL/DRL studies.
The poster and extended abstract are accepted at Machine Learning and the Physical Sciences workshop, NeurIPS 2022
The full manuscript is under review, and preprint is on arxiv Arxiv @ https://arxiv.org/abs/2211.14939
Once again, thank you very much for the gym-lattice environment and your blog posts on HP-model :)
from gym-lattice.
Related Issues (10)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gym-lattice.