praveen-1 / rl-library Goto Github PK

This is a big issue and it needs its own wiki page and people working on
it, but we need to write the phone home server or scripts.

The local glue code is cobbled together and needs to be written right and
tested.  IT doesn't implement everything, and I'm not sure its doing the
right thing in terms of time step counts and such.  Should look at the core
glue code and compare.

This tester is also an experiment program.  It will be Java based and will
phone home first and load the experiment Jar over the Internet.

Maybe we can do some online scoreboarding as this thing runs and show other
people's results as it goes, with bar graphs or something.  What I mean is
that as its benchmarking you could literally have it now how well its doing
which would make it fun to watch.

The phoning home will need to be network code written independent of the glue.

Each of the projects now can be built with Ant scripts, we should write
some more to do the whole things at once... the equivalent of clean-all
make-all

Fix this.

This is a big issue and will take lots of work, but we need the Java tester
program.

We need to make sure helicopter is working right and we need to contact
Pieter Abeel and make sure we have a good altered problem plan.

We need a sample agent for the Polyathalon

The value function resolution slider should only be there if you're using
it.  Not sure if that functionality should be moved into the visualization
somehow (maybe?) or what, but we need to fix it for the competition.

We had the viz frame working in the tech dist, we should be able to do it
for the competition package... but its not the highest of our priorities.

We need a simple idea of what the Polyathalon domain might look like from
the user perspective.

This is a big one.

We should create a super simple experiment program in C that could be used
as the basis for making a trainer.  This will show people that it really is
cross platform and doesn't require java.

We should work a little harder on designing EnvironmentShell.EnvLoadingHelper

I Want 2 of them.  One that lists and loads environments from the local
file system.  The other will go onto the web.  This works and there is an
example of it in there after an if statement that is false.

We should find a way that we can take a C or C++ env/agent and build it
into a dylib without having to edit it and extern "C" anything.

On useful visualization component would be to show the value of all the
next actions.... this would allow the user to visualize things more easily.

However, perhaps this should be an agent simulator....

It could be a generic component, like the value function visualizor.  I
guess you could literally have people turn on and off different
visualizations (in the viz program), like turning on or off a monitor of
various conditions.  Basically, you could make it like a "build you own"
visualizer. Hmm.

Should be a simple way to decide if agents/envs or both will be loaded
through the visualizer.

Make a way for the Java environments for the competition to take a
parameter configuration number instead of actual parameters.

Find out about signing jars and obfuscating source or class files.

See what we can do about building an appropriate manifest too, with
copyright and credits.

Shivaram is working on this

Java strings are immutable, and if you are building big ones they are horribly 
slow.  So, to serialize 
1600 observations that have 2 doubles each, building immutable strings took 
2182 ms.  This is 
compared to 240 ms to actually send the message over the network and receive a 
response.

After changing from immutable strings to using a stringbuffer and calling 
append, the time 
dropped from 2182 to 7 ms.

That's crazy.

We need the Helicopter Sample agent working

I thought I made this issue but I Can't find it.

I'd like the RL-Glue directory in the root of the training pack to actually
be an svn:externals to the RL-Glue repository.  I don't like the idea of
maintaining  2 copies of that code.

Adam has found some debug spew we need to clean up, and I think I have too
-- the JAVA rl glue seems to be printing task specs out.

Assigned to Mark Lee b/c he's the RL-Glue and SVN expert.

I created a mechanism for asking the environment for his RLVizVersion, and
I stored it in RLGlueLogic.

We need a good way to query this before starting down paths that will make
us send queries that won't be answered.

In RLVizLib, there is an interface that is called QuerableAgent, it should
be Queryable agent.  We should fix this and propagate the changes to all of
our docs and sample agents, and the viz components and messages that use this.

Right now the consoleTrainers don't really unload, they just load and load
and load again. Is this bad?

Look for memory leaks or other bad possibilities.

For now, we are going to crash with a stack trace if invalid actions are
selected.  In future (competition) we should keep running more robustly.

the Java Task Spec Parser is Broken. Currently we are hard coding values
for each environment in. BOOOO 

When I run MountainCarAgent against consoleTrainerJava, sometimes I get
double free errors.  I think it might be that cleanup() is or isn't being
called.

Should look into that, cause it looks bad.

RL_agent(7914) malloc: ***  Deallocation of a pointer not malloced:
0x3003e0; This could be a double free(), or free() called with the middle
of an allocated block; Try setting environment variable MallocHelp to see
tools to help debug
RL_agent(7914) malloc: ***  Deallocation of a pointer not malloced:
0x300400; This could be a double free(), or free() called with the middle
of an allocated block; Try setting environment variable MallocHelp to see
tools to help debug
RL_agent(7914) malloc: ***  Deallocation of a pointer not malloced:
0x3003e0; This could be a double free(), or free() called with the middle
of an allocated block; Try setting environment variable MallocHelp to see
tools to help debug
RL_agent(7914) malloc: ***  Deallocation of a pointer not malloced:
0x300400; This could be a double free(), or free() called with the middle
of an allocated block; Try setting environment variable MallocHelp to see
tools to help debug

Probably every single file we send out should have ownership and copyright
messages, as well as a license.

We should get on settling what this will be.

Make sure everything works in Linux

While neither Querable or Queryable are real words, the latter is in more
common use by programmers.

The place I found it was
rlVizLib.messaging.interfaces.getEnvMaxMinsInterface.getMaxValueForQuerableVaria
ble()

I've got an environment where actions are [-5,5]. The Random Agent only
chooses actions from [0,5].

Need to get the RTS Java visualizer integrated

We need a sample agent for mountain car, I think the plan was to make it a
Java Sarsa(lambda) tile coding agent.

Need fault tolerance in the protocol.  Need a query system to see if the
agent or environment supports it, and everthing needs to work well if they
don't.

The C/C++ files we need (ParameterHolder et. al?) probably shouldn't exist
both in the rl-competition repository and in the library.  We should decide
which is the "right" copy (probably the library) and then only point to it.

There should be a folder of jars and .dylibs (maybe just jars) for
visualizing and all of the visualizer specific code should be OUT of the
source for bt-Viz and the VizLib.  They should be independent, either
inside the environment project or someplace else.

Should investigate on how to make all of the connectivity stuff faster. 
Not sure if that should be via compressions, or being judicious, but I'd
like the visualization to be even faster and able to handle higher
resolution of value functions.

1)  We should figure out exactly what characters that break the parameter
Holder (like currently, having :," ", or "_" in either the names of the
parameters or their values (if using strings) might break things.

So, I propose a few changes
a) When parsing messages in RL-Viz, we shouldn't split on spaces all the
time, we should use spaces to split, TO, FROM, CMD, etc... but when we get
to VALS (payload) we should just take the whole rest of the string.  This
means that spaces in the payload won't break the payload parse.  This
should be one change in GenericMessage.

b) In ParameterHolder, when someone sets a variable name or a Strign
parameter, we should EXPLICITLY check right then for "_" and ":" and throw
an exception or die if we can't use it.  Or else we can replace it with
another character like ; and print a warning.  If we do this on both sides,
then invalidly specified parameters will work without crashing

c)  Test all of this and make sure we can't break it.

Right now someone using a value function might have to write like 10 lines
of code to get the slider up and running.  We should look at folding all of
that code into something inside the value function code, which lives in the
VizLib.

We're pretty desperate for documentation.

Someone needs to put together a getting started guide and get it online.

This has to wait a few days, until the trainers and how to run them settles
a bit.

There are various updates that need to be made to the competition website,
not sure what they all are, but that's important.

In Tetris, its important to tell the agent the number of Rows and Columns.
 We've decided it makes sense to use the messaging system for this.  So:

1)  The benchmark needs a way to ask the environment for the rows and cols
2)  The benchmark needs to send these to the agent
3)  The agent needs to parse these

We should be able to do this on the environment side using RLViz messages,
and on the agent side using  RLViz messages on the Java side and some
simple C/Python code otherwise.

We need a functioning trainer that handles agents and can be used to train
agents on the training version of the competition environments.

Title says it all

We should create a Python Trainer that does exactly what the Java and CPP
trainers do.

Here's what would be cool.

A bash script, which when run, goes through a bunch of scenarios, like
running a bunch of combinations of agents and trainers, such that in 60
seconds, one of us can verify that the combinations of things all work.

It's not perfect, but it would help.

We should add value function components and turn the value function slider
on and off depending on if the agent supports it.

Need to talk to Matt Taylor and get Soccer integrated.