se2p / tool-competition-av Goto Github PK
View Code? Open in Web Editor NEWThe repository hosts the code for the SBST CPS Tool competition for testing autonomous cars.
Home Page: https://sbst22.github.io/tools/
License: Other
The repository hosts the code for the SBST CPS Tool competition for testing autonomous cars.
Home Page: https://sbst22.github.io/tools/
License: Other
We need to define basic test oracles:
The current mechanism for generating the OOB reports fail when there's at least one invalid test
Setup a beamng executor that uses beamgpy to setup the scenario, configure bemangAI to drive from the first to the last point in the road, and collect all the simulation/sensor data except images, lidars. Data must be tagged using SIMULATION time (from the Timer sensor).
Add basic visualizations of the test, mostly for debugging purposes.
In the current version, we are always obtaining NaN as the "oob_distance" value in the JSON files.
Example: "steering": 44.62020603589313,
"steering_input": -0.08308082203328948,
"brake": 0,
"brake_input": 0,
"throttle": 0,
"throttle_input": 0,
"wheelspeed": 9.729266273285269,
"vel_kmh": 35,
"is_oob": false,
"oob_counter": 0,
"max_oob_percentage": 0,
"oob_distance": NaN
After inspecting the code, we realized that in line 30 of oob_monitor.py the clauses of the inline if seem to be flipped.
Currently:
last_max_oob_percentage = self.last_max_oob_percentage if oob_bb else float("nan")
(line 29)
oob_distance = float("nan") if oob_bb else self.oob_distance( wrt=wrt)
(line 30)
How we believe it should be:
last_max_oob_percentage = self.last_max_oob_percentage if oob_bb else float("nan")
(line 29)
oob_distance = self.oob_distance( wrt=wrt) if oob_bb else float("nan")
(line 30)
Could you confirm that our understanding is correct and fix the code in the repository?
Besides, can we understand a negative value of oob_distance means that the car is out of bound and the test failed?
Thank you very much in advance, and happy new year!
We need a component that collects some statistics about the generation, including
Tests might fail because the car drives out of the lane (OBE) or because of a timeout triggers (the ego-car does not move).
The statistics should keep track of this
The file generated by the executors unless said otherwise are to be considered internal/private and test generators should not rely on them. Instead, test generators should solely work with the execution data returned by the underling executor
We should sync public and private repos and integrate the analysis scripts uses for the competition here.
Improve the documentation to include installation instructions, example of usage, and instructions to register and setup BeamNG.research.
I am not sure if this is needed on all systems but I had to install also the PyOpenGL-accelerate
module. Maybe this can be added to the requirements-37.txt
.
Implement the listed test validators including:
4) Maybe link to the chair for the SBST tool competition somewhere, with links to home pages?
"Setup Guide and Program Description"
The goal of this part is to give a brief description of how the competition code works.
Supported Operating Systems
Pre-Requisites
Setup Information
For information about setup and use, please refer to the instruction provided here.
Pipeline architecture description
...
Pipeline installation
...
TO RUN THE TOOL
....
...
"
for the architecture description, with Alessio, we got a figure from the BeamNG team for a paper we wrote with them.
Add section reference, so that the original pipeline is linked to the repository
Let's add a separate section to the "Competition Evaluation Methodology" (or something similar) to describe what the competition actually does
Move installation consideration under the installation section of the pipeline
In general, try to have a README.md file for each sub-folder, so that who visit the repository does not get lost.
Catmull-Rom cubic works with at least 4 points. Enforce the min number of points as 4.
Avoid to use plain list with points and introduce a structured abstraction to handle tests instead
Many files miss the license information. This should be added
Test generators that expose the same problem (e.g., OBE) multiple times are not that effective. We should report in the statistics and after each failed test execution, whether or not a test failed the same way (or similarly) before.
Is there any specific version of python we should use?
There are several options for the competition.py
app. We should document better them
We need to define an organization of folders/files, and naming conventions to properly manage the results and artifacts of the test generation
Now we report the stats in the CLI
Improve visualization
It would be nice to record the additional information about generated tests:
Hi, I am new to this framework. I like the code pipeline and it is easy to use. I felt free to share a few thoughts about the interface for generating the test cases :-).
As a new user of this code pipeline, I want to implement only the part of a test generator that describes the actual test case such that I don't need to focus on the test execution and reporting.
Suggestions for refactoring:
start
and __init__
methods) which have to be implemented by the test generators.road_points
in the start
method:def start(road_points):
# add points to `road_points`
road_points
could look like road_points = [test1_list_of_points, test2_list_of_points]
These are just my thoughts. What do you think about this design option? Do you see some drawbacks in this?
The mock executor can rely on TUM logic for trajectory planning, but will not be able to provide all the data that BeamNG does.
@alessiogambi describes random_generator and one_test_generator
@p1ndsvin describes deepJanus_seed_generator
Would it be possible to know the order of magnitude of the time budget during the competition? Will it be minutes, hours, days? It would be useful to know that to determine which approach is more suitable.
Thank you very much again.
Add options to customize oob tolerance and speed limit
Possibly, we should report the duration in real-time and simulated time for executing tests as well as the time spent in generating them.
Check if the budget is over at the end of the last test case
At the moment it only contains click for parsing the command line arguments
Additionally, the simulation goes into an infinite loop.
this is the code to generate the test:
test.append( (10, 10) )
test.append( (10, self.map_size -10) )
test.append( (self.map_size -10, self.map_size -10) )
test.append( (self.map_size -10, 10) )
We understand that running multiple simulations in parallel is not possible. However, is there any other restriction on the number of threads we can use? For instance, is it possible to run a background thread while the simulator is running?
Thanks again!
We have a question regarding the output of the tool.
In the description of the competition, it says that "a test fail if the ego-car does not move, or does not reach the end of the road within a timeout (computed over the length of the road), or drives off the lane."
Given the path [(10, 100), (40, 113), (60, 115), (80, 99), (100, 80), (120, 70), (140, 58), (160, 38), (180, 30)], the ego car goes out of the lane in the very first curve. However, the test continues to the end and the output of the test is "success". Shouldn't the output of this test be failed? What does exactly mean to "drive off the lane"? We assume that crossing the yellow line should also be a failure.
Also, does the test always continue to the end even if there is a failure at an early stage? Or should the test actually stop around 00:08s, when the car goes out of the lane?
Thank you.
Mostly documentation
see #43
BeamNG executor requires some preconditions to be met, including env variables like BNG_HOME. Currently, those are not checked and the execution simply fails with an error message.
Steps to reproduce:
road_points = []
road_points.append((50, 50))
road_points.append((10, 10))
or simply run unit_tests.test_road_that_is_entirely_inside_the_map
Hello,
would it be possible to integrate in the code pipeline an oob_percentage reporting? After execution of a test case, to either obtain the oob percentage value per state or the maximum percentage value, regardless if the executed test case passed or failed (similar to the reporting of the distance value within the execution data).
BeamNG does not start with my non-standard userpath (Windows' "My Documents" moved from standard C: drive to F:). This results in a timeout when running the competition scripts.
2021-01-11 14:23:39,040 ERROR Uncaught exception:
Traceback (most recent call last):
File "road_definition.py", line 50, in <module>
main()
File "road_definition.py", line 39, in main
bng = beamng.open(launch=True)
File "F:\Anaconda3\envs\sbst21\lib\site-packages\beamngpy\beamng.py", line 318, in open
self.skt, addr = self.server.accept()
File "F:\Anaconda3\envs\sbst21\lib\socket.py", line 205, in accept
fd, addr = self._accept()
socket.timeout: timed out
2021-01-11 14:23:39,109 WARNING sys:1: ResourceWarning: unclosed <socket.socket fd=696, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 64256)>
2021-01-11 14:23:39,131 WARNING F:\Anaconda3\envs\sbst21\lib\subprocess.py:786: ResourceWarning: subprocess 11936 is still running
ResourceWarning, source=self)
At the moment we do not check the validity of the roads with respect to the curvature, but we should. For example, invalid roads have turns so sharp that it is physically impossible for a car to drive on them.
I just tried to follow the installation guide that recommends Python 3.6 (at least it states that stuff was tested with 3.6).
The time.time_ns() method on line 120 in competition.py was introduced in Python 3.7. This has to replaced by either int(round(time.time() * 1e9)) or the docs should not recommend Python 3.6.
If the script is executed from a directory different than root, beamng executor cannot find the "TIG" levels and fail.
For the moment the workaround is to manually copy
Add the following license header in all the source files:
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.You should have received a copy of the GNU General Public License
along with this program. If not, see https://www.gnu.org/licenses/.Acknowledge and link this repository and its authors in case of usage.
We do have in place a timeout mechanism that triggers if the test subject does not move. However, this is not yet tested, and we should probably introduce a way to specify a timeout on the entire simulation as well.
Setting up the pipeline is quite easy by following your documentation. However, I struggled a bit with the beamng-user
argument. In GUIDELINES.md
under section Technical considerations
is this argument not documented.
This makes it possible to uniquely identiy tests
Since a few commits, the requirements.txt file got out of sync. We need to update this
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.