A server that catalogs bag files and provides a web-based UI for accessing them.
License: Other
The Bag Database is a web-based application that monitors a directory for ROS bag files, parses their metadata, and provides a friendly web interface for searching for bags, downloading them, and running post-processing scripts on them. It also provides a REST API that can be used to perform automated tasks.
The Bag Database's documentation is available as a GitHub Pages site.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
It'd be nice to have a window where you can view a list of all of the duplicates, pick which ones you want to keep, and then delete all of the rest.
I am following the instructions given on your page, but when I reach the point where I have to start a PostgreSQL container with PostGIS support, I get the following error:
Unable to find image 'mdillon/postgis:latest' locally
latest: Pulling from mdillon/postgis
f7e2b70d04ae: Pull complete
027ad848ac9c: Pull complete
7c040ef66643: Pull complete
b891079ad2eb: Pull complete
cb64a97e42d9: Pull complete
1b88625f7d89: Pull complete
a6ac0b663e77: Pull complete
594497f0a694: Pull complete
ca7201b6a21f: Pull complete
48cdfad3f2fd: Pull complete
912fb62e7390: Pull complete
1e6365c64609: Pull complete
eda829b73ec7: Pull complete
1dafb86732d6: Pull complete
ad1854653222: Pull complete
410147bb9559: Pull complete
ee8d11af0d8f: Pull complete
Digest: sha256:ac5dd18c43b0ee89320764684aa5c2da5c3c2ce54cf01a6caab21f905302d1b5
Status: Downloaded newer image for mdillon/postgis:latest
57e50c25b828c065da1364a4e4c8f4b358cd6b019c6c62c898d457a2f000fbc9
docker: Error response from daemon: error while creating mount source path '/var/lib/bagdb-postgres': mkdir /var/lib/bagdb-postgres: read-only file system.
OS: Ubuntu 16.04
@evenator noticed that when dowloading a bag file, Firefox says it's an HTML document. Upon investigation, it looks like the "Content-Type" header on the response is "text/html;charset=UTF-8", which is definitely wrong. It doesn't seem to cause any problems, but it should probably be fixed.
I don't think there's an official MIME type for ROS bag files, but probably "application/octet-stream" is fine.
It would be nice to be able to connect the bag database to an S3 bucket that would hold the bag files instead of needing to have them on disk.
I uploaded to the server folder a few big bag files (around 4 GB each) and although they appear correcltly in the list, the message types and topics are empty. Do you know what could be the issue?
Here are some screenshots showing the problem:
We have noticed that GPS position inserts take an inordinate amount of time on our files (GPS is recorded at ~10Hz on files that can span 2 hours).
Are the inserts synchronously waiting for a response from the Postgres DB? Would it be easy to either batch insert or insert asynchronously?
Hello,
we are storing our bag files in a folder structure for a better overview. The bag file names contain only the date and time (hard to differentiate). I was wondering if there are any plans to implement a 'folder view' in the ROS Bag database as well. This would be very handy for us. For example:
....
It turns out that for many of our bag files the start and end times displayed are wrong (see below for an example). The duration is correct however for those files.
Would you know what could cause this or how to fix it?
Highlighted blue line is the one of interest
We could potentially test several of the fields in the config to make sure they're valid. Do so and return an error message if the user tries to submit invalid values through the config GUI.
The infrastructure for this should already be in place; Docker instructions involve starting up a Postgres database with PostGIS extensions, and Lucene database indexes are being generated. We still need to design and implement a way for users to actually do geospatial searches.
While starting up the bagdb in docker I receive 3 failed commands in entrypoint.sh:
/entrypoint.sh: line 12: perl: command not found
/entrypoint.sh: line 13: perl: command not found
/entrypoint.sh: line 14: perl: command not found
Which matches these lines in entrpoint.sh:
VEHICLE_NAME_TOPICS='['`echo ${VEHICLE_NAME_TOPICS} | perl -pe 's#([/\w+]+)#"\1"#g'`']'
METADATA_TOPICS='['`echo ${METADATA_TOPICS} | perl -pe 's#([/\w+]+)#"\1"#g'`']'
GPS_TOPICS='['`echo ${GPS_TOPICS} | perl -pe 's#([/\w+]+)#"\1"#g'`']'
I do not see perl as a direct dependency in the dockerfile. Should it be part of the apt-get in the dockerfile?
run on fresh install of ubuntu 18.04 x64 server + docker.io
Hello again...
Here is my situation:
I have some bags that cannot be indexed, I can see there are many errors said "Error checking bag file: could not execute statement; SQL [n/a]; nested exception is org.hibernate.exception.DataException: could not execute statement".
And I use docker logs bags | tail, it continuously shows:
com.github.swrirobotics.bags.reader.exceptions.BagReaderException: java.nio.file.FileSystemException: /bags/xxx.bag: Host is down
at com.github.swrirobotics.bags.reader.BagFile.forMessagesOnTopic(BagFile.java:346)
at com.github.swrirobotics.bags.BagService.getAllGpsMessages(BagService.java:1083)
at com.github.swrirobotics.bags.BagService.updateBagFile(BagService.java:1534)
at com.github.swrirobotics.bags.filesystem.BagScanner$FullScanner.lambda$0(BagScanner.java:441)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.FileSystemException: /bags/xxx.bag: Host is down
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at com.github.swrirobotics.bags.reader.BagFile.getChannel(BagFile.java:127)
at com.github.swrirobotics.bags.reader.BagFile.forMessagesOnTopic(BagFile.java:335)
I'm wondering if it will call a service but the service is down?
The current implementation worked when there were separate columns in the DB, but now that they're being stored as a geometry type, they need to be reworked.
Currently it can grow without bounds, so something like a huge number of duplicate files or unreadable files can potentially lead to hundreds of thousands of error messages over time. This can cause the GUI to take a long time to load and increase memory usage on the backend. At some point, older messages should be expired.
Maybe something like 10,000 messages would be a reasonable limit?
Although I am aware that you are using this internally, we are starting to look into storing bagfiles and this solution looked quite promising.
However the docker files do not seem to work on my PC. These are the two docker containers I am spawning locally:
docker run -d --name bagdb-postgres -v /var/lib/bagdb-postgres:/var/lib/postgresql/data -e POSTGRES_PASSWORD=letmein -e POSTGRES_USER=bag_database -e POSTGRES_DB=bag_database mdillon/postgis:latest
docker run -d -p 8888:8080 -v /home/tim/rosbag:/bags --name bagdb --link bagdb-postgres:bagdb-postgres -e DB_DRIVER=org.postgresql.Driver -e DB_PASS=letmein -e DB_URL="jdbc:postgresql://bagdb-postgres/bag_database" -e DB_USER=bag_database -e METADATA_TOPICS="/metadata" -e VEHICLE_NAME_TOPICS="/vehicle_name" -e GPS_TOPICS="/dory/gps/fix, /phone1/android/gps, /imu/fix" swrirobotics/bag-database:latest
And this is the resulting log files:
mkdir: cannot create directory ‘/root/.ros-bag-database’: File exists
08-Mar-2017 10:18:44.060 WARNING [main] org.apache.catalina.startup.SetAllPropertiesRule.begin [SetAllPropertiesRule]{Server/Service/Connector} Setting property 'maxSpareThreads' to '75' did not find a matching property.
08-Mar-2017 10:18:44.111 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/8.0.37
08-Mar-2017 10:18:44.111 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Sep 1 2016 10:01:52 UTC
08-Mar-2017 10:18:44.111 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 8.0.37.0
08-Mar-2017 10:18:44.111 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
08-Mar-2017 10:18:44.112 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 4.8.0-34-generic
08-Mar-2017 10:18:44.112 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
08-Mar-2017 10:18:44.112 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib/jvm/java-8-openjdk-amd64/jre
08-Mar-2017 10:18:44.112 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_102-8u102-b14.1-1~bpo8+1-b14
08-Mar-2017 10:18:44.112 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
08-Mar-2017 10:18:44.113 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /usr/local/tomcat
08-Mar-2017 10:18:44.113 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /usr/local/tomcat
08-Mar-2017 10:18:44.114 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties
08-Mar-2017 10:18:44.114 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
08-Mar-2017 10:18:44.115 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
08-Mar-2017 10:18:44.115 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.endorsed.dirs=/usr/local/tomcat/endorsed
08-Mar-2017 10:18:44.115 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat
08-Mar-2017 10:18:44.115 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat
08-Mar-2017 10:18:44.116 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
08-Mar-2017 10:18:44.116 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library 1.2.8 using APR version 1.5.1.
08-Mar-2017 10:18:44.117 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
08-Mar-2017 10:18:44.123 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized (OpenSSL 1.0.2i 22 Sep 2016)
08-Mar-2017 10:18:44.198 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-apr-8080"]
08-Mar-2017 10:18:44.216 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 480 ms
08-Mar-2017 10:18:44.239 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service Catalina
08-Mar-2017 10:18:44.239 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.0.37
08-Mar-2017 10:18:44.264 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive /usr/local/tomcat/webapps/ROOT.war
08-Mar-2017 10:18:47.390 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
2017-03-08 10:18:47.874 [localhost-startStop-1] INFO o.s.web.context.ContextLoader - Root WebApplicationContext: initialization started
2017-03-08 10:18:47.898 [localhost-startStop-1] INFO o.s.w.c.s.AnnotationConfigWebApplicationContext - Refreshing Root WebApplicationContext: startup date [Wed Mar 08 10:18:47 UTC 2017]; root of context hierarchy
2017-03-08 10:18:47.995 [localhost-startStop-1] INFO o.s.w.c.s.AnnotationConfigWebApplicationContext - Registering annotated classes: [class com.github.swrirobotics.config.ApplicationConfig,class com.github.swrirobotics.config.JpaConfig,class com.github.swrirobotics.config.LiquibaseConfig,class com.github.swrirobotics.config.WebSocketConfig,class com.github.swrirobotics.config.SecurityConfig]
2017-03-08 10:18:49.500 [localhost-startStop-1] INFO o.s.b.f.a.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2017-03-08 10:18:49.606 [localhost-startStop-1] INFO c.g.s.config.JpaConfig - entityManagerFactory
2017-03-08 10:18:49.692 [localhost-startStop-1] INFO c.g.s.config.JpaConfig - JDBC driver: org.postgresql.Driver
2017-03-08 10:18:49.696 [localhost-startStop-1] INFO c.g.s.config.JpaConfig - JDBC URL: jdbc:postgresql://bagdb-postgres/bag_database
2017-03-08 10:18:49.698 [localhost-startStop-1] INFO c.g.s.config.JpaConfig - JDBC Username: bag_database
2017-03-08 10:18:49.730 [localhost-startStop-1] INFO c.zaxxer.hikari.HikariDataSource - HikariPool-1 - Started.
2017-03-08 10:18:52.860 [localhost-startStop-1] WARN o.s.w.c.s.AnnotationConfigWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in com.github.swrirobotics.config.JpaConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean]: Factory method 'entityManagerFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in com.github.swrirobotics.config.JpaConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method 'dataSource' threw exception; nested exception is com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The connection attempt failed.
2017-03-08 10:18:52.863 [localhost-startStop-1] ERROR o.s.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in com.github.swrirobotics.config.JpaConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean]: Factory method 'entityManagerFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in com.github.swrirobotics.config.JpaConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method 'dataSource' threw exception; nested exception is com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The connection attempt failed.
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1128)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1023)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1076)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:851)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:541)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4851)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:725)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:701)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:940)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1816)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean]: Factory method 'entityManagerFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in com.github.swrirobotics.config.JpaConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method 'dataSource' threw exception; nested exception is com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The connection attempt failed.
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
... 27 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource' defined in com.github.swrirobotics.config.JpaConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method 'dataSource' threw exception; nested exception is com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The connection attempt failed.
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1128)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1023)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:381)
at com.github.swrirobotics.config.JpaConfig$$EnhancerBySpringCGLIB$$fa0036a7.dataSource(<generated>)
at com.github.swrirobotics.config.JpaConfig.entityManagerFactory(JpaConfig.java:108)
at com.github.swrirobotics.config.JpaConfig$$EnhancerBySpringCGLIB$$fa0036a7.CGLIB$entityManagerFactory$1(<generated>)
at com.github.swrirobotics.config.JpaConfig$$EnhancerBySpringCGLIB$$fa0036a7$$FastClassBySpringCGLIB$$a7820396.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:356)
at com.github.swrirobotics.config.JpaConfig$$EnhancerBySpringCGLIB$$fa0036a7.entityManagerFactory(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
... 28 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.sql.DataSource]: Factory method 'dataSource' threw exception; nested exception is com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The connection attempt failed.
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
... 49 common frames omitted
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The connection attempt failed.
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:512)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:105)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:71)
at com.github.swrirobotics.config.JpaConfig.dataSource(JpaConfig.java:88)
at com.github.swrirobotics.config.JpaConfig$$EnhancerBySpringCGLIB$$fa0036a7.CGLIB$dataSource$0(<generated>)
at com.github.swrirobotics.config.JpaConfig$$EnhancerBySpringCGLIB$$fa0036a7$$FastClassBySpringCGLIB$$a7820396.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:356)
at com.github.swrirobotics.config.JpaConfig$$EnhancerBySpringCGLIB$$fa0036a7.dataSource(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
... 50 common frames omitted
Caused by: org.postgresql.util.PSQLException: The connection attempt failed.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:275)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:218)
at org.postgresql.Driver.makeConnection(Driver.java:407)
at org.postgresql.Driver.connect(Driver.java:275)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:341)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:506)
... 63 common frames omitted
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.postgresql.core.PGStream.<init>(PGStream.java:64)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:147)
... 71 common frames omitted
08-Mar-2017 10:18:52.866 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
08-Mar-2017 10:18:52.880 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
08-Mar-2017 10:18:52.889 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
08-Mar-2017 10:18:52.889 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [logback-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
08-Mar-2017 10:18:52.890 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [logback-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
08-Mar-2017 10:18:52.893 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /usr/local/tomcat/webapps/ROOT.war has finished in 8,629 ms
08-Mar-2017 10:18:52.894 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-apr-8080"]
08-Mar-2017 10:18:52.898 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 8681 ms
The database was designed with the intent that users should be able to apply arbitrary tags to bag files and search for them. Implement a nice way in the GUI to view, search for, and add tags.
Dear Developers,
I tried to imitate the project on my machine, Initially, I used JDK 11 as it was mentioned that we can use JDK 8.0 or higher, but then I ran into the below error:
Could not find artifact com.sun:tools:jar:11.0.3 at specified path /usr/lib/jvm/java-11-openjdk-amd64
Some answers suggested to add an exclusion in the dependency tree, but this didn't solve my problem.
Tools.jar has been deprecated after Java 8, so no longer supported.
And since exclusion methods failed so I decided to fall back to Java 8 version.
After that everything built smoothly. But now I ran into some different issue, the interface is looking as it should, but the BagScanner is not scanning files in the specified directory(/bags)
2019-07-03 16:22:59.998 [pool-2-thread-1] INFO c.g.s.bags.filesystem.BagScanner - Scanning for new bag files in /bags.
2019-07-03 16:22:59.999 [pool-2-thread-1] DEBUG c.g.s.bags.filesystem.BagScanner - Hey I am inside the getBagFiles method
2019-07-03 16:23:00.007 [pool-2-thread-1] DEBUG c.g.s.bags.filesystem.BagScanner - Hey I am inside the file reader: /bags
2019-07-03 16:23:00.008 [pool-2-thread-1] DEBUG c.g.s.bags.filesystem.BagScanner - Now printing by method 1:
2019-07-03 16:23:00.010 [pool-2-thread-1] DEBUG c.g.s.bags.filesystem.BagScanner - Found 0 bag files on disk.
2019-07-03 16:23:00.019 [pool-2-thread-1] DEBUG c.g.swrirobotics.bags.BagService - Scanning bags already in the database.
2019-07-03 16:23:00.020 [pool-2-thread-1] DEBUG c.g.s.bags.filesystem.BagScanner - 0 bags exist in the database and are not missing.
2019-07-03 16:23:00.021 [pool-2-thread-1] DEBUG c.g.s.bags.filesystem.BagScanner - 0 in the DB are missing on disk.
and if I try with different directory it gives this error:
2019-07-03 16:23:49.342 [pool-2-thread-1] DEBUG c.g.s.bags.filesystem.BagScanner - Done checking bag files.
2019-07-03 16:23:55.268 [http-nio-8080-exec-22] INFO c.g.s.bags.filesystem.BagScanner - Scheduling a quick directory scan.
2019-07-03 16:23:55.270 [pool-2-thread-1] INFO c.g.s.bags.filesystem.BagScanner - Scanning for new bag files in /home/fnc11/Hiwi/IKA/testBag.
2019-07-03 16:23:55.271 [pool-2-thread-1] DEBUG c.g.s.bags.filesystem.BagScanner - Hey I am inside the getBagFiles method
2019-07-03 16:23:55.272 [pool-2-thread-1] ERROR c.g.s.bags.filesystem.BagScanner - Error parsing directory:
java.nio.file.NoSuchFileException: /home/fnc11/Hiwi/IKA/testBag
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:427)
at java.nio.file.Files.newDirectoryStream(Files.java:525)
at com.github.swrirobotics.bags.filesystem.BagScanner.getBagFiles(BagScanner.java:368)
at com.github.swrirobotics.bags.filesystem.BagScanner$FullScanner.run(BagScanner.java:428)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-07-03 16:23:55.274 [pool-2-thread-1] DEBUG c.g.s.bags.filesystem.BagScanner - Now printing by method 1:
2019-07-03 16:23:55.276 [pool-2-thread-1] WARN c.g.s.bags.filesystem.BagScanner - Unexpected exception when checking bag files:
java.lang.NullPointerException: null
at com.github.swrirobotics.bags.filesystem.BagScanner.getBagFiles(BagScanner.java:381)
at com.github.swrirobotics.bags.filesystem.BagScanner$FullScanner.run(BagScanner.java:428)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
And these are the different versions of the technologies that I am using:
03-Jul-2019 16:19:03.166 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version: Apache Tomcat/9.0.2
03-Jul-2019 16:19:03.166 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Nov 25 2017 21:08:02 UTC
03-Jul-2019 16:19:03.166 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number: 9.0.2.0
03-Jul-2019 16:19:03.166 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
03-Jul-2019 16:19:03.166 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 4.15.0-54-generic
03-Jul-2019 16:19:03.167 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
03-Jul-2019 16:19:03.167 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib/jvm/java-1.8-openjdk/jre
03-Jul-2019 16:19:03.167 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 1.8.0_151-b12
03-Jul-2019 16:19:03.167 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Oracle Corporation
03-Jul-2019 16:19:03.167 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /usr/local/tomcat
03-Jul-2019 16:19:03.167 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /usr/local/tomcat
The bag database is currently really only suitable for internal use because any user can modify bag metadata. If the bag database is going to be made suitable for a public-facing site, it needs a better way to control who has access to things. The database schema is already designed to support user accounts, but the front end will need a lot of work.
I expect the following need to be implemented:
Operations that should probably be restricted:
@matt-attack found an issue where if a topic is specified in the bag database's settings as being a GPS topic, but the message type on that topic does not have latitude and longitude fields, the Bag DB will throw an exception when trying to index that bag file and fail to add it.
A few people are using the bag database to programmatically search for and retrieve bag files. It has a relatively straightforward REST API, but it would be nice to have some proper documentation for it so that people don't have to go reading through Java/Spring code to figure it out.
If a bag file contains topics that have known Image types, we should be able to extract those images and display them to the user.
Hopefully we can do this reasonably fast in real time. We'll need to:
Sometimes you'd like to be able to do something like set the vehicle name for many bag files at once. That should be possible for any user-enterable field.
Hello,
in the readme it is mentioned that the bag-database can also be running behind e.g. haproxy.
In my case I would like to have it running behind an apache webserver for authentication and at a suburl like https://servername/bag
The issue I have is that the page tries to load the resources from https://servername/resources , not https://servername/resources/bag
Is there some value to change the base url? Or is there a change needed in the proxy?
currently my snippet looks like this
<Location /bag/>
SSLRequireSSL
SSLOptions +StdEnvVars
AuthType openid-connect
require valid-user
Options Includes FollowSymLinks
AllowOverride AuthConfig Limit
Order allow,deny
Allow from all
ProxyPass http://127.0.0.1:8080
ProxyPassReverse http://127.0.0.1:8080
</Location>
</VirtualHost>
Currently there is a hard-coded list of supported GPS message types and topics that will be examined. It'd be nice if this was configurable.
I tried to add some tags to bags I recorded, as stated in the features list:
Tagging: Bags can be tagged and searched for with arbitrary metadata strings. Existing tags on arbitrary metadata topics in bag files will be automatically read.
I started the bag database using:
-e METADATA_TOPICS="/metadata" \
on the docker run command, made sure rosbag record subscribed to /metadata and then typed:
rostopic pub /metadata std_msgs/String "name: John Doe". I have also tried both leading and trailing newlines.
The message was recorded in my bag and can be seen using rqt_graph. But when I copy the bag file into the Bag Database, the tags are not shown. The Interface correctly shows that something has been published to /metadata but the tags column on the right hand side doesn't list "name: John Doe" and I am unable to find anything searching for this tag.
Is there an error on my side during the process or is it even supposed to work that way?
It's a pain to have to reset your column configuration every time you reload the page. It's be nice if they could be preserved, probably through setting a cookie.
Settings that should be preserved:
When downloading bags from using the download button, Firefox identifies the file type as "HTML document", which I assume is because the type isn't being specified correctly in the header.
Hi,
would it by possible to add the functionality to read several key:value pairs from a specified topic?
I think of using a std::msgs/String message similar to the vehicle_name topic. But instead of providing just a single word, I would like to pass a whole dict of key:value pairs, possibly in yaml format:
General:
CreatedBy: user
Description: "A drive through the park"
Environment:
Location: Some Park
Weather: Sunny
Platform:
VehicleName: ...
Sensors: Cameras...This schema could even be assumed fixed, through e.g. a config file. I don't have enough experience with java to be able to fully understand how the database is build up. Could someone give me an estimate of how complicated that would be? (e.g. is it possible with the way the DB is build up right now)
The advantage of having such a functionality is, that you could insert this metadata into the bagfile at record time (through additional tooling, like rosbag_metadata). Forcing the user to specify data at the time of recording, and placing the metadata in the same file as the recording itself has several advantages, e.g. (you could access this information directly from the console, you don't have to rely on people providing the metainformation afterwards, when uploading the file to the database...)
I put 3 bags in the mounted volume (default is /bag/location), the explorer shows properly. However, when I delete one of them and refresh the explorer, all 3 bags are still there even though the remote folder (/data) inside the container remains 2 files.
If the volume is only used for uploading purpose, it would be better to allow deleting files from screen.
Currently, the bag database monitors a directory on a filesystem somewhere and assumes that users can add bag files to that location through some other means such as SFTP or Samba.
It would be nice to have a way in the GUI to upload bag files to that location, and maybe even drag-and-drop several bags at once into the GUI.
Currently it always looks at "/vms/vehicle_name".
Is it possible to make an upload button to the interface ? It would allow admin/users to upload the files instead of directly access the folder or using FTP. It would be good if you are welcome pull request.
CSRF tokens are generated when the page is loaded and submitted along with POSTs to prevent certain security issues. If a user has left their browser open for a long time, their token may have expired, and they have to refresh the page in order to make any more requests.
Look into what's necessary to occasionally get a new token and how long it takes to expire.
Hi,
Please also support message when position stored in filed position (message geographic_msgs/GeoPoint)
Is it possible to filter bag files by more than one parameter? For example, to filter bag files that have two set topics or a set topic and a tag.
Hey,
I was wondering if it is possible to manually add vehicle names and location? This is supported by the UI (both fields are editable and there is a save button). However after editing, pressing save and reloading the page, the entered vehicle name and location is gone.
I'm using a docker installation with BagDatabase 2.6.
It'd be super useful to be able to get a direct download link (e.g. to use in a test with wget) from the webui!
It'd be nice to have some continuous integration going on here.
As noted in a comment, the change at 22dac9c inadvertently removed a line from the vehicle name updater that actually saved the name to the bag file. This would result in doing a name scan from the admin console having no actual effect.
Turns out that there's a bug in the recent code added for the folder view that can cause the tabs to be blank after reloading the page.
This happens because it tries to save the current state of the page and reload it upon reloading; it's saving the id of the tab you have selected, and if the tab has a different generated id after reloading (which it will if you switch in and out of admin mode), it will cause the tabs to fail to load at all.
Currently it's possible to display the first image on a topic, but it'd be really handy to be able to stream the topic as a video.
The endpoint used to download bag files expects to be passed a database ID. The "Copy Link" context menu item in the List View puts the id in there as expected, but the Folder View puts the full path to the bag in there, which won't work at all.
Doing a full-text search of both of those at once is very slow. It'd be faster to search for them individually and then just combine the results, but that implies there's something going wrong with the joins in the SQL queries that is making them suboptimal. Figure out what's up with that.
It seems like if the database encounters a SQL exception when parsing and inserting a new bag file, that can kill the thread that scans for new files, and that will prevent it from picking up new files again until it's restarted. I haven't reliably reproduced this yet, but I have seen it multiple times.
Early versions of the bag DB simply used Hibernate to set up the database schema, which is bad because it's hard to track changes between versions and migrate the database properly.
It'd be better to use a tool like Liquibase to handle database schema changes.
Currently, for the sake of having clean code and making development easier, all of the Javascript classes are spread out into different files. It will make loading the page more efficient if we can concatenate them all together and minify them when generating a releasable war.
I ran bagbunker scan /mnt/bags, which seems to have successfully scanned my bag files. After that was done, I was able to go to its web page and see the listing of bag files.
After that, I ran bagbunker read-pending, which threw an exception when it encountered an unindexed bag file:
2016-07-25 21:48:51,622 ERROR marv.storage skipped <Fileset "2014-07-25-15-37-22.orig" type="bag" dir="/mnt/bags" files=1 md5="35f259cfdb50ffcaf37454b92b19dda0"> due to exception
Traceback (most recent call last):
File "/opt/bagbunker-venv/local/lib/python2.7/site-packages/marv/storage.py", line 106, in read_pending
instance = reader[fileset.type].read(fileset)
File "/opt/bagbunker-venv/local/lib/python2.7/site-packages/marv/reader.py", line 35, in read
return self.callback(fileset)
File "/opt/bagbunker-venv/local/lib/python2.7/site-packages/bagbunker/reader.py", line 44, in reader
rbag = rosbag.Bag(file.path)
File "/opt/ros/indigo/lib/python2.7/dist-packages/rosbag/bag.py", line 171, in __init__
self._open(f, mode, allow_unindexed)
File "/opt/ros/indigo/lib/python2.7/dist-packages/rosbag/bag.py", line 1094, in _open
if mode == 'r': self._open_read(f, allow_unindexed)
File "/opt/ros/indigo/lib/python2.7/dist-packages/rosbag/bag.py", line 1130, in _open_read
self._reader.start_reading()
File "/opt/ros/indigo/lib/python2.7/dist-packages/rosbag/bag.py", line 2304, in start_reading
raise ROSBagUnindexedException()
ROSBagUnindexedException: Unindexed bag
That seems to have left the database in a bad state; whenever I go to bagbunker's web page, I simply see this exception:
(pysqlite2.dbapi2.OperationalError) no such table: listing_entry [SQL: u'SELECT listing_entry.id AS listing_entry_id, listing_entry.remote AS listing_entry_remote, listing_entry.fileset_id AS listing_entry_fileset_id, listing_entry.storage_id AS listing_entry_storage_id, listing_entry.type AS listing_entry_type, listing_entry.name AS listing_entry_name, listing_entry.abbr_md5 AS listing_entry_abbr_md5, listing_entry.size AS listing_entry_size, listing_entry.file_count AS listing_entry_file_count, listing_entry.status AS listing_entry_status, listing_entry.job_count AS listing_entry_job_count, listing_entry.comment_count AS listing_entry_comment_count, listing_entry.tags AS listing_entry_tags, listing_entry.starttime AS listing_entry_starttime, listing_entry.endtime AS listing_entry_endtime, listing_entry.duration AS listing_entry_duration, listing_entry.robot AS listing_entry_robot, listing_entry.use_case AS listing_entry_use_case \nFROM listing_entry']
It seems strange that this would cause a "no such table" error, so I can't be 100% certain that the unindexed bag file is what caused it, but I don't see any other errors or warnings in the log.
I realize the obvious solution to this problem is "run rosbag reindex on your bag file!", but I actually intentionally keep around this unindexed bag file so that I can test tools I am developing to ensure that unindexed bag files do not break them. 😄 So, it would be nice if bagbunker also handled it gracefully.
hi I'm using the bag-database and it's really helpful.
Can it build index for lz4 compressed rosbags,
thx.
MapQuest has disabled access to their tile API (http://devblog.mapquest.com/2016/06/15/modernization-of-mapquest-results-in-changes-to-open-tile-access/). That's pretty inconvenient.
I should probably disable MapQuest-related config options entirely, and maybe add the ability to configure an arbitrary tile server in order to support services like ArcGIS or Stamen.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.