usdot-jpo-ode / jpo-ode Goto Github PK

US Department of Transportation (USDOT) Intelligent Transportation Systems Operational Data Environment (ITS ODE). This is the main repository that integrates and coordinates ODE Submodules.

Shell 0.34% Batchfile 0.06% Java 96.16% HTML 2.35% JavaScript 0.11% CSS 0.06% C 0.16% Dockerfile 0.17% Makefile 0.03% Python 0.57%

data-routing docker transportation

jpo-ode's People

Contributors

Stargazers

Watchers

jpo-ode's Issues

PPM Containers Exiting

The PPM containers are exiting occasionally.

Today the PPM TIM container exited at 16:35 UTC, and at that same time Zookeeper was throwing "Got user-level KeeperException when processing" messages.

Attached are the Zookeeper and PPM TIM container logs, and also the results from the "docker inspect" command on the PPM TIM container showing when exactly it started and exited.

ppmContainerInspect.txt
ppmContainerLogs.txt
zookeeperContainerLogs.txt

Thanks!
Kim

Strange Pattern in TIM data frequency

There is a strange pattern in the TIM frequency that WYDOT has in their Oracle database. For most days, TIM records come in pairs separated by a few milliseconds, sometimes with the exact same "odereceivedat" time stamp. The pairs come in at random times, minutes to hours apart, but they are consistently in these pairs. Only one of the messages in each pair gets submitted to the SDC firehose endpoint. This might not be an ODE issue, but it is worth looking into to cover all bases.
2018-10-24TimDataWYDOT.xlsx

Suggestion: Metadata value types.

In the metadata section, the sanitized element is a boolean; the validSignature is a string with boolean value. A reason for this inconsistency should be provided if it is needed. If not, consider standardizing value types.

{
"schemaVersion": 3,
"metadata": {
"schemaVersion":3,
"recordGeneratedBy": "OBU",
"recrodGeneratedAt": "2016-08-10T16:35:45.138Z[UTC]",
"logFileName": "bsm.uper",
"payloadType": "us.dot.its.jpo.ode.model.OdeBsmPayload",
"recordType": "bsmRxRecord",
"serialId": {
"streamId": "f9c7635a-7763-4e21-bece-cb1104f143b9",
"bundleSize": 1,
"bundleId": 12,
"recordId": 5,
"serialNumber": 0
},
"odeReceivedAt": "2017-09-26T20:00:08.48Z[UTC]",
"sanitized": false,
"validSignature": "true"
}
},

Log file parsing error

This issue was reported by @TonyEnglish over email.
We are finding a bit of problem with log processing through the ODE. This is happening with many files, but I would like to focus on the a single file (included). I have included a sample file with ODE logs (log.txt), original Lear file, and decoded file from the ODE. We are seeing most lines not getting processed. We have decoded the file with the Lear tool that has all the records, so I think the originating file is okay.

RSU Monitor end-point likely has a memory leak

ODE container periodically runs out of memory on the WYDOT server. By running “docker stats” we are seeing that the ODE container is constantly creating PIDs (well over 1000 after a few hours). Attached are the results from “docker stats.” The top container is the ODE. Whe this happens, the ODE container crashes and the logs show errors like “There is insufficient memory for the Java Runtime Environment”. Observing docker stats, it appears that as the ODE processes RSU queries using these instructions, the memory usage grows: https://github.com/usdot-jpo-ode/jpo-ode/wiki/Query-RSU-for-set-TIMs

We have a process to query RSUs repeatedly to check that they have the TIMs that we think they should have. When we shut this process off overnight, the ODE container memory usage goes down, and the PIDs from docker stats stop increasing at the rate they had before.

Attached are some graphs showing memory usage. The top graph shows the ODE’s memory increasing until RSU query process is turned off, and going back up once resumed. The bottom graph shows the total server memory during the same time period. This shows exactly when RSU monitor is turned off. It’s a very slow increase in memory, but it is evident.

First TIM after startup of ODE does not work

WyDOT has reported that the first Traveler Information Message sent to the ODE's deposit REST endpoint /tim does not appear in the output Kafka streams. This requires that the first message be sent twice, all subsequent messages appear to work.

Different content types for TIMs

TIMs have different content options: advisory, workZone, genericSign, speedLimit, exitService. It looks like right now the ODE is only supporting "advisory."

Attached is a TIM sent to the ODE with a content of "speedLimit" as well as the exception seen in the ODE logs.

Thanks!
Kim

tim.txt
contentError.txt

TIMs with ITIS code text not supported

WYDOT needs the ability to send TIMs with custom ITIS text. This will at a minimum be done when there aren't ITIS codes that match what comes from WYDOT's TMC. Some text that I've gotten from WYDOT that doesn't seem to match any ITIS codes includes:

Traffic being diverted onto interchange ramps
Traffic being diverted onto shoulder. Expect delays
Traffic being diverted onto [HIGHWAY]

The ODE currently supports ITIS codes from broadcast TIMs, but not custom text.

Thanks!
Kim

TIM messages: speed, lat, lon sometimes arrive as strings, not floats

On CV PEP side - we receive some TIM messages with either speed, or latitude, or longitude wrapped in quotation marks. The frequency of incorrectly formatted messages is reasonably high - about 5% or so.

Here are several messages - both correct and incorrect; if you copy-paste there should be one message per line. Please let us know if you need any more info.

Thanks!
-v

TIM metadata version 5 having undocumented fields

We checked some TIM messages we received in October, and there is a "request" structure present in "metadata". The structure is not documented for schema version 5. We believe the version needs to be updated to 6 and documentation to include this addition.

"request": {
"ode": {
"verb": 0,
"index": 3,
"version": 2
},
"rsus": {
"rsuTarget": "10.145.12.27",
"rsuUsername": "xxx",
"rsuRetries": 3,
"rsuTimeout": 5000,
"rsuPassword": "xxx"
},
"snmp": {
"mode": 1,
"deliverystop": "2019-01-01T00:00:00-06:00",
"rsuid": "00000083",
"deliverystart": "2018-01-01T00:00:00-06:00",
"enable": 1,
"channel": 178,
"msgid": 31,
"interval": 2,
"status": 4
}
},

ODE Slowdown issues

While uploading some of our larger data files through the web interface I have run into issues where the ODE slows to a crawl trying to process all of the records. It doesn’t completely crash, but it will go from running at its “normal” pace to processing one record every few minutes (seen from console output). If I try refreshing the web interface during this slow state I get a timeout error.

Once when looking into this I was able to find a “GC overhead limit exceeded” exception in one of the log files. I don’t remember which log file this came from unfortunately. We restarted the server at that point to get the ODE working again.

Attached is a zip file with our testing files and a screenshot of the log file with the GC error. You might be able to reproduce this behavior by uploading all of the files through the website, one after the other.
bsmFiles.zip

DOCKER_HOST_IP setting not documented

Setting of the DOCKER_HOST_IP is not well documented as part of the install instructions.

Maven build error

Trying to build using maven I get the attached failure.

Here's my maven -version:

Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T11:41:47-05:00)
Maven home: /opt/apache-maven-3.3.9
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-oracle/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-24-generic", arch: "amd64", family: "unix"

install_failure.txt

Decode a pcap file

Hi,

I'm able to successfully build and decode the messages from any of the supported files. I was tryig to decode a pcap file(packet capture). Is it possible to decode a pcap file? Please help me on the same.

TIM Input JSON samples don't work

They currently have the incorrect values for crc. Should be a bit string

Error: MessageProducer -Error Sending record.org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

Hi,
I am able to successfully build,deploy and run the project. But the files which are being uploaded are decoded to produce messages on the webUI. It throws the above error. uploaded files data is getting published to the kafka topic. I followed the same instructions. Please help on the same

ODE_Output_Schema_Reference BSM locationData data as strings?

In:

https://github.com/usdot-jpo-ode/jpo-ode/blob/develop/docs/ODE_Output_Schema_Reference.docx

The details in the locationData section are stored as string, why are these not stored as floats? especially the lon/lats.

Git flow

It looks there are ways to integrate git into eclipse to help start integrating gitflow into your regular workflow. http://www.eclipse.org/egit/. Alternatively, the git-cli is fairly easy to master.

testIssue

for bugbot slack integration, please ignore

Fraction of records received by ODE are deposited to SDC data warehosue.

There is significant discrepancy between the number of BSM messages that are making it into the WYDOT TMC’s Oracle DB and those making it into the SDC data warehouse. (roughly 10 times more messages in the Oracle).

Looking into the S3/Firehose depositor logs (covering approximately 10 minutes of operation):

Currently it appears to be around ~95 minutes of delay between when a message is received by ODE and when Firehose receives it
In 10 minutes of operation the only 2 seconds worth data were received by SDC
-- First message: received: 14:35:20, processed: 16:10:42
-- Last message: received: 14:35:22, processed: 16:19:04
Throughput appears to be approximately records per second. This is not adequate for the projected full loads of hundreds of messages per second
Exceptions are logged when communicating to the Firehose and possibly as a result messages are dropped as there are no retries. The rate of lost connections however does not account for the amount of data loss.

Delta Degree Support in TIMs

I wanted to try out sending different deltas in the TIMs for the path nodes. When I try sending:

"path": {
"type": "xy",
"nodes": [
{
"nodeLong": "0.00006",
"nodeLat": ".000015",
"delta": "node-XY2"
},
{
"nodeLong": "0.000548",
"nodeLat": "-.0000015",
"delta": "node-XY2"
},
{
"nodeLong": "0.000509",
"nodeLat": "-0.000007",
"delta": "node-XY2"
}
]
}

I get back:

Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "node-XY2" (class us.dot.its.jpo.ode.plugin.j2735.timstorage.Delta), not marked as ignorable (2 known properties: "node-LL3", "node-LatLon"])

Is this the correct format for sending deltas?

Querying RSU from ODE not working

Reported by Tony E, the querying call /tim/query is not working specifically for the RSU that he has connected at his desk. We've noticed that the command line snmpget call works, but its not through the API. We've also confirmed that the rest of the network connected WyDOT RSUs seem to work, but this one does not.

Driver Alert files don't have correct data

Driver alert files are not getting decoded with the correct data. Received message details all have values of zero, and the record generated at date is the same for each record in a file.

Attached are the files that we observed this with.

Thanks!
Kim

driverAlert_1548196627_fe80%3A%3A226%3Aadff%3Afe05%3A1911.csv.gz
driverAlert_1548196849_fe80%3A%3A226%3Aadff%3Afe05%3A1911.csv.gz

TIMs not being received by HMI, Distribution Type field not being correctly encoded

Overview

WyDOT reported that Traveler Information Messages to be distributed via satellite are not being received/displayed by the OBU HMI. These expected flow of these messages looks like: ODE --> SDW --> OBU --> HMI. We know from querying that the messages are successfully being received by the SDW.

It was noticed that the distType field in the AdvisoryDetails ASN header is not being encoded as expected, this difference may cause messages to either not be distributed by the SDW or considered valid by the OBU.

Additional Considerations

The R8 and R9 versions of the specification for Traveler Situation Data messages defines this field with different values.

R8:

distributionType
--indicates   how the ASDM is distributed; 1=rsu, 2=ip, 4=satellite, 8=reserved

R9:

distributionType
--indicates   how the ASDM is distributed; 1=rsu, 2=ip

The latest Southeast Michigan (SEMI) J2735 extension ASN.1 specification that governs communication schema with the SDC/SDW defines values for this field are as follows:

DistributionType ::= OCTET STRING (SIZE(1))
	--	none		(0),	"00000000", not intended for redistribution
	--	rsu		(1),	"00000001", intended for redistribution over DSRC
	--	ip		(2),	"00000010" intended for redistribution over IP

It appears as though "satellite" would be the most appropriate option for this field, but this is not present in the most recent (R9) specification.

Additionally, WyDOT has reported that messages created and encoded outside of the ODE specify distType to 1, and those messages do appear on the HMI. This is inconsistent with the documentation.

Technical Details

Messages sent to the ASN1c encoder are in XML form, and return as an encoded hex dump. When decoding the resultant hex, the issue can be seen:

Input:
<distType>2</distType>

Expected Output:
distType '02'H,

Actual Output:
distType '20'H,

Lev mentioned in a reply to an issue with ASN1c reported on Stack Overflow that OCTET STRING variable types are printed in byte order.

For example, an input integer value of 2 looks like this in binary form:

0000 0010

However, when encoded into hex, the two nibbles are read in opposite order:

0010 0000

This seems to be why an input value of 0x02 becomes an output value of (in hex) 0x20.

Multiple .gitignore files

Typically a repository will only have 1 .gitignore file at the root of the directory, especially when the repository is primarily one language.

Docker latest tagging pulling untested version

The Docker latest tags are now pulling Ubuntu 18.04 and Java 10.

Incompatibility issues have been observed in the ODE with Spring applications that use Java 8, as well as the jpo-cvdp module which uses gcc 4.9.

TMC Data Logs Needed

re:[[email protected]]

We are working to add the TIMs generated at the TMC to the SDC and finding the message format is not consistent. We are getting this by subscribing to the BroadcastTIM topic. How can we best fix this so that we have a consistent TIM messaging to the SDC. We are seeing differences in the TIM data itself as well as the header.

JSON on Metadata page does not parse.

Attempting to use the JSON examples for TIM and BSM and it looks like there is a bracket mismatch in both of them.

TIM units and conversions

I've trying to figure out what units the ODE is expecting from TIMs.

For example, the ODE is validating elevation based on meters (-409.5 to 6143.9) while the standard asks for 10 cm units. Doing this conversion before sending the TIM can put the elevation out of bounds.

Also the standard specifies lat/longs in .1 microdegrees, should the ODE be responsible for the conversion or should that fall on the user?

I've attached JSON that I've generated that deposits a TIM to an RSU, the nodeLat and nodeLongs are in .1 microdegrees with deltas that match.

Thank you!
Kim
timJson.txt

ODE missing driver alert location metadata

Reported by WyDOT, driver alert messages are missing the location fields highlighted it metadata documentation

jpo-ode/docs/ODE_Output_Schema_Reference.docx

in the ode-769 branch the ODE_Output_Schema_Reference.docx is updated to reflect the new changes to the BSM, but I think we forgot to update the schemaVersion metadata tag to 5?

In:
Chapter 7.3.1. BSM from bsmTx.
Chapter 7.3.2 BSM from bsmLogDuringEvent

and in Chapter 7.3.3 BSM from rxMsg there is no schemaVersion metadata tag.

PPM container exited

Our PPM container went down today around 18:28:41Z, I grabbed the logs for it and zookeeper at this time and attached them.

High memory consumption from two modules

We noticed two modules taking up a lot of memory last week:

PPM BSM module - 16GB
CVPEP (SDC) BSM Depositor - 8GB

This was during a huge data dump, and crashed the ODE server.

TIM Message Frames

We heard from Lear that RSUs are expecting TIMs to come with message frames. Currently TIMs generated from the ODE deposit REST endpoint are encoded as "TravelerInformation" objects, can this be adjusted to include the message frame wrapper as well?

Thanks!
Kim

ODE file parse error

I recently tried uploading about 2500 files into the ODE at once. One file failed, and the ODE hung trying to parse it until I removed the file and restarted the ODE. Attached is the problem file, and below is the exception.

Thanks!
Kim

us.dot.its.jpo.ode.importer.parser.FileParser$FileParserException: Error parsing bsmLogDuringEvent_1509403346_2620:31:40e0:876:226:adff:fe05:1521.csv
at us.dot.its.jpo.ode.importer.parser.BsmLogFileParser.parseFile(BsmLogFileParser.java:77)
at us.dot.its.jpo.ode.coder.stream.LogFileToAsn1CodecPublisher.publish(LogFileToAsn1CodecPublisher.java:51)
at us.dot.its.jpo.ode.coder.FileAsn1CodecPublisher.publishFile(FileAsn1CodecPublisher.java:35)
at us.dot.its.jpo.ode.importer.ImporterProcessor.processAndBackupFile(ImporterProcessor.java:73)
at us.dot.its.jpo.ode.importer.ImporterProcessor.processDirectory(ImporterProcessor.java:45)
at us.dot.its.jpo.ode.importer.ImporterDirectoryWatcher.run(ImporterDirectoryWatcher.java:56)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: us.dot.its.jpo.ode.importer.parser.FileParser$FileParserException: Error parsing step 6
at us.dot.its.jpo.ode.importer.parser.LogFileParser.parseStep(LogFileParser.java:101)
at us.dot.its.jpo.ode.importer.parser.BsmLogFileParser.parseFile(BsmLogFileParser.java:71)
... 10 common frames omitted
Caused by: java.lang.IndexOutOfBoundsException: null
at java.io.BufferedInputStream.read(BufferedInputStream.java:338)
at us.dot.its.jpo.ode.importer.parser.LogFileParser.parseStep(LogFileParser.java:84)
... 11 common frames omitted

bsmLogDuringEvent_1509403346_2620%3A31%3A40e0%3A876%3A226%3Aadff%3Afe05%3A1521.csv.gz

Documentation for Private Repo isn't clear

There are no clear instructions on getting the private repo and and running. This is is related to needing a public ANS.1 compiler

Docker container auto-restart on reboot

Currently the ODE does not have a mechanism for auto-restarting Docker containers on system restart.

Metadata document name recommendation.

People (I am doing it now) will link to the Metadata_v3.md document. If you change the version number their link will break. Could you name the current version Metadata.md (or similar) and just change that file. These are version controlled. If you want to keep all copies, then you could just "retire" old versions as the next higher number in the retirement pool.

AEM/ADM Containers Exiting

The AEM/ADM containers occasionally exit, I think this might be related to connection issues with Zookeeper and Kafka.

I've been able to replicate this a few times by stopping and restarting the Zookeeper and Kafka containers. Logs are attached for reference.

Thanks!
Kim

Need jpo-ode-private Dependency

I'm going through the Build Process outlined in the README.md; however, I'm stuck at Step 1 that requires the jpo-ode-private repository. If I try to clone it, it requests credentials. Please assist. Thank you.

Unit test Code coverage

As mentioned by #5, it looks the unit tests are only hitting 3.5% of the code by SonarQube's account. Within SonarQube it looks like your thresholds are set to 65%.

A colleague who conducted a manual code review had a similar conclusion.

Cross Origin Requests for hte External URL

I’ve been working on a simple website for sending a TIM to an RSU. I found that the ODE was blocking my website from making the /tim POST and was giving me a “cross origin request blocked” error.

Unable to send multiple TIMs to an RSU

The TIM unique message id comes from the “index” of the Snmpset commands, which is the “.X” at the end of each flag. For example, rsuSRMPayload.3 would create “activemsg_3.conf” on the RSU, and rsuSRMPayload.4 creates “activemsg_4.conf.” The TIMs we’ve been sending through the ODE seem to default to 3. This is only allowing us to send one TIM to an RSU, because sending a TIM with an index that is the same as an existing TIM on an RSU fails.

Thank you!
Kim

TIM UTC Start Date Time

Broadcast TIMs sent to the ODE with a start time in UTC seem to incorrectly convert the MinutesOfTheYear start time that gets sent to RSUs and satellite.

For example:

startDateTime: “2018-06-11T16:00:00.000Z” yields a startTime of 232800

startDateTime: “2018-06-11T10:00-06:00” yields a startTime of 232440

I think 232440 is the correct one based on time calculator websites.

This is not critical because we can keep using local time for now, but it might be nice to have the option of using UTC in the future.

Non-proprietary code base (ASN.1)

It is our intent to phase out any proprietary elements of the ODE code base so it is entirely non-proprietary. This will enhance user experience (one open code base - no pieces sitting behind a closed code repo that need to be integrated) and replicability (anyone can use the software regardless of their ability to purchase a particular license). The main obstacle right now is the ASN.1 encoder/compiler. I'd like to start a dialog about options for replacing the current proprietary solution with an open source one, while retaining the performance and modularity of the software. Is there existing code that can be modified to meet our needs? Does something need to be developed from scratch? Please discuss!

TIM to RSU REST Request not returning successful SNMP bindings

I'm trying to send a TIM to the RSU in our office using the TIM REST request with a JSON body.

When I use the the "admin" for the RSU username and "password" for the password it is returning:

{SDW={success: true, message: "Deposit to SDW was successful"}, 192.168.0.145={success: true, message: "SNMP deposit successful. RSU IP = 192.168.0.145, Status Code: 0"}}

When “v3user” is used for the username the response is:

{SDW={success: true, message: "Deposit to SDW was successful"}, 192.168.0.145={success: false, message: "SNMP deposit failed for RSU IP 192.168.0.145, error code=5(General variable binding error)"}}

In either case I am not seeing the TIM getting created on the RSU. I am not sure if this could be an issue with my JSON, and I'm also not sure which username is correct.

Thanks!
Kim

Linking SonarQube to Travis

Here is the documentation around hooking SonarQube to Travis.
https://docs.travis-ci.com/user/sonarqube/

Potential Getting Started Readmes to take a look at

I crowd sourced a couple of examples of good local development Readme docs if you and to take a look. While I personally prefer to have the getting started on the readme, some repos link to a separate doc from the readme.

https://github.com/18F/micropurchase/blob/develop/docs/local_development.md
https://github.com/18F/tock#getting-started
https://github.com/18F/epa-notice
https://github.com/18F/openFEC#set-up

PacketIds are larger than Integer types

Hi there,

When I upload dnMsg files through the ODE I'm seeing PacketIds coming out that are much larger than ints, (hex string "EC9C236B0000000000", converts to decimal 4364682555337299984384). Can the packetId in the J2735TravelerInformationMessage class be changed to a datatype large enough to accommodate?

Thanks!
Kim

BsmSource not in BSM Metadata

We noticed that the "bsmSource" field is missing from BSM files.

Thanks!
Kim

usdot-jpo-ode / jpo-ode Goto Github PK

jpo-ode's People

Contributors

Stargazers

Watchers

Forkers

jpo-ode's Issues

Overview

Additional Considerations

Technical Details

Recommend Projects

Recommend Topics

Recommend Org