Job fails for a Python process due to RowTime field being null. about amazon-kinesis-data-analytics-examples HOT 4 CLOSED

I was also getting the same error on Ververica Platform CE. I noticed that the column names in the tables I created are not the same as the names of the fields from Kinesis. Even the difference between upper and lower case letters matters.

from amazon-kinesis-data-analytics-examples.

I guess I found a workaround by using an additional timestamp that is provided by the Kinesis-Connector and use it for the watermark:

CREATE TABLE {0} (
    ....
    StreamInsertTime TIMESTAMP_LTZ(3) METADATA FROM 'timestamp' VIRTUAL,
    WATERMARK FOR StreamInsertTime AS StreamInsertTime - INTERVAL '10' SECOND
)

from amazon-kinesis-data-analytics-examples.

Closing it as the issue seemed to be related to this line ts_time AS TO_TIMESTAMP(FROM_UNIXTIME(ts)), which yielded null, it should be ts_time AS TO_TIMESTAMP(FROM_UNIXTIME(ts/1000)), instead.

from amazon-kinesis-data-analytics-examples.

I am facing the same error. If the TIMESTAMP-Field is not available or it contains an invalid timestamp ( like a random string ), then this error appears.

The problem even gets bigger after deploying it to AWS Kinesis Analytics. Everytime an event with invalid timestamp comes in, the App crashes with this error. Then it restarts and crashes again on the same event. Also the following settings do not solve the problem

      'json.fail-on-missing-field' = 'false',
      'json.ignore-parse-errors' = 'true'

A solution would be to filter out invalid events with AWS Lambda, but is there no easy solution for this to ignore such events?

from amazon-kinesis-data-analytics-examples.