Comments (4)
I was also getting the same error on Ververica Platform CE. I noticed that the column names in the tables I created are not the same as the names of the fields from Kinesis. Even the difference between upper and lower case letters matters.
from amazon-kinesis-data-analytics-examples.
I guess I found a workaround by using an additional timestamp that is provided by the Kinesis-Connector and use it for the watermark:
CREATE TABLE {0} (
....
StreamInsertTime TIMESTAMP_LTZ(3) METADATA FROM 'timestamp' VIRTUAL,
WATERMARK FOR StreamInsertTime AS StreamInsertTime - INTERVAL '10' SECOND
)
from amazon-kinesis-data-analytics-examples.
Closing it as the issue seemed to be related to this line ts_time AS TO_TIMESTAMP(FROM_UNIXTIME(ts)),
which yielded null, it should be ts_time AS TO_TIMESTAMP(FROM_UNIXTIME(ts/1000)),
instead.
from amazon-kinesis-data-analytics-examples.
I am facing the same error. If the TIMESTAMP-Field is not available or it contains an invalid timestamp ( like a random string ), then this error appears.
The problem even gets bigger after deploying it to AWS Kinesis Analytics. Everytime an event with invalid timestamp comes in, the App crashes with this error. Then it restarts and crashes again on the same event. Also the following settings do not solve the problem
'json.fail-on-missing-field' = 'false',
'json.ignore-parse-errors' = 'true'
A solution would be to filter out invalid events with AWS Lambda, but is there no easy solution for this to ignore such events?
from amazon-kinesis-data-analytics-examples.
Related Issues (19)
- S3ParquetSink example
- Run Python Process Failed - Aagain HOT 1
- Issue with Pom dependency for getting started project
- Running python example "GettingStarted" with flink-sql-connector-kinesis_2.12-1.13.2 and amazon-kinesis-sql-connector-flink-2.3.0 HOT 1
- java.lang.IllegalArgumentException: Invalid lambda deserialization
- Beam example fails to compile because of unused avro code in directory
- Kafka example instead of Kinesis Data Stream
- Run python process failed when running tumbling-window.py on KDA HOT 23
- Python examples: please clarify which kinesis connector JAR should be used HOT 1
- As long as java 11 or above, then fine
- Scala NoSuchMethodError: 'scala.collection.mutable.Buffer (and all other Scala packages) HOT 1
- Issue with the Beam example documentation
- java.io.NotSerializableException: Non-serializable lambda
- Beam examples use deprecated IO API HOT 1
- Couldn't deploy python getting started example HOT 4
- Kafka example at https://docs.aws.amazon.com/managed-flink/latest/java/example-msk.html is not working
- Issue with S3 Sink parquet format using PyFlink HOT 3
- scala examples do not work for packaging applications which use the TableAPI
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from amazon-kinesis-data-analytics-examples.