uralian / ignition Goto Github PK

Need to implement a step for importing data from a jdbc-compliant database.

Implement the following listeners:

• Step listener to be notified on step computation
• Flow listener to be notified when flow starts/stops
• Stream data listener to be notified on each batch

Need to implement a step for writing data into a JDBC-compliant database.

Need to provide both options for further deployment:

• Building fat jar with all dependencies included
• Building distribution package with all dependencies in lib folder

The UpdateState function can be exposed as a Merger step for 2 inputs:

• the first argument is a DataFrame wrapper around Seq[Row] - the input data
• the second argument is the optional state as a DataFrame (0 rows corresponds to None)
• the output is the result state as a DataFrame (0 rows corresponds to None)

Currently there's only one artifact, ignition.jar
To make it flexible, need to refactor that into multiple jar files:

• ignition-core
• ignition-db
• ignition-dsa
• etc.

Plus one fat ignition-all.jar

Currently, each output value is recomputed every time the output is accessed. Need to implement the internal step cache to avoid that, and reset operation to reset the value and force the recomputation.

Date functions are not working because of the serialization to string.

Need to create IT configuration and move the appropriate unit tests there or create new ones:

• RestClient
• Cassandra
• Mongo
• Kafka

Combine the two steps into one and extend its functionality to provide the following:

• Row separation strategy
  • newline - use textFile
  • regex - use Source.fromFile, then split
  • none - use Source.fromFile
• Column separation strategy
  • regex - use split on each row
  • fixed - use take(), etc.
  • none - whole row
• Column names/types
  • none - use COL0, COL1, etc. with String type
  • schema - validate and apply conversion

Implement a step to provide caching of intermediate results

Currently spark libraries and their dependencies are added to the distribution; change their scope to provided, but also allow them to be added at runtime when running the examples

how to use directionly

Implement a step for generating a set of numeric values in a given range

Implement a step which would allow increasing or decreasing the number of data partitions.

Need to add AddFields step to the mix

frame.Main and stream.Main initialize their own contexts, need to refactor that to use a single SC

Operator IN does not work for stream filter, because it cannot parse it from the String. IN needs to be added as a UDF function.

Currently, Formula will fail if the input step does not have any rows, because of the way the schema gets computed. This needs to be fixed.

There's inconsistency in naming various elements of step representations: "group-by" vs "groupBy", "columns" vs "fields" etc. Need to make it consistent throughout the app. Also, think of making the tags shorter (like "csv-input" vs "csv-file-input", "debug" vs "debug-output" etc.)

Implement an input step for streaming data from DSA

Upgrade Spark to the latest version (1.4.1 as of 9/15), update the Cassandra spark connector accordingly.