lightbend / fdp-sample-applications Goto Github PK

Hello,
I'm currently working with flink example in project.
But after activate kafka/zookeeper server and start ingest:

sbt:ingestRun> ingest
[info] Running (fork) com.lightbend.fdp.sample.flink.ingestion.DataIngestion 
[info] Got /Users/jung/workspace/etc/fdp-sample-applications/apps/flink/source/core/data/nycTaxiFares.csv to process
[info] Got /Users/jung/workspace/etc/fdp-sample-applications/apps/flink/source/core/data/nycTaxiRides.csv to process

...it does not go further after this line.

I'm running in local system(not using docker, DC/OS...), and my config is:

dcos {

  kafka {
    brokers = "localhost:9092"
    brokers = ${?KAFKA_BROKERS}

    group = "group"
    group = ${?KAFKA_GROUP}

    intopic = "taxiin"
    intopic = ${?KAFKA_IN_TOPIC}

    outtopic = "taxiout"
    outtopic = ${?KAFKA_OUT_TOPIC}

    zookeeper = "localhost:2181"
    zookeeper = ${?ZOOKEEPER_URL}

    ## settings for data ingestion
    loader {
      sourcetopic = ${dcos.kafka.intopic}
      sourcetopic = ${?KAFKA_IN_TOPIC}

      directorytowatch = "/path/to/fdp-sample-applications/apps/flink/source/core/data"
      directorytowatch = ${?DIRECTORY_TO_WATCH}

      pollinterval = 1 second
    }
  }
}

I fetched taxi data from http://training.data-artisans.com/exercises/taxiData.html , and put files in ./apps/flink/source/core/data.
Is there something missing?

Thanks.

I'm currently evaluating FDP. In the process, I have attempted to install the killrweather app as described here: https://developer.lightbend.com/docs/fast-data-platform/1.3.1/user-guide/sample-apps/index.html#fast-data-platform-killrweather

The documentation points to the README.MD in this repository: https://github.com/lightbend/fdp-sample-applications/blob/develop/apps/killrweather/README.md

That file instructs me to deploy the killrweather app using this command:
dcos marathon app add killrweather-app/src/main/resources/killrweatherAppDocker.json
I'm assuming this refers to https://github.com/lightbend/fdp-sample-applications/blob/develop/apps/killrweather/source/core/killrweather-app/src/main/resources/killrweatherAppDocker.json. I think that's right, because installing the loader worked with a small modification (see below).

However, there are a couple of issues with this JSON file:

• the FDP platform version filled in is 1.3.0. The current version is 1.3.1. Furthermore, running process-templates.sh 1.3.1, as instructed, does not update the file.
• there is a syntax problem with the file at line 51, setting the CASSANDRA_HOSTS: a comma is missing at the end of the line. The same problem exists in killrweatherloaderDocker.json and the corresponding .json.template files.
• after fixing these two issues, deploying the service results in the following error:
  Error while fetching [http://fdp-dcos-elasticl-5us6x2035mlk-426186471.eu-west-1.elb.amazonaws.com/service/marathon/v2/pods]: HTTP 422: "{"message":"Invalid JSON","details":[{"path":"/containers","errors":["'containers' is undefined on object: <snipped by me>

Indeed, the killrweatherloaderDocker.json specifies a containers array with two extensive elements, whereas killrweatherAppDocker.json only has a rather simplistic container object.

I am unfamiliar with DC/OS, so I am unsure how to fix this.