This repository provides various developer tutorials and samples for building data pipelines with Spring Cloud Data Flow. The samples are included in a document available in multi-page and single-page HTML.
To build the documents
$mvn package
Sample starter applications and code for use with the Spring Cloud Data Flow project
Home Page: http://cloud.spring.io/spring-cloud-dataflow/
This repository provides various developer tutorials and samples for building data pipelines with Spring Cloud Data Flow. The samples are included in a document available in multi-page and single-page HTML.
To build the documents
$mvn package
FileIngest demo was not working correctly for me. It was missing start and end times on the task and never showing information in the dashboard. Fix was to simply add the @EnableTask, which seems to have been a regression.
I just deployed as shared batch job sample in the Spring Data Flow where i could see the job ran and provided the exit code as zero but when i look at the debug statements which hasn't executed in the log file where it being thrown taskLifeCycleListener exception. Please help me to understand what could be a issue in PCF when deploying this sample program.
17-06-29T19:23:43.89+0530 [CELL/0] OUT Creating container
17-06-29T19:23:44.68+0530 [CELL/0] OUT Successfully created container
17-06-29T19:23:48.94+0530 [APP/TASK/simpletTask/0] OUT \/ )| |)| | | | | || (| | ) ) ) )
17-06-29T19:23:48.94+0530 [APP/TASK/simpletTask/0] OUT =========||==============|/=////
17-06-29T19:23:48.95+0530 [APP/TASK/simpletTask/0] OUT :: Spring Boot :: (v1.5.2.RELEASE)
17-06-29T19:23:48.94+0530 [APP/TASK/simpletTask/0] OUT ( ( )___ | '_ | '| | ' / ` | \ \ \
17-06-29T19:23:48.94+0530 [APP/TASK/simpletTask/0] OUT . ____ _ __ _ _
17-06-29T19:23:48.94+0530 [APP/TASK/simpletTask/0] OUT ' || .__|| ||| |_, | / / / /
17-06-29T19:23:48.94+0530 [APP/TASK/simpletTask/0] OUT /\ / _' __ _ () __ __ _ \ \ \
17-06-29T19:23:49.09+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:49.093 INFO 7 --- [ main] pertySourceApplicationContextInitializer : Adding 'cloud' PropertySource to Appli
tionContext
17-06-29T19:23:49.20+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:49.204 INFO 7 --- [ main] io.spring.BatchJobApplication : The following profiles are active: clo
17-06-29T19:23:49.20+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:49.204 INFO 7 --- [ main] io.spring.BatchJobApplication : Starting BatchJobApplication on f11a42
-29c9-4fdb-acfa-43de5e286c8a with PID 7 (/home/vcap/app/BOOT-INF/classes started by vcap in /home/vcap/app)
17-06-29T19:23:49.31+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:49.314 INFO 7 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context
nnotation.AnnotationConfigApplicationContext@25f38edc: startup date [Thu Jun 29 13:53:49 UTC 2017]; root of context hierarchy
17-06-29T19:23:49.18+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:49.181 INFO 7 --- [ main] nfigurationApplicationContextInitializer : Adding cloud service auto-reconfigurat
n to ApplicationContext
17-06-29T19:23:49.85+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:49.859 INFO 7 --- [ main] o.s.b.f.s.DefaultListableBeanFactory : Overriding bean definition for bean 't
nsactionManager' with a different definition: replacing [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=f
se; factoryBeanName=org.springframework.cloud.task.configuration.SimpleTaskConfiguration; factoryMethodName=transactionManager; initMethodName=null; destroyMethodName=(inferred); defined in
g.springframework.cloud.task.configuration.SimpleTaskConfiguration] with [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidat
true; primary=false; factoryBeanName=org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration; factoryMethodName=transactionManager; initMethodName=null; destroyMetho
ame=(inferred); defined in class path resource [org/springframework/batch/core/configuration/annotation/SimpleBatchConfiguration.class]]
17-06-29T19:23:50.25+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:50.258 INFO 7 --- [ main] urceCloudServiceBeanFactoryPostProcessor : Auto-reconfiguring beans of type javax
ql.DataSource
17-06-29T19:23:50.26+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:50.266 INFO 7 --- [ main] urceCloudServiceBeanFactoryPostProcessor : No matching service found. Skipping au
-reconfiguration.
17-06-29T19:23:50.29+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:50.293 WARN 7 --- [ main] o.s.c.a.ConfigurationClassEnhancer : @bean method ScopeConfiguration.stepSc
e is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @resource and @Postc
struct within the method's declaring @configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @bean javadoc for complete details.
17-06-29T19:23:50.31+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:50.315 WARN 7 --- [ main] o.s.c.a.ConfigurationClassEnhancer : @bean method ScopeConfiguration.jobSco
is non-static and returns an object assignable to Spring's BeanFactoryPostProcessor interface. This will result in a failure to process annotations such as @Autowired, @resource and @PostCo
truct within the method's declaring @configuration class. Add the 'static' modifier to this method to avoid these container lifecycle issues; see @bean javadoc for complete details.
17-06-29T19:23:50.41+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:50.412 INFO 7 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.b
ch.configuration.TaskBatchAutoConfiguration' of type [org.springframework.cloud.task.batch.configuration.TaskBatchAutoConfiguration$$EnhancerBySpringCGLIB$$9003e312] is not eligible for gett
g processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
17-06-29T19:23:50.42+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:50.420 INFO 7 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.
notation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$a943786] is not eligible
or getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
17-06-29T19:23:50.46+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:50.462 INFO 7 --- [ main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.cloud.task.b
ch.listener.BatchEventAutoConfiguration' of type [org.springframework.cloud.task.batch.listener.BatchEventAutoConfiguration$$EnhancerBySpringCGLIB$$d859ed15] is not eligible for getting proc
sed by all BeanPostProcessors (for example: not eligible for auto-proxying)
17-06-29T19:23:51.01+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.007 DEBUG 7 --- [ main] o.s.c.t.c.SimpleTaskConfiguration : Using org.springframework.cloud.task.c
figuration.DefaultTaskConfigurer TaskConfigurer
17-06-29T19:23:51.25+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.250 DEBUG 7 --- [ main] o.s.c.t.r.s.TaskRepositoryInitializer : Initializing task schema for h2 databa
17-06-29T19:23:51.25+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.257 INFO 7 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executing SQL script from class path r
ource [org/springframework/cloud/task/schema-h2.sql]
17-06-29T19:23:51.30+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.308 INFO 7 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executed SQL script from class path re
urce [org/springframework/cloud/task/schema-h2.sql] in 51 ms.
17-06-29T19:23:51.47+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.472 INFO 7 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executing SQL script from class path r
ource [org/springframework/batch/core/schema-h2.sql]
17-06-29T19:23:51.49+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.497 INFO 7 --- [ main] o.s.jdbc.datasource.init.ScriptUtils : Executed SQL script from class path re
urce [org/springframework/batch/core/schema-h2.sql] in 25 ms.
17-06-29T19:23:51.60+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.605 INFO 7 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on
artup
17-06-29T19:23:51.61+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.611 INFO 7 --- [ main] o.s.c.support.DefaultLifecycleProcessor : Starting beans in phase 0
17-06-29T19:23:51.69+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.690 INFO 7 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Unregistering JMX-exposed beans on shu
own
17-06-29T19:23:51.69+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.692 ERROR 7 --- [ main] o.s.c.t.listener.TaskLifecycleListener : An event to end a task has been receiv
for a task that has not yet started.
17-06-29T19:23:51.69+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.690 WARN 7 --- [ main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context i
tialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'taskLifecycleListener'; nested exception is java.lang.IllegalArgument
ception: Invalid TaskExecution, ID 15 not found
17-06-29T19:23:51.70+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.700 INFO 7 --- [ main] utoConfigurationReportLoggingInitializer :
17-06-29T19:23:51.70+0530 [APP/TASK/simpletTask/0] OUT Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT 2017-06-29 13:53:51.720 ERROR 7 --- [ main] o.s.boot.SpringApplication : Application startup failed
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT org.springframework.context.ApplicationContextException: Failed to start bean 'taskLifecycleListener'; nested exception is java.lang.Il
galArgumentException: Invalid TaskExecution, ID 15 not found
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:879) ~[spring-co
ext-4.3.7.RELEASE.jar!/:4.3.7.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737) [spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:370) [spring-boot-1.5.2.RELEASE.jar!/:1.5.2.
LEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:175) ~[spring-context-4.
7.RELEASE.jar!/:4.3.7.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT ... 20 common frames omitted
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-4.
7.RELEASE.jar!/:4.3.7.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:348) ~[spri
-context-4.3.7.RELEASE.jar!/:4.3.7.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.boot.SpringApplication.run(SpringApplication.java:1162) [spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.boot.SpringApplication.run(SpringApplication.java:1151) [spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at io.spring.BatchJobApplication.main(BatchJobApplication.java:14) [classes/:na]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_121]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_121]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [app/:na]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [app/:na]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [app/:na]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.util.Assert.notNull(Assert.java:134) ~[spring-core-4.3.7.RELEASE.jar!/:4.3.7.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.cloud.task.listener.TaskLifecycleListener.doTaskStart(TaskLifecycleListener.java:200) ~[spring-cloud-task
ore-1.3.0.BUILD-SNAPSHOT.jar!/:1.3.0.BUILD-SNAPSHOT]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:151) ~[spring-context
.3.7.RELEASE.jar!/:4.3.7.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:114) ~[spring-context-
3.7.RELEASE.jar!/:4.3.7.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_121]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT Caused by: java.lang.IllegalArgumentException: Invalid TaskExecution, ID 15 not found
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_121]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.cloud.task.listener.TaskLifecycleListener.start(TaskLifecycleListener.java:282) ~[spring-cloud-task-core-
3.0.BUILD-SNAPSHOT.jar!/:1.3.0.BUILD-SNAPSHOT]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:50) ~[spring-context-
3.7.RELEASE.jar!/:4.3.7.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:545) ~[spring-context-
3.7.RELEASE.jar!/:4.3.7.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) [spring-boot-1.5.2.RELEASE.jar!/:1.5.2.RELEASE]
17-06-29T19:23:51.72+0530 [APP/TASK/simpletTask/0] OUT at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [app/:na]
17-06-29T19:23:51.75+0530 [APP/TASK/simpletTask/0] OUT Exit status 1
17-06-29T19:23:51.78+0530 [CELL/0] OUT Destroying container
17-06-29T19:23:52.98+0530 [CELL/0] OUT Successfully destroyed container
Took a look at the docs directory and this file is missing.
Give wget coordinates for local server and shell, import bacon release train for apps
As a developer, I'd like to publish a sample on how InfluxDB/Grafana, so it can be used to reason through SCDF's stream level metrics and other stats.
Note: This is specifically only to showcase SCSt 2.0 + Micrometer integration in the context of SCDF's "stream" primitive.
As a developer, I'd like to add a sample to showcase how rate-limiting can be handled when launching Tasks based on upstream events.
Child of #66
This sample would build upon the existing file-ingest sample that relies on the SFTP-source. The configuration of rate-limiting, the new task-launcher for SCDF, and the overall orchestration play in PCF would be the highlight of this sample.
Hi,
Can you please add some samples for Spring Cloud dataflow to integrate it with Apache Spark on Yarn? Able to find examples to invoke Spark with Spring XD, but not with Spring Cloud dataflow. It would really help to integrate Spring Cloud dataflow with Spark with some reference.
Can I apply the new security-ldap-uaa-example on SCDF running on PWS? As SSO is not available for normal user.
Create basic batch ingest sample
Use GS "Creating a Batch Service" (https://spring.io/guides/gs/batch-processing/) guide contents as a template for batch job.
outdated content, will shortly be replaced by another approach that is much better.
As a developer, I'd like to add a new CI plan to build the project/docs and publish it to https://docs.spring.io/spring-cloud-dataflow-samples/docs/.
Should not need to build a local server/shell, give wget instructions instead.
Import the Bacon release train, not 1.0.4...
child of #81
In continuation to the dev-marketing efforts, as a developer, I'd like to add walkthrough guide on how to use the acceptance-test framework to develop and test e2e data pipelines.
child of #66.
child of #61
child of #61
The goal of this epic is to redesign the file-ingest reference architecture with the latest improvements including the concurrent Task launches and the throttling optimizations.
See: https://jira.spring.io/browse/BATCH-2695
Child of #81
In continuation to the dev-marketing efforts, as a developer, I'd like to add walkthrough guide on how to develop and test custom stream applications.
Child of #66
Use-case: File -> processing -> some storage.
As a continuation to the existing ketacoda efforts, we would like to formalize the initial MVP work into a publicly available collateral.
In continuation to the dev-marketing efforts, as a developer, I'd like to add walkthrough guide on how to use the test-binder to develop and test custom applications.
As a developer, I'd like to add a gemfire-source and gemfire-sink samples included in a stream definition. The idea is to showcase how a source and sink can be used with other OOTB apps.
Hi, I implemented iris PA example, and the stream is in deployed status.
while I do
http post --target http://localhost:8080 --contentType application/json --data "{ \"sepalLength\": 6.4, \"sepalWidth\": 3.2, \"petalLength\":4.5, \"petalWidth\":1.5 }"
I got
401 UNAUTHORIZED
{
"path" : "/",
"error" : "Unauthorized",
"message" : "Full authentication is required to access this resource",
"timestamp" : 1484714522151,
"status" : 401
}
While I used Postman with the Basic Auth
username: user
password: 0f9555a9-e5ae-448d-80c5-bcfc80acf7db
I still got the same problem.
OS: debian:jessie
SCDF: 1.1.1.RELEASE
This repo needs to be updated with the latest changes on spring-cloud-dataflow
After spring-attic/sftp#9 is merged, documentation on this branch: https://github.com/chrisjs/spring-cloud-dataflow-samples/tree/dl_file needs to be updated to reflect getting file metadata from the SFTP source. The code changes in this branch also supports the functionality in the above PR, specifically in terms of things like StepExecutionListener
to download the file and so on
As a developer, I'd like to load test the file-ingest sample with large files of size >2G or more each.
child of #66
Notes:
From @sabbyanandan on January 29, 2018 15:50
Streams
Java DSL - @viniciusccarvalho
HTTP to Cassandra - @viniciusccarvalho
HTTP to MySQL - @viniciusccarvalho
HTTP to Gemfire - @tzolov
Gemfire CQ to Log Demo - @tzolov
Gemfire to Log Demo - @tzolov
Custom Processor
Analytics
Data Science
Functions
Copied from original issue: spring-cloud/spring-cloud-dataflow#2008
Related to spring-cloud/spring-cloud-dataflow#1965
ENV:
using docker on CentOS server
OS: debian:jessie
SCDF local server: 1.1.2.RELEASE
HTTP source version: 1.1.2.RELEASE
PMML processor version: 1.1.1.RELEASE
in log*/streamname-*/streamname.pmml/stdout_o.log
file:
2017-01-20 13:58:49.598 ERROR 922 --- [afka-listener-1] o.s.kafka.listener.LoggingErrorHandler : Error while processing: ConsumerRecord(topic = main.http, partition = 0, offset = 2, CreateTime = 1484920726590, checksum = 2209571897, serialized key size = -1, serialized value size = 97, key = null, value = [B@25d77dac)
org.springframework.messaging.MessageDeliveryException: failed to send Message to channel 'input'; nested exception is java.lang.IllegalArgumentException: payload must not be null
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:449) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:373) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45) ~[spring-messaging-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105) ~[spring-messaging-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:292) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:212) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:129) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:115) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.channel.FixedSubscriberChannel.send(FixedSubscriberChannel.java:70) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.channel.FixedSubscriberChannel.send(FixedSubscriberChannel.java:64) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45) ~[spring-messaging-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105) ~[spring-messaging-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:171) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$000(KafkaMessageDrivenChannelAdapter.java:54) ~[spring-integration-kafka-2.1.0.RELEASE.jar!/:na]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:288) ~[spring-integration-kafka-2.1.0.RELEASE.jar!/:na]
at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:279) ~[spring-integration-kafka-2.1.0.RELEASE.jar!/:na]
at org.springframework.kafka.listener.adapter.RetryingAcknowledgingMessageListenerAdapter$1.doWithRetry(RetryingAcknowledgingMessageListenerAdapter.java:77) ~[spring-kafka-1.1.1.RELEASE.jar!/:na]
at org.springframework.kafka.listener.adapter.RetryingAcknowledgingMessageListenerAdapter$1.doWithRetry(RetryingAcknowledgingMessageListenerAdapter.java:72) ~[spring-kafka-1.1.1.RELEASE.jar!/:na]
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:276) ~[spring-retry-1.1.4.RELEASE.jar!/:na]
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:172) ~[spring-retry-1.1.4.RELEASE.jar!/:na]
at org.springframework.kafka.listener.adapter.RetryingAcknowledgingMessageListenerAdapter.onMessage(RetryingAcknowledgingMessageListenerAdapter.java:72) ~[spring-kafka-1.1.1.RELEASE.jar!/:na]
at org.springframework.kafka.listener.adapter.RetryingAcknowledgingMessageListenerAdapter.onMessage(RetryingAcknowledgingMessageListenerAdapter.java:39) ~[spring-kafka-1.1.1.RELEASE.jar!/:na]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:764) [spring-kafka-1.1.1.RELEASE.jar!/:na]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:708) [spring-kafka-1.1.1.RELEASE.jar!/:na]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.access$2500(KafkaMessageListenerContainer.java:230) [spring-kafka-1.1.1.RELEASE.jar!/:na]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer$ListenerInvoker.run(KafkaMessageListenerContainer.java:981) [spring-kafka-1.1.1.RELEASE.jar!/:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
Caused by: java.lang.IllegalArgumentException: payload must not be null
at org.springframework.util.Assert.notNull(Assert.java:115) ~[spring-core-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.integration.support.MutableMessage.<init>(MutableMessage.java:57) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.support.MutableMessage.<init>(MutableMessage.java:53) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.support.MutableMessageBuilder.withPayload(MutableMessageBuilder.java:86) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.support.MutableMessageBuilderFactory.withPayload(MutableMessageBuilderFactory.java:35) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.support.MutableMessageBuilderFactory.withPayload(MutableMessageBuilderFactory.java:26) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.cloud.stream.binding.MessageConverterConfigurer$ContentTypeConvertingInterceptor.preSend(MessageConverterConfigurer.java:194) ~[spring-cloud-stream-1.1.0.RELEASE.jar!/:1.1.0.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel$ChannelInterceptorList.preSend(AbstractMessageChannel.java:538) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:415) ~[spring-integration-core-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
... 31 common frames omitted
nothing abnormal in *.log/stdout.0.log/
and *.http/stdout.0.log/
file
While I created my own stream
stream create --name test --definition "http --server.port=8080 | log" --deploy
everything works fine, this should mean my kafka works fine
Child of #66
As a developer, I'd like to add a static Boot App as a placeholder template for OOTB Apps, so the users can list the desired App coordinates to download, bundle, and serve the uber-jar's. This App would be an alternative approach to Maven artifact resolution, and it would also be useful for environments with no-internet connectivity.
Notes:
A potential directory structure could be:
(at root)
./download.sh
./stream-apps.properties
time-source-1.0.0.RELEASE
log-sink-1.0.0.RELEASE
./task-apps.properties
timestamp-2.0.0.RELEASE
download.sh
would populate:./src/main/resources/static/stream-apps/time-source-1.0.0.RELEASE.jar
./src/main/resources/static/stream-apps/log-sink-1.0.0.RELEASE.jar
./src/main/resources/static/task-apps/timestamp-2.0.0.RELEASE.jar
Basically, stream-apps.properties
and stream-apps.properties
serve as a template; users can add app coordinates in it, the download script will wget
it from the Spring repo.
Executing mvn package
would build the static App with all the JARs in it.
This static App can now serve the Apps via localhost
or on a CF route in PCF (i.e. cf push
ready) etc.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.