spring-cloud / spring-cloud-task Goto Github PK

View Code? Open in Web Editor NEW

415.0 55.0 308.0 5.47 MB

Short lived microservices with Spring Batch

Home Page: https://cloud.spring.io/spring-cloud-task/

License: Apache License 2.0

Java 100.00%

microservices batch-processing short-lived-microservices

spring-cloud-task's Introduction

Spring Cloud Task

Is a project centered around the idea of processing on demand. A user is able to develop a “task” that can be deployed, executed and removed on demand, yet the result of the process persists beyond the life of the task for future reporting.

Requirements:

Java 17 or Above

Build Main Project:

  $ ./mvnw clean install

Example:

  @SpringBootApplication
  @EnableTask
  public class MyApp {

      @Bean
      public MyTaskApplication myTask() {
          return new MyTaskApplication();
      }

      public static void main(String[] args) {
          SpringApplication.run(MyApp.class);
      }

      public static class MyTaskApplication implements ApplicationRunner {

          @Override
          public void run(ApplicationArguments args) throws Exception {
              System.out.println("Hello World");
          }
      }
  }

Code of Conduct

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to [email protected].

Building the Project

This project requires that you invoke the Javadoc engine from the Maven command line. You can do so by appending javadoc:aggregate to the rest of your Maven command. For example, to build the entire project, you could use mvn clean install -DskipTests -P docs.

spring-cloud-task's People

Contributors

Stargazers

Watchers

Forkers

mminella trisberg ericbottard sabbyanandan markpollack ahmadimt sixgodx dsyer pkdevboxy erwy cuilaixi sobychacko ilayaperumalg mbogoevici didiez skinisbizapps felipeg48 jvalkeal cppwfs markfisher fengshao0907 personal-mkkm kaysonyang 72miguelgomes damozhiying zjpjohn duyunqiang zmyer renyichan scbrown guquanjiang seansonggi pco1853 mianbaoshitou rongfengliang adonis2014 liuzantian mkiredjian ssiahetiong efbiz-spring-cloud appsou tsachev javacodemood shahbour huhtmail vishukadam chenlxspace bit-linjiang lqnasa schatterjee4 dturanski redcodes zhoudaqing xbchena javiergithub lokihjl alihammoud90 tomitsa kevingreene ahmadjawidjami mnenad spacewen rksharma82 spider-man-git lethetann ddkutukculer rpy123 hellotengyu tcl1122 jq384044342 ekswai seanwan1989 easy2go karthikitzmi isaacyd praveendev1 garyfub dbcdcw vcshasi mixergit vxiaoming mt-village gitter-badger limingzhou mohanjagadheeswaran dotnetrockstar ayscb haozigithub ps199309141 viniciusccarvalho diaonannan1234 cccyyy kangtaku ramcharlapalli royclarkson buzzardo huangyongchao kirtichoudhuri bloodwind wanglike

spring-cloud-task's Issues

User wants ability to store common exit code when running multiple CommandLineRunners.

User wants ability to store exit code using the same logic as Spring boot in cases where more than CommandLineRunnners are within the spring boot app. Currently implementation uses last exit code in returned list.
Using the getExitCode as a template or submitting a PR to spring boot to make SpringApplication.getExitCode public. Here is a link to the code: https://github.com/spring-projects/spring-boot/blob/9c412232d5fe0353f0fd117ba438ed080683eee1/spring-boot/src/main/java/org/springframework/boot/SpringApplication.java#L1167

Create gh-pages branch and content for Spring Cloud Task

As a project, I need to be represented on the spring.io site.

Create PostProcessor to register a Task for each batch job.

As a developer, I want to be able to run Spring Batch jobs packaged in Spring Boot as a Task.

Add TaskRepository implementation to keep track of task executions

As a user, I'd want an out of the box TaskRepository implementation that logs task execution status updates.

Explore the ability to publish TaskExecutionListener events to Spring Cloud Streams

As a developer, I want to be able to react to events that occur during the execution of a task (TaskExecutionListner calls).

User wants to support pagination for execution result sets

for result sets from the Explorer interface should return pageable.

Add 'timeout' support for Tasks

As a user, I'd want the ability to place timeout on a task running in stand alone.

User wants to databases initialized if datasource is provided using defaults.

User wants Spring Cloud Task repository initialized for Oracle, Mysql and Postgres if user provides a datasource but does not provide a TaskConfigurer.
Currently only hsqldb is supported.

Need to have sample cloud task spring application.

Use https://github.com/cppwfs/sct-demo as a template

A user wants command line arguments stored in the task repository

As a user, I want the ability to see what command line arguments were used associated with a Task's execution.

Update to use new Boot events

Update Spring Cloud Task to use the new Spring Boot events that expose the ExitCode

Create JobLauncherTask

As a developer, I want to be able to use Spring Cloud Task to run a Spring Batch job.

Add Reference Documentation

As a developer, I want reference documentation so that I can fully utilize Spring Cloud Task

Add support to receive task lifecycle events from another module

As a user, I'd want to be able to receive task lifecycle events from another module, so I can orchestrate set of modules in a composite workflow.

Acceptance:
I should be able to do something like: fooTask | barSink where the fooTask and barSink modules are connected by a bus to send and received events

SimpleTaskRepository should use its own id from a number sequence for a primary key

User wants timestamp batch job task for Task Modules project

Migrate Timestamp Sample to SCT-modules

Add ability to externally configure the task name

As a system, I need to be able to override the task name as configured within the task.

Add TaskExplorer implementation

As a user, I'd want the TaskExplorer implementation, so I can use this contract to query task details [such as: name, execution-count etc.] from the database.

Add MS SQL Server as supported databases

We should support the most commonly used databases, so this would mean we should add MS SQL Server his would also be good for cloud deployments since Azure has SQL Server

For popularity numbers see - http://db-engines.com/en/ranking_osvsc

Release infrastructure

As a committer, I want to be able to release and promote Spring Cloud Task to the appropriate maven repositories.

Persist task execution statuses in database

As a user, I'd want the ability to store and update task execution statuses in database, so I can use the persisted events for later analysis.

Developers want methods that return lists to be pageable

Currently findRunningTaskExecutions and getTaskExecutionsByName return either the full list or have a partially implemented paging paradigm. Developers would like to pass in a pageable object and get a page instead of a list or set.

Add @EnableTask and @Task annotation

As a user, I'd want a @Task API, so I can associate task-name via this annotation and have it interpreted as runnable Spring Boot application.

User wants ability to have the reported exitCode be for the specific task

Developers want H2 DB support for tasks

Add support to launch task from another module

As a user, I'd want to be able to launch a task from another module, so I can orchestrate mix and match of task and stream modules as composite workflows.

Acceptance:
I should be able to do: fooSource | barTask, where fooSource would be responsible for launching barTask, which is a task module.

Developer wants to use mvnw for task builds

To remain consistent with other cloud projects SCT needs to build using mvnw

User wants boot exit code recorded when boot app ends

When Spring boot supports exit events with exit code, SCT will need to use that event instead of ContextClosedEvent.

User wants EnableTask at SpringApplication level vs. *Runner

Is this possible or do we stay at the *Runner Level

Developers want to remove unused taskExecution Fields

Remove externalExecutionID & StatusCode from classes and schema.

Clean up Sonar Issues

With us tracking Sonar issues now, we should start by cleaning things up so we have a clean slate.

Add Sonar to build

As a developer I want to see Sonar metrics for Spring Cloud Tasks.

Should @EnableTask accept a dataSource property?

Add ability to associate batch job with task it was executed within

As a developer, when I run a Spring Batch job as a Spring Cloud Task, I want to be able to associate the two.

Task Execution Insert fails for postgresDB

The argType for task execution id in the task_execution_parameter table is still set to varchar. H2, hsqldb & mysql were forgiving, but postgresql was strict.

Remove requirement for executionId to display step execution in shell

When viewing a job's step execution via the shell, the user is required to provide both the job execution id and the step execution id. Since the job repository is backed by a database and the step execution id is unique across jobs, the step execution id should be enough.

Acceptance:
The following command should return the details of the step:
xd:> job execution step display --id 36

User wants to record externalExecutionID for each task execution

Allows a user to pass in a externalExecutionID (DepoymentID) if one is generated by the system (CF, Yarn, etc) that is executing the task. This externalExecutionID will be stored in the taskExecutionTable.
possibly --spring.cloud.task.externalexecutionid

Depends on #115

Add support for deployment properties

As a user, I'd like to launch tasks with deployment properties, so I can define how the deployment will be orchestrated.

Acceptance:

Following task definition accepts deployment manifest to override certain behavior of the sampleJob application

task create myjob --definition "sampleJob"

task deploy myjob --properties "module.sampleJob.foo=bar"

Deployment manifest is an optional DSL attribute for task deploys

Add no-op TaskExplorer implementation

As a user, I'd want a default no-op TaskExplorer implementation such that they can work with spring cloud task out of the box.

Explore on-demand data processing specs

As a developer, I'd like to study the core design of a task that can be executed and removed on demand.

Create a BOM for users to add SCT dependencies

As a user, I want to be able to use Spring Cloud Task (SCT) and manage the version of Spring Boot independently.

Back Story: Since SCT depends on Boot, we need a BOM that contains only the dependencies for SCT so that users can layer that on top of the Boot dependencies they are already bringing in.

Users need sequence table to use MYISAM

As discussed here the engine for the table should be ISAM vs. InnoDB.

http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/jdbc/support/incrementer/MySQLMaxValueIncrementer.html

Users want tasks to initialize schema only if tables not present.

TaskExecutionListener

As a developer, I want to be able to execute code before and/or after the execution of a task.

Spike: Investigate the ability to override Boot's launching of jobs.

As a developer, I want to be able to execute a Spring Batch Job in the context of a Task.

Restricting Spring Cloud Task instances

There are use cases where an application is triggered by cron scheduler but due to communication with 3rd party API(s), the application runs slow and hence the cron scheduler triggers another instance of the application again while the first instance is still running.