Spring Batch Lightmin

Table of Contents

About
Architecture
Documentation
Samples
Getting Started
Remote Repository Server
- Maven
- Configuration
Embedded Lightmin
- Maven
- Configuration
Server User Interface

About

Spring Batch Lightmin is a client/server batch and scheduling platform for the Spring Boot stack. The batch stack based on Spring Batch.

Architecture

Documentation

The full documentation can be found at

2.1.x (Spring Boot 2.2.x)
2.0.x (Spring Boot 2.1.x)
1.x (Spring Boot 1.5.x) - End of life
0.5.x (old design - Spring Boot 1.5.x) - End of life

Samples

All samples for Spring Batch Lightmin can be found at Sample Applications

Getting Started

Important

Spring 5.1.x set the the property spring.main.allow-bean-definition-overriding to false as a default. In order to use the current version of Spring Batch Lightmin, this property has to be set to true.

This issue can hopefully be solved in the upcoming releases.

The following section describes a quick start guide to setup the client server architecture. If your want to get an overview of the configuration options, please read the documentation. The getting started guide for the Embedded Mode can be found in the section Embedded Lightmin.

Spring Batch Lightmin BOM

Add the Spring Batch Lightmin BOM to the dependency management section of your project.

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-bom</artifactId>
    <version>${spring-batch-lightmin.version}</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>

Setting up the Server

The server component is currently only available in a standalone mode. This means, the servers do not sync the state between each other. A cluster mode will be added in a future release.

The server itself is also a Spring Boot application.

Maven Dependencies

The following dependency has to added to your project.

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-server-standalone</artifactId>
    <version>${spring-batch-lightmin.version}</version>
</dependency>

Enable the Server

In order to enable the server, the annotation @EnableLightminServer has to added to one of the @Configuration classes.

@EnableLightminServer
@SpringBootApplication
public class LightminServer {

}

Service Discovery

Depending on the clients, the server is also able to find the clients via service discovery.

To enable the feature, a DiscoveryClient implementation has to added and configured(e.g. Consul, Eureka, etc…) and the following property has to set to true

spring::
  batch:
    lightmin:
      server:
        discovery-enabled: true

Server behind a Proxy

The server frontend uses redirects and the HOST header is taken to create the redirect urls. This is the default behaviour of Spring MVC. If the server frontend is running behind a proxy, the implementation of the server takes care, that the X-FORWARD-PREFIX header is used as well.

If the proxy cannot pass the HOST header or you do not want to change the defaults, e.g. Zuul, the following property can force the server to use the X-FORWARDED-HOST header to build the redirect urls.

spring:
  batch:
    lightmin:
      server:
        use-x-forwarded-headers: true

Setting up a Client

The client applications are responsible to provide Spring Batch Job definitions. The Spring Batch Lightmin client framework provides all the configurations to set up Spring Batch and the communication with the server.

Step one - Client type

The type of the client decides how the registration to the server should be done. Currently to types are supported, classic and via service discovery.

Classic

The classic client has to know where the servers are located and will send a registration request after the start up.

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-client-classic</artifactId>
    <version>${spring-batch-lightmin.version}</version>
</dependency>

The dependency above will provide everything which is required for the classic client. The annotation @EnableLightminClientClassic has to added to one of the configuration classes.

@SpringBootApplication
@EnableLightminClientClassic
public class ClientApplication {

    public static void main(final String[] args) {
        SpringApplication.run(ClientApplication.class, args);
    }
}

The following configuration properties have to be present

spring:
  application:
    name: my-client-application (1)
  batch:
    lightmin:
      client:
        classic:
          server:
            url: http://myserver1.domain:8180, http://myserver2.domain:8180 (2)

The spring.application.name is used to identify a client and handle a cluster of the instances.
The url property is a list of server to which the registration request should be send.

Service Discovery

Spring Batch Lightmin provides two implementations for the discovery client type. Both implementations add a tags to the underlying service discovery technology, so the server could identify lightmin clients.

Consul Client

The following dependency has to added for the consul client

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-client-discovery-consul</artifactId>
    <version>${spring-batch-lightmin.version}</version>
</dependency>

The annotation @EnableLightminClientConsul enables the fully integration with Consul.

@SpringBootApplication
@EnableLightminClientConsul
public class ClientApplication {

    public static void main(final String[] args) {
        SpringApplication.run(ClientApplication.class, args);
    }
}

The configuration options for Consul can be found in the Spring Cloud Consul documentation.

The client is sending per default events to the server, in order to find the server via service discovery, the following property has to be set.

spring:
  batch:
    lightmin:
      client:
        discovery:
          server-discovery-name: lightmin-server (1)

The service discovery name of the server.

Eureka Client

The following dependency has to added for the eureka client

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-client-discovery-eureka</artifactId>
    <version>${spring-batch-lightmin.version}</version>
</dependency>

The annotation @EnableLightminClientEureka enables the fully integration with Eureka.

@SpringBootApplication
@EnableLightminClientEureka
public class ClientApplication {

    public static void main(final String[] args) {
        SpringApplication.run(ClientApplication.class, args);
    }
}

The configuration options for Eureka can be found in the Spring Cloud Netflix documentation.

The client is sending per default events to the server, in order to find the server via service discovery, the following property has to be set.

spring:
  batch:
    lightmin:
      client:
        discovery:
          server-discovery-name: lightmin-server (1)

The service discovery name of the server.

Step two - The Configuration Repository

The Configuration Repository is the component which stores the scheduler and listener configurations of the lightmin clients. This configurations are loaded at start time and can be managed with the server frontend or API calls.

Spring Batch Lightmin provides three implementation of the repository

map - In memory repository, all changes will be gone after a restart.
jdbc - The client fetches and stores the configurations in a database.
remote - The client fetches and stores the configurations via API calls to a repository server.

Map Repository

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-repository-map</artifactId>
    <version>${spring-batch-lightmin.version}</version>
</dependency>

The annotation @EnableLightminMapConfigurationRepository enables the in memory repository.

@SpringBootApplication
@EnableLightminClientConsul
@EnableLightminMapConfigurationRepository
public class ClientApplication {

    public static void main(final String[] args) {
        SpringApplication.run(ClientApplication.class, args);
    }
}

Jdbc Repository

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-repository-jdbc</artifactId>
    <version>${spring-batch-lightmin.version}</version>
</dependency>

The annotation @EnableLightminJdbcConfigurationRepository enables the jdbc repository.

@SpringBootApplication
@EnableLightminClientConsul
@EnableLightminJdbcConfigurationRepository
public class ClientApplication {

    public static void main(final String[] args) {
        SpringApplication.run(ClientApplication.class, args);
    }
}

The jdbc repository requires a configured datasource bean with the name dataSource.

If the project configuration requires a specific datasource for the lightmin repository, the bean name can set via configuration property. More configuration options can be found in the documentation.

spring:
  batch:
    lightmin:
      repository:
        jdbc:
          data-source-name: myDataSource (1)

Overriding the default datasource name.

The database schema ddl and drop scripts for various databases are located in the dependency above under the path:

org/tuxdevelop/spring/batch/lightmin/repository

Remote Repository

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-repository-remote</artifactId>
    <version>${spring-batch-lightmin.version}</version>
</dependency>

The annotation @EnableLightminRemoteConfigurationRepository enables the remote repository.

@SpringBootApplication
@EnableLightminClientConsul
@EnableLightminRemoteConfigurationRepository
public class ClientApplication {

    public static void main(final String[] args) {
        SpringApplication.run(ClientApplication.class, args);
    }
}

The remote repository can be located via url and service discovery.

For the url way, the following properties has to be set:

  spring:
    batch:
      lightmin:
        repository:
          remote:
            server-url: http://my-server.domain:8280 (1)

The url to the remote repository server

For the service discovery approach, the following properties has to set and a DiscoveryClient bean has to be present.

  spring:
    batch:
      lightmin:
        repository:
          remote:
            discover-remote-repository: true (1)
            server-discovery-name: remoteRepositorySever (2)

Enables the discovery feature
The discovery name of the remote repository server

Further configuration options can be found in the documentation.

Step three - Configure Spring Batch

The client configurations are enabling the Spring Batch stack as well. Spring Batch itself has to have a configured JobRepository. This JobRepository can be in memory via map or jdbc.

The configuration of the JobRepository can be done via properties, so Spring Batch Lightmin knows what to configure.

Map JobRepository

For the map repository, the following configuration is enough:

spring:
  batch:
    lightmin:
      batch:
        repository-type: map

Jdbc Repository

For the jdbc repository, the following configuration is enough:

spring:
  batch:
    lightmin:
      batch:
        repository-type: jdbc

Properties like dataSource name, table prefix etc. can be overridden as well. Please check the documentation for more details.

Clients in Containers

If a client runs inside a container like Docker, the dns name of the host systems has to be transferred to server on registration time. For this use case, a property is available.

spring:
  batch:
    lightmin:
      client:
        hostname: FQDN of the host

Remote Repository Server

The Remote Repository Server is a Spring Boot application which provides a REST API for clients.

The server itself needs a job configuration repository itself. The server supports map and jdbc.

Maven

For the Jdbc repository, the following dependencies have to be added.

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-repository-server</artifactId>
    <version>${spring-batch-lightmin.version}</version>
</dependency>

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-repository-jdbc</artifactId>
    <version>${spring-batch-lightmin.version}</version>
</dependency>

Configuration

The configuration of the used repository can be found in the client section.

The annotation @EnableLightminRepositoryServer enables the server and the corresponding annotation the job configuration repository.

@SpringBootApplication
@EnableLightminRepositoryServer
@EnableLightminJdbcConfigurationRepository
public class RepositoryServerApplication {

    public static void main(final String[] args) {
        SpringApplication.run(RepositoryServerApplication.class, args);
    }
}

Embedded Lightmin

If the client server architecture does not fit the requirements, Spring Batch Lightmin also provides am embedded mode, which ships the client and server in one package.

Maven

<dependency>
    <groupId>org.tuxdevelop</groupId>
    <artifactId>spring-batch-lightmin-embedded</artifactId>
    <version>${spring-batch-lightmin.version}</version>
</dependency>

Configuration

In this case, a specific client does not have to be configured. The Job Configuration Repository and Spring Batch have to be configured like for a regular client.

@SpringBootApplication
@EnableLightminEmbedded
@EnableLightminMapConfigurationRepository
public class EmbeddedLightminApplication {

    public static void main(final String[] args) {
        SpringApplication.run(EmbeddedLightminApplication.class, args);
    }
}

spring:
  batch:
    lightmin:
      batch:
        repository-type: map

Server User Interface

Applications

The start page of the SpringBatchLightmin shows all register applications. The status icon shows the current health status of the application.

Application Instance Dashboard

The application dashboard is the entry point to the monitoring and administration of a client application instance. The overview shows the important endpoints, all known Spring Batch Jobs and configured external links of the client application.