Giter VIP home page Giter VIP logo

request-batcher's Introduction

Request Batcher

Library to collect requests into batches to reduce request overhead.

When dealing with a remote requests, request overhead can be significant. Especially for high request volumes.

But for high volumes, it is often possible (desirable even) to trade a bit of extra latency for a (much) higher throughput. This can be done by batching multiple requests together. This library takes care of the lowlevel stuff around that. All you have to do is handle the batched requests.

Timing

Collecting multiple requests (to other servers) into batches means that multiple threads and multiple servers are involved. This means we'll need to consider timeouts.

The following timeouts are available:

  1. When enqueuing requests, you can specify the maximum time to block before failing.
  2. A queue has a linger time: how long requests can wait on the queue to collect a larger batch
  3. A queued request has a timeout: how long it may take to complete
  4. When acquiring a batch, you specify the maximum timeout to wait for elements to arrive

The queue linger time and the timeout to acquire a batch influence eachother: when acquiring a batch, the specified timeout is an upper bound. The remaining linger time of the elements on the queue is honoured if possible.

Logging and Metrics

This library uses SLF4J for logging and micrometer for metrics.

Logs are written only using the name net.fs.opk.batching.BatchRunner, describing changes in run state.

The following metrics are exposed:

Metric Type Description
<queue name>.queued Timer The time spent on the queue
<queue name>.processing Timer The time it took to process an item since enqueueing

Usage example

In this (contrived) example we'll batch numbered requests, returning a description. In real scenario's you'd perform a network call.

import net.fs.opk.batching.BatchQueue;
import net.fs.opk.batching.BatchElement;
import net.fs.opk.batching.BatchRunnerFactory;

import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;

import static java.util.concurrent.TimeUnit.MILLISECONDS;
import static java.util.concurrent.TimeUnit.SECONDS;

public class Example {
	public static void main(String[] args) {

		// Step 1 (configuration): create a BatchQueue, and start at least one BatchRunner to consume it

		BatchQueue<Integer, String> batchQueue = new BatchQueue<>(10_000,
			1, MILLISECONDS, 30, SECONDS);

		ExecutorService executor = Executors.newFixedThreadPool(1);
		executor.execute(BatchRunnerFactory.forConsumer(batchQueue, 5, batch -> {
			// Here, you'd probably implement a call to a bulk API...
			String suffix = " with " + (batch.size() - 1) + " other elements";
			for (BatchElement<Integer, String> element : batch) {
				element.success("Batched " + element.getInputValue() + suffix);
			}
		}));

		// Step 2 (using): use the queue to batch requests

		for (int request = 0; request < 1000; request++) {
			// Here, you'll want to implement your actual application
			CompletableFuture<String> futureResult = batchQueue.enqueue(request);
			futureResult.thenAccept(System.out::println);
		}

		// Step 3 (cleanup): to close your application, the queue must be shutdown
		// (after that, the BatchRunner will stop itself when the queue is empty)

		batchQueue.shutdown();
		// Optional: wait until the queue is empty and the BatchRunner has terminated.
		batchQueue.awaitShutdownComplete(100, MILLISECONDS);
		// Optional: shutdown the executor if you used a dedicated one.
		executor.shutdown();
	}
}

request-batcher's People

Contributors

opwvhk avatar dependabot[bot] avatar

Stargazers

 avatar Oleksandr avatar  avatar

Watchers

 avatar

request-batcher's Issues

Linger time is not honoured

The linger time is not honoured: when acquiring a batch, a large timeout lets the method wait longer than the linger time allows.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.