Giter VIP home page Giter VIP logo

rebouncer's Introduction

Rebouncer

A Powerful Debouncer for your Conjuring Needs

Maintenance Go Report Card Go version Go Reference

Rebouncer is a package that takes a noisy source of events and produces a cleaner, fitter, happier (not drinking too much) source. It is useful in scenarios where you want debounce-like functionality, and full control over how events are consumed, filtered, queued and flushed to the consuming process.

Concepts

The NICE Event

NiceEvent is simply the type of event you pass in to Rebouncer. It exists only as a concept so we can have something to refer to:

type Rebouncer[NICE any] interface {
	Subscribe() <-chan NICE 	// the channel a consumer can subsribe to
	emit()                  	// flushes the Queue
	readQueue() []NICE      	// gets the Queue, with safety and locking
	writeQueue([]NICE)      	// sets the Queue, handling safety and locking
	ingest(Ingester[NICE])
	quantize(Quantizer[NICE])   // decides whether the flush the Queue
	reduce(Reducer[NICE], NICE) // removes unwanted NiceEvents from the Queue
	Interrupt()                 //	call this to initiate the "Draining" state
}

type myType struct {
	...
}

bufferSize = 1024 // how much buffer space do we want for incoming events?

//	myRebouncer is a Rebouncer of type myType
myRebouncer := rebouncer.NewRebouncer[myType](ingest, reduce, quantize, bufferSize)

Rebouncer has two run-loops:

Ingest ☞ Reduce

The Ingestor runs in it's own loop, pushing events to a channel in Rebouncer. Every time an event is pushed, Reducer runs. Reducer operates on the entire queue of events, filtering out unwanted events or modifying to taste. Here are the definitions of these functions. NICE is a type parameter. Internally, your custom event type is known as a "Nice Event".

type Ingester[NICE any] func(chan<- NICE)
type Reducer[NICE any] func([]NICE) []NICE

Quantize ☞ Emit

Quantizer returns true or false. True when we want to flush the queue to the consumer, and False when we don't. As soon as Quantizer is returned, it's run again. So to throttle it, do time.Sleep().

When the program enters the Draining state, it shuts down after the last Emit(). Otherwise it keeps looping.

type Quantizer[NICE any] func([]NICE) bool

Ensure that your Ingestor, Reducer, and Quantizer all operate on the same type:

//	Example

type myEvent struct {
	id int
	name string
	timestamp time.Time
}

//	ingest events
ingest := func(incoming<- myEvent) {
	for ev := range mySourceOfEvents() {
		incoming<-ev
	} 
}

//	we're not interested in any event involving .DS_Store
reduce := func(inEvents []myEvent) []myEvent {
	outEvents := []myEvent{}
	for ev := range inEvents {
		if ev.name != ".DS_Store" {
			outEvents = append(outEvents, ev)
		}
	}
	return outEvents
}

//	flush the queue every second
quantize := func(queue []myEvent) bool {
	time.Sleep(time.Second)
	if len(queue) > 0 {
		return true
	} else {
		return false
	}
}

re := rebouncer.NewRebouncer[myEvent](ingest, reduce, quantize, 1024)

for ev := range re.Subscribe() {
	fmt.Println(ev)
}

rebouncer's People

Contributors

sean9999 avatar smacdonald-parallelz avatar

Stargazers

 avatar

Watchers

 avatar  avatar

rebouncer's Issues

improve docs

docs: improve

we need far better documentation, plus examples, plus the removal of exported members no longer needed

add gRPC interface

to make it more generic and composable across languages and systems, let's add gRPC.

Is so, we should think about conforming to the CloudEvent spec

plug-in architecture

a plugin architecture would be smart. Inotify would then be a plug-in. I could try to think of other plug-ins for other use-cases

simplify

fields such as bootID are not required now, let's remove everything non-essential

quantizer

create a default Batcher function that simply creates buckets based on time (ex: 1 second).

  • call it Quantize or quantize
  • create an interface called Batcher that Quantize can implement
  • set the state for passing in funcs (like Batcher) during config phase

switch to inotify

the platform-independant events are too lossy. we need more detail. go all in on inotify

Docker in workflow

A Dockerfile and related docs should be set up. Docker can be used to:

  • run tests
  • make builds targeting different environments
  • run vuln checks
  • proof of concept for a live-reloaded that can work in docker world by issuing ad-hoc docker exec or docker cp commands in response to rebouncer events. it possibly could even be turned into to Docker Desktop extension

more standard event format

I thought of using the CloudEvent standard, but it's too big and complex. Instead, let's use this format: https://github.com/mustafaturan/bus#processing-events

type Event struct {
    ID         string      // identifier
    TxID       string      // transaction identifier
    Topic      string      // topic name
    Source     string      // source of the event
    OccurredAt time.Time   // creation time in nanoseconds
    Data       interface{} // actual event data
}

The TxID can be the cookie id.

add cancelability

we need to handle situations like the root directory being moved and other ways to cancel the incoming event stream, and to cascade that cancellation along the path.

other lifecycle hooks may be important. For example, special events like NiceEvent{Topic: "rebouncer/lifecycle/purge"} could trigger an Emit() or delete the batch without triggering a Emit().

we probably want the context package for this. It seems to make sense to me that an Event should have access to the context of it's parent.

Remove unnecessary NiceEvent wrapping

There is no need to wrap our original events into NiceEvents just to add fields that aren't really necessary.

Let's instead have two generic types: dirty event and nice event.

or Naughty and Nice.

this will obviate the need for user-defined functions to traffic in types that need not know anything about.

Remove Deadlock

there is a deadlock arising from bad design.

IncomingEvents should not be buffered. Rather, it should accept one event at a time, and process it fully before accepting another.

We also need a LifeCycleState property, or a mutex, or another channel to handle the hand-off between Reducer and Quantizer

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.