NOTE: This code's discussion and overview can be found in Couchbase Blog's post Timed tasks using Couchbase and Go.
The goal of the project is to demonstrate how to use specific Couchbase features (pessimistic-locking, indexing, N1QL) to implement a distributed timed tasks system.
The project consists of the following programs:
- Producer - inserts tasks into Couchbase.
- Consumer - retrieves tasks and processes them.
- Consumer cluster - runs multiple consumers in parallel.
cdintodockerfolder.- Run
docker compose upto deploy a local Couchbase server. - Wait ~20 seconds for complete initialization.
- Install
govendor:go get -u github.com/kardianos/govendor. - Install dependencies with
govendor sync. - Edit
cmd/shared/config.goproducer and consumer variables as desired.
Note: a big enough amount of samples (>= 1000) is suggested, to gather correct statistics.
- Start the producer.
- Start the consumer cluster.
- Wait for both programs to automatically terminate (if configured), or press
Enterand wait for them to exit.
Note: execution could take up to 1-2 minutes, depending on configuration.
Run with Windows PowerShell:
go build cmd\producer\producer.go ; .\producer.exe ; rm producer.exe
go build cmd\consumer_cluster\consumer_cluster.go ; .\consumer_cluster.exe ; rm consumer_cluster.exe
Run with bash:
go build cmd/producer/producer.go && ./producer && rm producer
go build cmd/consumer_cluster/consumer_cluster.go && ./consumer_cluster && rm consumer_cluster
When the consumer cluster execution terminates gracefully, a statistics table will be printed.
consumerProcessingTime = 500
consumerProcessingTimeRandomMultiplier = 0.25
When consumers have a low concurrency factory, there will be less failed locking attempts.
totFound| totLockedFound| totProcessed| couldNotLock| loopEfficiency (%)| lockEfficiency (%)|
1060| 108| 106| 0| 100.000000| 49.532711|
1050| 121| 105| 0| 100.000000| 46.460175|
1050| 110| 105| 0| 100.000000| 48.837208|
1060| 94| 106| 0| 100.000000| 52.999996|
1050| 105| 105| 0| 100.000000| 50.000000|
1050| 96| 105| 0| 100.000000| 52.238804|
1050| 113| 105| 0| 100.000000| 48.165138|
1050| 106| 105| 0| 100.000000| 49.763031|
1060| 101| 106| 0| 100.000000| 51.207726|
1060| 103| 106| 0| 100.000000| 50.717705|
consumerProcessingTime = 100
consumerProcessingTimeRandomMultiplier = 1.0
When consumers have a high concurrency factory, there will be more failed locking attempts.
totFound| totLockedFound| totProcessed| couldNotLock| loopEfficiency (%)| lockEfficiency (%)|
2540| 1043| 253| 1| 99.606300| 19.521605|
2600| 1056| 259| 1| 99.615387| 19.695816|
2540| 1067| 254| 0| 100.000000| 19.227858|
2540| 1080| 254| 0| 100.000000| 19.040480|
2560| 985| 255| 1| 99.609375| 20.564516|
2530| 1044| 252| 1| 99.604744| 19.444445|
2570| 1029| 257| 0| 100.000000| 19.984447|
2580| 1052| 258| 0| 100.000000| 19.694656|
2550| 1009| 255| 0| 100.000000| 20.174049|
2510| 1069| 251| 0| 100.000000| 19.015152|
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent