Comments (6)
The original line of thinking was that we only need one item but if performance suffers because of that then I am more than happy to increase it. We could certainly do more.
I guess redis has to do more iterations, the testing perhaps becomes a little more complex. Would be cool to have a real world test for the performance of this since we have had some issues around the reaper performance.
If I recall correctly I have some performance related tests already.
from sidekiq-unique-jobs.
Hi @dedman,
I have tried to run some performance tests on the line of code you suggested.
With 1_000_000 (1 million) rows in the retry queue, I get this:
1) SidekiqUniqueJobs::Orphans::RubyReaper#in_sorted_set? when retried with the job in retry is expected to perform under 2 ms
Failure/Error: it { expect { service.send(:in_sorted_set?, "retry", digest) }.to perform_under(2).ms }
expected block to perform under 2 ms, but performed above 4.26 ms (± 6.11 ms)
Setting the count to 1 or 1000 is hardly noticeable. Agreeably this isn't the real world so might not get real world results.
from sidekiq-unique-jobs.
You can check the progress at #777
from sidekiq-unique-jobs.
I think we're hitting this problem too at the moment
from sidekiq-unique-jobs.
Sorry I didn't follow up here earlier. I chatted to my management about this issue earlier in the year and we decided to wait until we were ready to use Sidekiq Enterprise which has a unique jobs feature.
For the last 6 months we kept an eye on the retry queue and tried to keep it under 200-300 jobs. We also increased the timeout on the reaper which lessened the problem a little. We had cron jobs not run probably once a month because of this issue, but we generally noticed early enough to stop it having any big impact.
We were already thinking about moving to Sidekiq Enterprise, so when this issue came up it was the final confirmation it would make sense for us to move over to Enterprise.
from sidekiq-unique-jobs.
Great stuff @dedman, if people paid me those monthly fees I could afford the same quality :)
from sidekiq-unique-jobs.
Related Issues (20)
- Reaper manager registration is subject to race conditions HOT 1
- Should client middleware also be added to the Sidekiq server config? HOT 2
- After sidekiq shutdown (with running job) lock digest is not pushned back HOT 8
- Returning same job id HOT 7
- Sidekiq 7: Unsupported command argument type: TrueClass HOT 1
- register_reaper_process nx: true crash Sidekiq on startup HOT 2
- uniquejobs:digests sorted set seems to grow forever HOT 3
- should respond to `has_valid_sidekiq_options?` HOT 4
- Compatibility with Sidekiq7 HOT 4
- until_and_while_executing not entering perform method on initial run HOT 7
- No 'Changelog' link is being displayed on https://rubygems.org/gems/sidekiq-unique-jobs HOT 1
- Documentation of lock_ttl is inconsistent HOT 5
- Catched a deadlock on Sidekiq 7+ set up using middleware.
- Conflicts with sidekiq-lock gem HOT 4
- Read Timeouts and high CPU spike after upgrading from 8.0.5 to 8.0.10 HOT 9
- Issue with until_and_while_executing lock when job is not enqueued through sidekiq
- until_and_while_executing with separate server and client lock_args broken since 8.0.8 HOT 7
- Replace MD5 with SHA256+ HOT 2
- lock_prefix argument is not respected
- TypeError · Unsupported command argument type: Array HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sidekiq-unique-jobs.